diff options
Diffstat (limited to 'zarb-ml/mageia-webteam/2011-January/000159.html')
| -rw-r--r-- | zarb-ml/mageia-webteam/2011-January/000159.html | 378 |
1 files changed, 378 insertions, 0 deletions
diff --git a/zarb-ml/mageia-webteam/2011-January/000159.html b/zarb-ml/mageia-webteam/2011-January/000159.html new file mode 100644 index 000000000..85611b712 --- /dev/null +++ b/zarb-ml/mageia-webteam/2011-January/000159.html @@ -0,0 +1,378 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> +<HTML> + <HEAD> + <TITLE> [Mageia-webteam] Forum VM needs + </TITLE> + <LINK REL="Index" HREF="index.html" > + <LINK REL="made" HREF="mailto:mageia-webteam%40mageia.org?Subject=Re%3A%20%5BMageia-webteam%5D%20Forum%20VM%20needs&In-Reply-To=%3CA31E65CCB1B23F49A8B97A3A0947018D01870767%40EXCHANGE2003.ccq.org%3E"> + <META NAME="robots" CONTENT="index,nofollow"> + <META http-equiv="Content-Type" content="text/html; charset=us-ascii"> + <LINK REL="Previous" HREF="000156.html"> + <LINK REL="Next" HREF="000147.html"> + </HEAD> + <BODY BGCOLOR="#ffffff"> + <H1>[Mageia-webteam] Forum VM needs</H1> + <B>Dubeau, Patrick</B> + <A HREF="mailto:mageia-webteam%40mageia.org?Subject=Re%3A%20%5BMageia-webteam%5D%20Forum%20VM%20needs&In-Reply-To=%3CA31E65CCB1B23F49A8B97A3A0947018D01870767%40EXCHANGE2003.ccq.org%3E" + TITLE="[Mageia-webteam] Forum VM needs">Patrick.Dubeau at ccq.org + </A><BR> + <I>Sat Jan 15 19:04:11 CET 2011</I> + <P><UL> + <LI>Previous message: <A HREF="000156.html">[Mageia-webteam] Forum VM needs +</A></li> + <LI>Next message: <A HREF="000147.html">[Mageia-webteam] [Mageia-sysadm] Forum VM needs +</A></li> + <LI> <B>Messages sorted by:</B> + <a href="date.html#159">[ date ]</a> + <a href="thread.html#159">[ thread ]</a> + <a href="subject.html#159">[ subject ]</a> + <a href="author.html#159">[ author ]</a> + </LI> + </UL> + <HR> +<!--beginarticle--> +<PRE> + +----- Message d'origine ----- +De : Maât [mailto:<A HREF="https://www.mageia.org/mailman/listinfo/mageia-webteam">maat-ml at vilarem.net</A>] +Envoyé : Saturday, January 15, 2011 06:41 AM +À : Mageia Web team discussion list <<A HREF="https://www.mageia.org/mailman/listinfo/mageia-webteam">mageia-webteam at mageia.org</A>> +Objet : Re: [Mageia-webteam] Forum VM needs + +Le 13/01/2011 13:29, Michael Scherer a écrit : +><i> Le mercredi 12 janvier 2011 à 21:27 +0100, Maât a écrit : +</I>>><i> Hi there, +</I>>><i> +</I>>><i> As it seems VM creation takes a little bit of time due +</I>>><i> to people being under heavy load at work Anne and misc +</I>>><i> considered the option of creation the Xen VM on one of +</I>>><i> our servers (we could migrate the VM on atalante later) +</I>><i> The exact technology should not matter much, that's also what puppet is +</I>><i> made for. Ie, unless we plan to do a migration at the system image +</I>><i> level, we could simply install the 2nd vm, put puppet, clone the +</I>><i> computer, migrate the db and ip. +</I>><i> +</I>><i> ( not that I do not like xen, but I would prefer something else ). +</I>Well I mentionned Xen because Atalante uses it... but if this is not a +problem for puppet then ok for me :) + + +>><i> For that misc asked for Forum needs... +</I>><i> I think I didn't make myself clear. I wanted information to deploy it +</I>><i> like where is the git stored ( a url, not "it is on a server" ), who +</I>><i> will need what access, etc. But the information you gave are also +</I>><i> important ( and bring lots of question as you can see ). +</I>><i> +</I>Atm it's stored on Ennael's dedibox :) + + + +>><i> For the beginning i'll consider that we are going to put everything on the +</I>same machine +>><i> (DB and PHP). This is not rally brilliant to virtualize DB servers but i +</I>guess this will +>><i> not kill the VM in the first monthes as the tables will not be big. +</I>><i> AFAIK, using virtio and proper cache, this should not be much a problem. +</I>ok then + +>><i> So phpBB needs a LAMP Stack : Apache + PHP5 + MysSQL5 (it prefers to have +</I>MySQLi extention) +><i> No specific requirement in term of version, using 2010.1 rpm should be +</I>><i> ok, I assume ? +</I>indeed + +>><i> And we'll need with php the optional : +</I>>><i> -- zlib compression (better having it) +</I>>><i> -- remote ftp support (well... i'm not in favor even if documentation asks +</I>for it) +><i> We could drop outgoing connexion if needed. +</I>><i> +</I>><i> ( yes, php make me paranoid in term of security ) +</I>><i> +</I>quite understandable :o) + +>><i> -- XML support (better having it) +</I>>><i> -- Image Magick support (better having it) +</I>><i> php-image-magick. I do think there is a conspiration to make me have a +</I>><i> stroke. Security research by a friend of mine on ImageMagick do not make +</I>><i> feel safe to know we will use it, but if this is required, we have no +</I>><i> choice. +</I>><i> +</I>><i> Just to know, what will it be used for ( I assume this will be used to +</I>><i> resize avatar ) ? +</I>yes +>><i> -- GD support (same as Image magick) +</I>><i> Does the forum support suhoshin, or various php hardening measures ? +</I>><i> +</I>Dunno... but we can check that. phpBB forum's got a few threads of errors +mentionning suhosin but i suspect they did not try very hard + +><i> Did you do various testing with a hardened configuration with dangerous +</I>><i> call disabled ( mainly remote url access for a start, but i also think +</I>><i> we can use opendir restriction, etc, etc ). +</I>><i> +</I>><i> Does it have non regression testing ( so we can enable stuff and see if +</I>><i> anything break ? ) ? +</I>><i> +</I>Testing and integration envs are perfect for that :) + + +>><i> For source management git will be used... so we'll need it too :) +</I>><i> Just git clone ? +</I>><i> I have a puppet module for this, just need tests before I commit. +</I>><i> +</I>><i> For git hosting, again, while I am in favor, there is a few questions to +</I>><i> answer and prepare it, see my previous mail about what is needed. +</I>><i> +</I>we need cloning, branch/tag switching and sync with reference repos + +>><i> As forum have often to face bruteforce having Fail2ban would be really +</I>great... +>><i> for every open service like ssh +</I>><i> On ssh level, and for me, that's a vote in favor of "no". We use ssh +</I>><i> keys only for admins, so fail2ban will just cause trouble. +</I>... + +>><i> but also for forums... i'd like to have Fail2ban +</I>>><i> parse a file of phpBB failed login to trigger a IP low level ban during a +</I>>><i> few hours or more... +</I>><i> Well, if you give us the configuration, we can see. +</I>ok + +><i> We can also use the trick that Olivier deployed on d-c to avoid numerous +</I>><i> connexions from the same IP ( in case someone decide to be smart and do +</I>><i> simultaneous attempts to log ). +</I>ok too... i'm curious to see what the trick is :) + +>><i> For forum management we'll need : +</I>><i> ---- 8<---- +</I>><i> or for those who are not CxO-fluent ( private joke ), who is 'we', in +</I>><i> term of organisation ( ie, do we need to create a ldap group, etc ) ? +</I>><i> +</I>We = those who maintain forum at forum admin level (ash and i atm) + +For integration and testing we'll prehaps need a little bit more + +For production you can keep that for sysadmins i guess (till now i got to be +sysadmin & forum admin so this will need a little bit of tuning to adapt +myself) + +>><i> -- access to sources (read/write) +</I>><i> I rather keep this automated from git, for security reasons and to avoid +</I>><i> human errors. I would even add a cron job that does a git diff or +</I>><i> something similar, to detect if someone uploaded a file manually, or +</I>><i> touched to it using apache. +</I>><i> +</I>><i> In fact, as a security measure, I think the user that will write the +</I>><i> source should put it read only for apache. Ie, use a separate system +</I>><i> user for that. +</I>yup that's wise + +but some dirs (avatars upload dir, file upload dir, emoticon uplad dir, cache +dir) will need to be writable for apache... + +>><i> -- access to data zones (avatars and uploaded things) (read/write) +</I>><i> You mean apache will need it, no ? +</I>indeed... apache will need it in the fist place, but sometimes there can be +need to delete an avatar, resize it, replace it with something not offensive, +make it read only to prevent user re-changing again and again... + +><i> Direct access seems to me a pretty rare event, we can grant access if +</I>><i> there is a really lots of request, ie if you annoy admin enough to make +</I>><i> them give it rather than doing themselves. +</I>Well you're right we can tune organisation later... so let's say that +sysadmin will take care of low level tasks on production forum... + +>><i> -- access to accesslogs and errorlogs (read) +</I>><i> then this should be merged with the webmasters concept that romain +</I>><i> explained. For now, we didn't setup anything ( we didn't even split log +</I>><i> file on alamut, even if this should be trivial ). +</I>><i> +</I>indeed forum admins have obviously tasks that are close to this "webmaster" +concept + +>><i> -- ability to change php log levels +</I>><i> This can be done by php, I think. +</I>><i> +</I>if php global config allows it. + +>><i> -- access to php logs (read) +</I>><i> same as accesslogs +</I>yup + +>><i> -- console access to database(s) (i'd prefer to avoid completely +</I>phpMyadmin on the forum server) +><i> I would prefer avoid giving console access until there is a real need. I +</I>would favor then a remote +><i> mysql access, and forcing ssl, maybe even limited by fixed ip address if +</I>you wish to avoid bruteforce. +><i> +</I>><i> ( I will not go to the point of proposing to use a vpn too, but +</I>><i> almost ). +</I>><i> +</I>><i> Maybe we could think of some kind of ssh bastion for such access ( or +</I>><i> maybe that's overkill too ). +</I>><i> +</I>well for forum administration i daily use sql CLI because that's loads more +effective than using phpBB SQL interface or uglier things like phpMyadmin :-/ + +>><i> For performance questions : i guess forum opening will trigger a rather +</I>vast +>><i> amount of people coming (at least to register their nicks)... i'd be happy +</I>to +>><i> avoid the server being loaded to death. +</I>><i> Registration will be done on catdap from what I think we agreed on, no ? +</I>><i> ( correct me if I am wrong ). +</I>you are right but once registered they'll go to the forum (in particular if +they came to registration form through a link on the forum) + +><i> So we need to work on that part ( starting more processes, and so +</I>><i> letting us tune that with puppet ( this is hardcoded now, AFAIK ). So +</I>><i> depending on where we host the forum, we can surely avoid this effect. +</I>:<i>-/ +</I> +>><i> So i'm targetting at least one thousand simultaneous users being active on +</I>the +>><i> forum... that will do for apache tuning. +</I>><i> Ok so let's say 120 simultaneous process for apache, which also mean we +</I>><i> need to keep apache process as lean as possible ( ie, no unused module +</I>><i> loaded ). I assume that there is no guarantee on being thread safe from +</I>><i> php and associated library, so we will use mpm-prefork. +</I>><i> +</I>><i> Since the server is isolated and serve only for php hosting, I guess +</I>><i> using fast-cgi will not bring much to the equation, when compared to +</I>><i> mod_php. +</I>><i> +</I>><i> Let's also assume 30 processes for forum registration on catdap ( if I +</I>><i> am not wrong on that part, of course ) ? We could surely mitigate the +</I>><i> potential overload by not announcing this on every possible channel at +</I>><i> the same time ( ie, first a mail, then a blog post, then +</I>><i> identica/twitter ). +</I>><i> +</I>><i> Should we also maybe need to tune ldap ? +</I>well nginx could be an option as Thierry said... dunno if ldap will be loaded +though + + +>><i> For database that will mean 800 to 1200 requests per seconds... +</I>>><i> +</I>>><i> We'll have 2 - 3 months to see the tables grow and tune the indexes and +</I>the memory accordingly. +><i> That mean that we will have to deploy some monitoring, and we didn't +</I>><i> decided anything ( buchan proposed hobbit, I proposed munin, purely by +</I>><i> familiarity ). +</I>><i> +</I>That can be done a little bit later :) + +><i> What metrics would you need so we can work on them in priority ( once we +</I>><i> start to set up something ) ? +</I>><i> +</I>the ram used by database and by the web server, the number of threads, the +cpu load, and perhaps some data from the database server (cache fill level, +number of locked requests, time taken by long requests - over 1 sec -)... + +>><i> But i think our needs will stabilize around 4-6 GO for RAM if the forum +</I>gets really +>><i> used (we'll have to tune mysql to keep many requests in cache) +</I>apache+mysql all +>><i> included... if we split later apache and mysql on separate machines the +</I>needs on +>><i> each machine will be obviously lower. +</I>><i> No cache ( squid, varnish ) ? +</I>><i> No php level cache too ? +</I>><i> +</I>><i> ( not that it may be requested now ) +</I>we can see that later indeed :) + +>><i> For app disk space code is under 50 megs... and with hundred of avatars +</I>uploader +>><i> we will not grow above 1GO +</I>>><i> +</I>>><i> For database disk space even after years of activity we'll remain under +</I>5GO +><i> Ok so let's allocate a 10 g partition for the db + ssthat on lvm. +</I>><i> We should take in account logs, and logs backup ( french law requires 1 +</I>><i> year of logs ). +</I>><i> +</I>><i> How many logs are to be expected per day ? +</I>><i> The only busy webserver I can think of is d-c, but Nanar and I just +</I>><i> discovered that the configuration is not good. +</I>><i> So now, that's 5g of log, uncompressed, per month. +</I>><i> +</I>it will depend on the success of mageia but that can be rather high + + +>><i> We'll need to set up some tables with heavy read and write accesses with +</I>InnoDB (not all) : +>><i> that would be great to have one file per table innodb option enabled +</I>><i> Ok, I guess it should be safe to enable it for all mysql db I guess. +</I>yes + +>><i> Nota : i'd like to use https (at least for admin accesses)... so that will +</I>mean to enable +>><i> ssl and open 443 port also +</I>><i> We did not plan to let people use their password under cleartext at all. +</I>><i> Centralized auth have been setup ( and should be used for forum too ), +</I>><i> so people will reuse their password, the same used at others part of the +</I>><i> infrastructure, and that mean svn, or bugzilla, etc. Since people with +</I>><i> access will use it, no cleartext at all when the password is sent ( or +</I>><i> over my dead body, after fighting my ghost ). +</I>so https for all ? ok for me but that will be a little bit heavier for cpu + +><i> I guess we can make exception for the cookie, as long as it is not +</I>><i> shared ( ie, we will have to rethink the scheme if we deploy SSO ). +</I>><i> +</I>><i> That also mean that people will complain because of firefox if we do not +</I>><i> buy a certificate. +</I>><i> +</I>:<i>-( +</I> +>><i> That's all for system level... i think directory structures (Which +</I>concerns apache web root config) can be dealt with later... +>><i> +</I>>><i> Tell me if you have got everything you need for VM creation... +</I>><i> What I needed was more information for forum deployment, not vm +</I>><i> requirements, I guess I didn't express myself clearly. The requirement +</I>><i> for the vm in term of memory/disk have been roughly drafted before. What +</I>><i> I would prefer is a deployment document. +</I>><i> +</I>ash and i are writing one atm + +your mail changes some things... so let's adapt it :) + +----8<---- + +(pfeeew... sorry for the length guy) + +Maât + + + + +_______________________________________________ +Mageia-webteam mailing list +<A HREF="https://www.mageia.org/mailman/listinfo/mageia-webteam">Mageia-webteam at mageia.org</A> +<A HREF="https://www.mageia.org/mailman/listinfo/mageia-webteam">https://www.mageia.org/mailman/listinfo/mageia-webteam</A> +</PRE> + + +<!--endarticle--> + <HR> + <P><UL> + <!--threads--> + <LI>Previous message: <A HREF="000156.html">[Mageia-webteam] Forum VM needs +</A></li> + <LI>Next message: <A HREF="000147.html">[Mageia-webteam] [Mageia-sysadm] Forum VM needs +</A></li> + <LI> <B>Messages sorted by:</B> + <a href="date.html#159">[ date ]</a> + <a href="thread.html#159">[ thread ]</a> + <a href="subject.html#159">[ subject ]</a> + <a href="author.html#159">[ author ]</a> + </LI> + </UL> + +<hr> +<a href="https://www.mageia.org/mailman/listinfo/mageia-webteam">More information about the Mageia-webteam +mailing list</a><br> +</body></html> |