[Mageia-dev] Will this work for a build system?

Sun Sep 26 19:08:55 CEST 2010

On Sun, Sep 26, 2010 at 02:38, Michael Scherer <misc at zarb.org> wrote:
+> Le samedi 25 septembre 2010 à 20:56 -0700, herman a écrit :
+>> On Sat, 2010-09-25 at 10:44 -0700, R James wrote:
+>> > Yeah, if I move the physical server about 2 miles. I can plug it into
+>> > a better ISP.
+>> BTW, I once calculated (test plus extrapolation) how long it would take
+>> to rebuild every package in Mandriva on a low end 2 GHz Celeron server
+>> that I had available and it came to about 80 days.
+>>
+>> Usually one won't rebuild every package all in one go, but one has to do
+>> that the first time.  So, a build system needs to be a big ass piece of
+>> iron, otherwise it will be frustrating to use.
+>
+> Well, for the moment, Mandriva has ( for i586 ) 5 servers with 4 core at
+> 2.8 ghz
+> ( fast scsi disk, 2 g of ram ). So by your estimate and using similar
+> hardware ( which date back to 3/4 years ago ), it would take
+> 80/(5*4*1.5) = 3 days
+>
+> now, that doesn't take in account several factors, like :
+> - others bottleneck ( memory, i/o )
+> - overhead of using iurt and chroot regeneration
+> - overhead of uploading to the mirror
+> - overhead of a non linear repartition ( kde, kernel, openoffice,
+> sagemath, etc, take more time ).
+
+Using a nice machine (24 'cpu', actually 2 * 6 cores + HT, 12GB RAM),
+using few GB tmpfs for the build, I rebuilt main with iurt for one
+arch in less than a day.
+This can be optimized (not extracting chroot each time will improve
+time for small packages, building several in parallel may improve as
+nothing requires 24 cpus 99% of the time and we don't have disk seek
+issue anyway with tmpfs) but that's to give an idea.
+
+The upload to the mirror part can be done in parallel so it does not
+slow the overall process.
+