[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download

Wed Jan 12 04:29:43 CET 2011

Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit :
+> Michael Scherer a écrit :
+> >
+> > Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit :
+> >> hi all,
+> >> i have one question (maybe it can be a proposal): is it possible to implement
+> >> the torrent protocol to faster download the updates of the distro? it could be
+> >> an interesting features for the coming Mageia releases
+> >
+> > I think the issue of faster download could be simply taken care by
+> > having more mirror, or faster one.
+> >
+> > I had under the impression that some ISP throttle down bittorrent, and
+> > that it may not be very nat and firewall friendly..
+> 
+> Some suggestions for faster downloads without bittorrent.
+> 1) use aria2c (or a similar application), which uses multiple 
+> connections, defaulting to 5, and allows multiple mirrors.
+> By default it starts by allocating space for the file to be downloaded, 
+> which allows non-sequential downloading of the file, facilitating faster 
+> downloading from multiple sites.
+> 
+> 2) use mirrors which allow multiple connexions.
+> (Of course, with download software that takes advantage of this.)
+> 
+> 3) use multiple mirrors.
+> (Again, according to download software.)
+
+Theses 3 suggestions basically put X time the load of the mirror for
+each client. ( or on more mirror, for that matters ).
+
+And that's quite bad from the point of view of a mirror manager.
+
+For example, distrib-coffee could blacklist you if you do this, if you
+are not alone on your network connexion. And when we deployed this
+measure to protect the server, the limit was 2 connexion per address,
+since this was taking too much ressources on the old server ( each http
+request taking 1 process and so memory ). Hopefully, the hardware was
+upgraded but not everybody can afford 32g of ram and 8*2 ghz CPU. 
+
+[root at distrib-coffee ~]# grep -B 6 -A 3
+MaxConnPerIP  /etc/httpd/conf.d/distrib-coffee.conf
+
+<Directory /var/ftp/pub>
+   order allow,deny
+   Allow from all
+   Options +Indexes +MultiViews +SymLinksIfOwnerMatch
+   <IfModule mod_limitipconn.c>
+       MaxConnPerIP 5
+       ErrorDocument 503 "5 connections at the same time only allowed."
+   </IfModule>
+</Directory>
+
+So I think pissing off mirror maintainers is likely the wrong way of
+solving the problem ( who was not properly explained nor looked at
+besides "it should be faster" ).
+
+> 4) use ftp instead of http
+
+Based on ?
+
+If this is based on using d-c, again, that's our custom QOS rules. If
+this is because of throttling on your provider, not everybody have the
+same provider, and so the same throttling.
+The only difference between http and ftp is that ftp server will likely
+scale better server side. But that should have no impact on file serving
+and download ( both nowadays use fast syscall, such as sendfile, etc ).
+
+> 5) use closer mirrors.  (less delay in handshaking, etc.)
+
+I think tcp handshake is not much a problem, given the fact it happen
+once per rpm, compared to the number of tcp packet for the rest of the
+download. Use wireshark to see. 
+
+
+> In my case, using aria2c with 2 mirrors and the default 5 connexions is 
+> at least 3 times as fast as a single connexion (to my closest mirror).
+> And a much greater improvement over other download options I've tried.
+> 
+> I also have configured rpmdrake to use aria2c -- it seems to give me 
+> faster and more reliable updating, but I don't have any figures.
+> 
+> aria2c is a console app, but it works well enough for me that I haven't 
+> (yet) bothered to install the available GUI frontend.
+
+That's because it worked in your case that it would work for every
+possible case, especially without giving a proper analysis of the issue
+on your side.
+
+-- 
+Michael Scherer
+
+