diff options
Diffstat (limited to 'zarb-ml/mageia-dev/20110112/002077.html')
| -rw-r--r-- | zarb-ml/mageia-dev/20110112/002077.html | 211 |
1 files changed, 211 insertions, 0 deletions
diff --git a/zarb-ml/mageia-dev/20110112/002077.html b/zarb-ml/mageia-dev/20110112/002077.html new file mode 100644 index 000000000..200ff664e --- /dev/null +++ b/zarb-ml/mageia-dev/20110112/002077.html @@ -0,0 +1,211 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> +<HTML> + <HEAD> + <TITLE> [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download + </TITLE> + <LINK REL="Index" HREF="index.html" > + <LINK REL="made" HREF="mailto:mageia-dev%40mageia.org?Subject=Re%3A%20%5BMageia-dev%5D%20Proposal%20for%20Mageia%3A%20implement%20bitorrent%20protocol%0A%20to%20allow%20updates%20download&In-Reply-To=%3C4D2D6CF4.6010309%40laposte.net%3E"> + <META NAME="robots" CONTENT="index,nofollow"> + <META http-equiv="Content-Type" content="text/html; charset=us-ascii"> + <LINK REL="Previous" HREF="002083.html"> + <LINK REL="Next" HREF="002082.html"> + </HEAD> + <BODY BGCOLOR="#ffffff"> + <H1>[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download</H1> + <B>andre999</B> + <A HREF="mailto:mageia-dev%40mageia.org?Subject=Re%3A%20%5BMageia-dev%5D%20Proposal%20for%20Mageia%3A%20implement%20bitorrent%20protocol%0A%20to%20allow%20updates%20download&In-Reply-To=%3C4D2D6CF4.6010309%40laposte.net%3E" + TITLE="[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download">andr55 at laposte.net + </A><BR> + <I>Wed Jan 12 09:57:24 CET 2011</I> + <P><UL> + <LI>Previous message: <A HREF="002083.html">[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download +</A></li> + <LI>Next message: <A HREF="002082.html">[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download +</A></li> + <LI> <B>Messages sorted by:</B> + <a href="date.html#2077">[ date ]</a> + <a href="thread.html#2077">[ thread ]</a> + <a href="subject.html#2077">[ subject ]</a> + <a href="author.html#2077">[ author ]</a> + </LI> + </UL> + <HR> +<!--beginarticle--> +<PRE>Michael Scherer a écrit : +><i> +</I>><i> Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit : +</I>>><i> Michael Scherer a écrit : +</I>>>><i> +</I>>>><i> Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit : +</I>>>>><i> hi all, +</I>>>>><i> i have one question (maybe it can be a proposal): is it possible to implement +</I>>>>><i> the torrent protocol to faster download the updates of the distro? it could be +</I>>>>><i> an interesting features for the coming Mageia releases +</I>>>><i> +</I>>>><i> I think the issue of faster download could be simply taken care by +</I>>>><i> having more mirror, or faster one. +</I>>>><i> +</I>>>><i> I had under the impression that some ISP throttle down bittorrent, and +</I>>>><i> that it may not be very nat and firewall friendly.. +</I>>><i> +</I>>><i> Some suggestions for faster downloads without bittorrent. +</I>>><i> 1) use aria2c (or a similar application), which uses multiple +</I>>><i> connections, defaulting to 5, and allows multiple mirrors. +</I>>><i> By default it starts by allocating space for the file to be downloaded, +</I>>><i> which allows non-sequential downloading of the file, facilitating faster +</I>>><i> downloading from multiple sites. +</I>>><i> +</I>>><i> 2) use mirrors which allow multiple connexions. +</I>>><i> (Of course, with download software that takes advantage of this.) +</I>>><i> +</I>>><i> 3) use multiple mirrors. +</I>>><i> (Again, according to download software.) +</I>><i> +</I>><i> Theses 3 suggestions basically put X time the load of the mirror for +</I>><i> each client. ( or on more mirror, for that matters ). +</I>><i> +</I>><i> And that's quite bad from the point of view of a mirror manager. +</I> +Why would it make _any_ difference to the mirrors ? +Besides spreading the download over multiple mirrors. +We download the same amount of data from the mirrors in any case. +So we want to accelerate the download. Some mirrors might want to +throttle the download, or limit the connexions, to spread the download +over a longer period of time, and they still can. +This approach just works around such limitations for the end user. + +BTW, aria2c monitors the download speed, to choose the faster mirrors +for multiple connexions. (From the urls put on the command line.) + +><i> For example, distrib-coffee could blacklist you if you do this, if you +</I>><i> are not alone on your network connexion. And when we deployed this +</I>><i> measure to protect the server, the limit was 2 connexion per address, +</I>><i> since this was taking too much ressources on the old server ( each http +</I>><i> request taking 1 process and so memory ). Hopefully, the hardware was +</I>><i> upgraded but not everybody can afford 32g of ram and 8*2 ghz CPU. +</I> +Such restrictions are perfectly legitimate for a mirror. + +><i> [<A HREF="https://www.mageia.org/mailman/listinfo/mageia-dev">root at distrib-coffee</A> ~]# grep -B 6 -A 3 +</I>><i> MaxConnPerIP /etc/httpd/conf.d/distrib-coffee.conf +</I>><i> +</I>><i> <Directory /var/ftp/pub> +</I>><i> order allow,deny +</I>><i> Allow from all +</I>><i> Options +Indexes +MultiViews +SymLinksIfOwnerMatch +</I>><i> <IfModule mod_limitipconn.c> +</I>><i> MaxConnPerIP 5 +</I>><i> ErrorDocument 503 "5 connections at the same time only allowed." +</I>><i> </IfModule> +</I>><i> </Directory> +</I>><i> +</I>><i> So I think pissing off mirror maintainers is likely the wrong way of +</I>><i> solving the problem ( who was not properly explained nor looked at +</I>><i> besides "it should be faster" ). +</I> +Again, the same amount of data is downloaded collectively from the +mirrors used. +And I agree totally with mirrors using whatever download speed controls +they wish. (It is strongly advisable for mirrors with limited resources +to put such controls in place.) +Just as downloaders are free to work within the limits of such controls +to their advantage. + +>><i> 4) use ftp instead of http +</I>><i> +</I>><i> Based on ? +</I> +My experience. Most downloads are noticeably faster downloading larger +files via ftp. (Not just Mandriva downloads by any means.) +And not just my current provider, either. I use a very fast internet +access for most of my ISO downloads. (The local library.) + +><i> If this is based on using d-c, again, that's our custom QOS rules. If +</I>><i> this is because of throttling on your provider, not everybody have the +</I>><i> same provider, and so the same throttling. +</I>><i> The only difference between http and ftp is that ftp server will likely +</I>><i> scale better server side. +</I> +An explanation ... +And maybe at least some http servers don't allow out-of-order packet +requests ? + +>><i> 5) use closer mirrors. (less delay in handshaking, etc.) +</I>><i> +</I>><i> I think tcp handshake is not much a problem, given the fact it happen +</I>><i> once per rpm, compared to the number of tcp packet for the rest of the +</I>><i> download. Use wireshark to see. +</I> +Again based on my experience. +BTW, this is mostly ISOs, since the download time of smaller files -- +like typical .rpms -- is relatively fast. +Isn't there some sort of handshaking and verification by tcp packet ? +Also propagation time in the event of errors should explain at least +part of the difference. + +I could have added download to the fastest disk available, but that +seemed obvious. (e.g. one could download to a usb disk, but that would +be painfully slow, unless one has a usb3 disk on a usb3 port.) + +>><i> In my case, using aria2c with 2 mirrors and the default 5 connexions is +</I>>><i> at least 3 times as fast as a single connexion (to my closest mirror). +</I>>><i> And a much greater improvement over other download options I've tried. +</I>>><i> +</I>>><i> I also have configured rpmdrake to use aria2c -- it seems to give me +</I>>><i> faster and more reliable updating, but I don't have any figures. +</I>>><i> +</I>>><i> aria2c is a console app, but it works well enough for me that I haven't +</I>>><i> (yet) bothered to install the available GUI frontend. +</I>><i> +</I>><i> That's because it worked in your case that it would work for every +</I>><i> possible case, especially without giving a proper analysis of the issue +</I>><i> on your side. +</I> +True. "Suggestions" based on my experience, in response to a request +for faster downloads. +My figures were based on the same very fast internet access point (the +same evening, in fact), so the differences have to be attributed server +side. The ratio was actually about 3.5 : 1 (according to aria2c). The +closest mirror allows only 1 connexion. Tests in the past have shown +that other mirrors (not much further) have about the same speed for a +single connexion. But they allow multiple connexions, which is the +default for aria2c. +In case you haven't used aria2c, it prints the download speed with the +estimated finishing time and current number of connexions, once every +minute. (In this comparison, the number of connexions fluctuated +between 4 and 5.) + +Agreed, I could have provided more suppositions as to why. + +Regards :) + +-- +André +</PRE> + + + + + + + +<!--endarticle--> + <HR> + <P><UL> + <!--threads--> + <LI>Previous message: <A HREF="002083.html">[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download +</A></li> + <LI>Next message: <A HREF="002082.html">[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download +</A></li> + <LI> <B>Messages sorted by:</B> + <a href="date.html#2077">[ date ]</a> + <a href="thread.html#2077">[ thread ]</a> + <a href="subject.html#2077">[ subject ]</a> + <a href="author.html#2077">[ author ]</a> + </LI> + </UL> + +<hr> +<a href="https://www.mageia.org/mailman/listinfo/mageia-dev">More information about the Mageia-dev +mailing list</a><br> +</body></html> |