There is one large file in my server. I find that multi thread download can get 20Mbs, but single thread can get 10Mbps, can anyone explain this?
14 Answers
Usually this is because somewhere between you and the other server there's a firewall limiting each HTTP stream to 10Mbps. When you use multi-thread, you get 2x 10Mb (one for each thread).
2This is due to your ping between you and the server and packet size/tcpip window size used by your downloading software.
Basically, if you have 100ms ping to the server, and request packets of 100kb, you can only get 10 packets per second using 1 connection, even if your internet speed is infinite.
2TCP works best when you "keep the pipe full" — when the sending app keeps sending buffers quickly enough to keep the sender TCP stack constantly supplied with data so that it can always have data "in flight" on the network, and when the receiver app keeps reading from the receiver TCP stack fast enough that the receiver TCP window never fills up (again, so the sending TCP stack can always keep data "in flight" on the network).
I could imagine a poorly-written single-threaded sender app that passes one buffer to the TCP stack, waits to hear that it's been fully Acked, and then passes another buffer. That means that once the end of the first buffer is "in flight" on the network, the sending TCP stack is starved for data to send, which means the pipe drains and isn't refilled until after the Ack comes back and the sending app passes it a new buffer.
I could also imagine a poorly-written single-threaded receiver app that doesn't read from the receiving TCP stack fast enough and thus lets the TCP stack's buffers fill up, which means the TCP window fills up, which causes the sending TCP stack to stop sending until the window opens up some. Increasing the receiver's TCP window size may help a little, but the real solution on this end is to read the data faster.
2Well, that's probably because you can only transfer so much data over one connection. However in a multi-threaded program you can have two connections receiving data at the same time and doubling the amount of information you can obtain. There are some limitations to this for example the speed of the server you are downloading from... Hats off two whoever wrote the multi-threaded downloader, those are not easy to write.
1
