Latency v Throughput

Which is the faster way to get your cargo across the United States. A plane or a train? Some might think the answer is obvious. A plane travels 500 mph (or so) and a train does maybe 80 mph. Therefore the plane is faster. Or is it? The question is really a matter of latency vs. throughput.

Imagine you have to move a bunch of coal across the country and deliver it to a coal processor. Now say that on the west coast, the receiver of the coal can process 100 units of coal an hour. You have 1 train that can haul 10,000 units of coal and takes 48 hours to get to its destination. You have 1 plane that can deliver 100 units of coal in 12 hours.

If the most important thing was to have the coal soon, then the plane is faster (lower latency). But, if the most important thing is to have the coal-processing pipeline filled on the west coast over time then train is faster (higher throughput). Every 96 hours they get 10k units of coal with the train (remember there’s only one train and, just like the plane, it must make the return trip to the east coast). That works out to about 100 units an hour which is just what you need. With the plane, every 96 hours you get 800 pounds of coal. Not nearly fast enough.

The above discussion may seem obvious but I have this conversation all the time when talking about Software: what is fast and what is slow. I've had people tell me it's impossible to do 10 thousand transactions per second in Terracotta when persistent because the disk seek time is 10 millis. Well they would be right if you serialize things. But in infrastructure software, the game is throughput with acceptable latency and it turns out 10 thousand transactions per second isn't all that hard. With parallelism, batching, and windowing, the disk isn't even usually the bottleneck.

Anyway, just wanted to get the throughput v latency thing off my chest.