How is iperf so fast?

[DUOLabs333]

I'm writing a program that requires high throughput and low latency --- however, no matter what I do, my program is only able to achieve around half of the throughput of iperf (eg, on local TCP (ie, from one process to another), iperf is able to get ~ 14 GBps, while my program is only able to get ~5 GBps), even though after looking through iperf's source code, both seem to be doing the same thing (ie, use send/recv and disable Nagle's).

For my own program, I'm calculating bitrate by dividing the amount of data sent/received (since in my benchmarks, I essentially just send data, then immediately request it again) by the time it took to send it --- does iperf use a different formula?

[davidBar-On]

... calculating bitrate by dividing the amount of data sent/received (since in my benchmarks, I essentially just send data, then immediately request it again) by the time it took to send it ...

iperf3 does not receiver does not send back the data it received. With --bidir each side sends data to the other side in parallel. Each side calculates the bit rate of the data it sent/received (with TCP should be the same for both sides, with UDP it is different because of lost messages). At the end of the test, the server sends it statistics to the client. Therefore, it may be that the difference is because in you test the receiver is sending back the data, so the sender is waiting to receive the data before sending the next message.

Another point is that by default iperf3 sends 128Kbyte messages. If messages are much smaller (e.g. 1KB) then there is a quite big throughput difference.

[DUOLabs333]

Sorry for the formatting issue --- I've just went back and escaped the asterisks.

Yes, you are right that the sender will be sending only half of the time --- however, I count the time for the sender to send and receive, and I count the amount of data the sender sends and receives, so this should have already been taken into account (ie, if the sender sends and receives, but I only consider the data the sender sends, then it would make sense for the throughput to be half of the theoretical maximum).

[davidBar-On]

I think that sending and receiving is not the same as continuously sending. E.g. after the sender sends the last chunk of the 4MB it takes time for that packet to reach the receiver and be processed. Only then the receiver starts to send back the packet. If this is your code, I suggest to try sending all the data at once, i.e. a 256 times (practically, maybe fewer) loop of sending 4MG and see what happens.

Also to compare with iperf3 I suggest to use the same message size. Since iperf3 is limited to 1MB message, set -l 1M and in your program reduce the messages to 1MB.

Another way to check what happens is using Wireshark to monitor the traffic for both the iperf3 and your program' and see what are the differences. E.g. you may find that for some reason there are many TCP retries with your program, you can see the delays between sending by the sender and sending back by the receiver, etc.

[DUOLabs333]

Hmmm.. I just checked, and modifying my benchmark to just send data doesn't lead to much better results, and using -l 1M in iperf still gives the same high throughput, but using 1MB buffers doesn't improve throughput in my own code.

Wireshark dropped too many packets to be much of use (decreasing the amount of packets sent by iperf to 256 dramatically decreases the reported throughput, which is interesting)

[davidBar-On]

I don't have other suggestions then logging the traffic using Wireshark or tcpdump (usually in 'pcap' file format) and comparing between your program and iperf3 logs (if you can send such logs I may be able to help with the evaluation).

[DUOLabs333]

UPDATE: It turns out that boost asio was limiting my benchmark's throughput by 10-20% --- removing it and using raw sockets got me much closer (11 GBps vs 14 GBps).

[bmah888]

Hi @DUOLabs333, the biggest question in my mind is what is your program supposed to actually do? Unless you're writing a toy program just for fun (which by the way is a great learning exercise) your program is trying to do accomplish something specific by transferring some data over the network to another process. That pattern of communication (bulk data transfer? request-response?) might be different from what iperf3 does, but it's required for whatever your application needs to do. If that's true, you can't really compare the performance of your program to iperf3 because you measure different things.

iperf3 basically shovels data down a socket (or several sockets). It might rate-limit itself based on command-line options, but the default for TCP transfer is just to keep sending data as fast as the TCP congestion control algorithm (and flow control on the receiver side) will let it go. On the sending side (which be either the client or server depending on command-line options) it counts the number of bytes sent and divides by the time it took to send them all. Think something similar for the receiver. There's no magic here. Is this what your program is doing?

You should also supply a few other details. What operating system? What's the network path look like between the two hosts (like is this a single Ethernet network in your house or a campus LAN or a national-scale WAN)? (iperf3 was designed around Linux hosts, generally attached to wide-area networks with link speeds at least 10Gbps.)

[DUOLabs333]

For my benchmark, it is just sending bytes from one process to another (both processes are on the same network). I'm running macOS.