The OpenFabrics Alliance distribution includes several benchmark tools, e.g.:
However, ib_send_lat's results (and probably others...) are a little difficult to make sense of. Here are some things worth keeping in mind:
In addition, it appears that the ib_write_lat benchmark uses writes in one direction, and ib_post_sends in the other to signal completion. If times are calculated like ib_send_lat, then the values obtained are 1/2 RTT, which conflates RDMA and send/receive operation times. This is probably more honest for an actual RDMA user, but may give a worse one-way time for the write operation.
Note:
Packet Size |
Inlined? |
Type |
Avg. Infiniband usec |
Avg. RoCE usec |
---|---|---|---|---|
4 |
N |
UD |
5.10 |
6.32 |
4 |
Y |
UD |
3.24 |
4.48 |
4 |
N |
RC |
5.41 |
6.49 |
4 |
Y |
RC |
3.56 |
4.62 |
128 |
N |
UD |
5.50 |
6.90 |
128 |
Y |
UD |
3.88 |
5.28 |
128 |
N |
RC |
5.45 |
6.86 |
128 |
Y |
RC |
3.94 |
5.19 |
1024 |
N/A |
UD |
7.14 |
9.59 |
1024 |
N/A |
RC |
7.07 |
9.46 |
client did 1000000 RTTs in 12355458471 ticks (12355 ticks/RTT) client min ticks/RTT: 11361, max ticks/RTT: 329547 avg: 5.148 usec histogram: 4 : 111740 (cdf 11.1740%) 5 : 836703 (cdf 94.8443%) 6 : 38461 (cdf 98.6904%) 7 : 7673 (cdf 99.4577%) 8 : 3820 (cdf 99.8397%) 9 : 340 (cdf 99.8737%) 10 : 53 (cdf 99.8790%) 11 : 1084 (cdf 99.9874%) 12 : 29 (cdf 99.9903%) 13 : 30 (cdf 99.9933%) 14 : 22 (cdf 99.9955%) 15 : 10 (cdf 99.9965%) 16 : 12 (cdf 99.9977%) 17 : 2 (cdf 99.9979%) 21 : 1 (cdf 99.9980%) 23 : 1 (cdf 99.9981%) 24 : 1 (cdf 99.9982%) 28 : 1 (cdf 99.9983%) 29 : 2 (cdf 99.9985%) 31 : 1 (cdf 99.9986%) 34 : 1 (cdf 99.9987%) 35 : 1 (cdf 99.9988%) 38 : 2 (cdf 99.9990%) 41 : 1 (cdf 99.9991%) 60 : 1 (cdf 99.9992%) 70 : 1 (cdf 99.9993%) 93 : 1 (cdf 99.9994%) 97 : 1 (cdf 99.9995%) 102 : 1 (cdf 99.9996%) 104 : 1 (cdf 99.9997%) 106 : 1 (cdf 99.9998%) 109 : 1 (cdf 99.9999%) client did 1000000 RTTs in 12355458471 ticks (12355 ticks/RTT) |
client did 1000000 RTTs in 12630466584 ticks (12630 ticks/RTT) client min ticks/RTT: 11901, max ticks/RTT: 253923 avg: 5.263 usec histogram: 4 : 159 (cdf 0.0159%) 5 : 951073 (cdf 95.1232%) 6 : 37846 (cdf 98.9078%) 7 : 7682 (cdf 99.6760%) 8 : 2831 (cdf 99.9591%) 9 : 264 (cdf 99.9855%) 10 : 59 (cdf 99.9914%) 11 : 16 (cdf 99.9930%) 12 : 12 (cdf 99.9942%) 13 : 14 (cdf 99.9956%) 14 : 6 (cdf 99.9962%) 15 : 12 (cdf 99.9974%) 16 : 10 (cdf 99.9984%) 17 : 3 (cdf 99.9987%) 18 : 1 (cdf 99.9988%) 24 : 2 (cdf 99.9990%) 40 : 1 (cdf 99.9991%) 49 : 1 (cdf 99.9992%) 50 : 2 (cdf 99.9994%) 90 : 1 (cdf 99.9995%) 93 : 1 (cdf 99.9996%) 101 : 1 (cdf 99.9997%) 104 : 2 (cdf 99.9999%) client did 1000000 RTTs in 12630466584 ticks (12630 ticks/RTT) |