A single backup operation (ClusterPerf with 100-byte writes, 1 master, 3 backups). For each operation, a timeline of events was logged. Not all timelines had the same "shape", as not all writes are handled by the same sequence of events. Thus, the most common timeline "shape" was chosen, and the timelines below represent the average of the most common timeline shape. This procedure was done for both the backup and the master.
Averaged over 1912 same-shape timelines.
0 us --- Begin backup (BackupManager::sync()) | | 2.0 us --- First write RPC sent out | | 3.3 us --- Second write RPC sent out | | 4.5 us --- Third write RPC sent out | | | [~ 4 us "dead time"] | | 8.6 us --- First write RPC completes (duration: 6.6 us) | 9.8 us --- Second write RPC completes (duration: 6.5 us) | 10.8 us --- Third write RPC completes (duration: 6.3 us) 10.9 us --- End backup |
Average over 9584 same-shape timelines.
0 us ---- InfRcTransport Poller picks ups incoming RPC [dispatch thread] | 255 ns -- Invoke service.handleRpc() [worker thread] | 833 ns -- Completed service.handleRpc() [worker thread] | 991 ns -- Begin sending reply [dispatch thread] | | | 1.8 us -- Completed worker->rpc->sendReply() [dispatch thread] |
Simple program to benchmark 56-byte write.
Averaged over 100 samples.
Using RDMA: 2.50495 us
Using IB send: 4.969 us (explains write RPC latency seen in RAMCloud: 5 + 1 = 6 us)
Using RDMA: 4.866 us
We see that a one-way RDMA easily beats the round-trip IB send's currently used RAMCloud RPC.