Metrics Files

Here are some representative metrics files to consult and compare against as the system evolves.

SOSP '11 Paper Experiments

Note that all data files (full logs, not just metrics) are in the paper repo. 

Figure 8 (Proportional Scaling, 1 to 20 masters, 6 backup processes per master)

Note that the paper results were averages of 5 runs.

1 Master, 6 backups:

20 Masters, 120 Backups:

Figure 9 (Proportional Scaling, 2 to 60 masters, 2 backup processes per master)

After rerunning a few of the data points from this graph it looks like we (unfortunately) scaled the cluster by adding processes on two machines at a time before starting to double up master/backup processes with backup processes on the same machine. Doubling up processes right away doesn't yield the same results as the paper even with the code from SOSP'11.

2 Masters, 4 Backups:

10 Masters, 20 Backups:

=== Summary === Recovery time: 1.410 s Masters: 10 Backups: 20 Total nodes: 20 Replicas: 3 Objects per master: 592415.0 avg / max 592415.0 Object size: 1024 bytes avg / max 1024 bytes Total live objects: 5924150 Total recovery segment entries: 5924150 Total live object space: 5785.30 MB Total recovery segment space w/ overhead: 5999.99 MB Storage type: disk Log directory: /home/stutsman/src/ramcloud-sosp/recovery/20121018083553 === Coordinator Time === Total: 1407.7 ms / 99.85% of total recovery Starting recovery on backups: 4.5 ms / 0.32% of total recovery Starting recovery on masters: 2.2 ms / 0.16% of total recovery Tablets recovered: 1.0 ms / 0.07% of total recovery Completing recovery on backups: 0.2 ms / 0.01% of total recovery Set will: 0.0 ms / 0.00% of total recovery Get tablet map: 75.4 ms / 5.35% of total recovery Other: 1324.7 ms / 93.96% of total recovery Receiving in transport: 8.9 ms / 0.63% of total recovery === Master Time === Total: 1398.7 ms avg / max 1404.1 ms / 99.21% avg of total recovery Waiting for incoming segments: 562.7 ms avg / max 585.6 ms / 39.91% avg of total recovery Inside recoverSegment: 784.0 ms avg / max 827.1 ms / 55.61% avg of total recovery Backup.proceed: 21.1 ms avg / max 28.3 ms / 1.50% avg of total recovery Verify checksum: 0.0 ms avg / max 0.0 ms / 0.00% avg of total recovery Segment append: 464.6 ms avg / max 487.4 ms / 32.96% avg of total recovery Segment append copy: 49.6 ms avg / max 55.2 ms / 3.52% avg of total recovery Segment append checksum: 224.4 ms avg / max 235.0 ms / 15.92% avg of total recovery Other (HT, etc.): 298.3 ms avg / max 317.9 ms / 21.16% avg of total recovery (other) Final log sync: 13.0 ms avg / max 15.4 ms / 0.92% avg of total recovery Removing tombstones: 0.0 ms avg / max 0.0 ms / 0.00% avg of total recovery Other: 38.9 ms avg / max 53.2 ms / 2.76% avg of total recovery Receiving in transport: 14.0 ms avg / max 24.8 ms / 0.99% avg of total recovery Transmitting in transport: 152.7 ms avg / max 164.8 ms / 10.83% avg of total recovery Replicating one segment: 1.6 ms avg / max 1.7 ms During replay: 1.6 ms avg / max 1.7 ms During log sync: 1.1 ms avg / max 1.3 ms RPC latency replicating one segment: 2.7 ms avg / max 2.9 ms (for R-th replica) During replay: 2.7 ms avg / max 2.9 ms (for R-th replica) During log sync: 1.5 ms avg / max 1.9 ms (for R-th replica) Replication: 1291.3 ms avg / max 1313.0 ms / 91.59% avg of total recovery Client RPCs Active: 1363.4 ms avg / max 1370.1 ms / 96.71% avg of total recovery Average GRD completion time: 6.3 ms avg / max 6.3 ms Task iterations: 2984825 avg / max 3355883 === Backup Time === RPC service time: 379.0 ms avg / max 545.4 ms / 26.88% avg of total recovery startReadingData: 0.2 ms avg / max 0.2 ms / 0.01% avg of total recovery Open/write segment: 369.5 ms avg / max 535.5 ms / 26.21% avg of total recovery Open segment memset: 0.1 ms avg / max 0.1 ms / 0.00% avg of total recovery Copy: 367.8 ms avg / max 532.5 ms / 26.09% avg of total recovery Other: 1.7 ms avg / max 2.8 ms / 0.12% avg of total recovery getRecoveryData: 6.1 ms avg / max 27.4 ms / 0.43% avg of total recovery Other: 3.2 ms avg / max 7.9 ms / 0.23% avg of total recovery Transmitting in transport: 130.5 ms avg / max 164.8 ms / 9.25% avg of total recovery Filtering segments: 248.6 ms avg / max 316.2 ms / 17.64% avg of total recovery Reading segments: 1310.0 ms avg / max 1372.7 ms / 92.92% avg of total recovery Using disk: 1304.0 ms avg / max 1370.3 ms / 92.49% avg of total recovery getRecoveryData completions: 376 avg / max 390 getRecoveryData retry fraction: 0.365 avg / max 0.898 === Efficiency === recoverSegment CPU: 1.04 ms avg (per filtered segment) Writing a segment: 0.73 ms avg (backup RPC thread) Filtering a segment: 6.63 ms avg Memory bandwidth (backup copies): 2.51 GB/s avg / min 1.85 GB/s === Network Utilization === Aggregate: 249.47 Gb/s / 49.89% of network capacity (overall) Master in: 8.50 Gb/s avg / min 7.98 Gb/s / 84.95 Gb/s total (overall) Master out: 11.63 Gb/s avg / min 11.62 Gb/s / 116.29 Gb/s total (overall) Master out during replication: 12.70 Gb/s avg / min 12.47 Gb/s / 127.00 Gb/s total (overall) Master out during log sync: 15.15 Gb/s avg / min 12.29 Gb/s / 151.52 Gb/s total (overall) Backup in: 6.65 Gb/s avg / min 4.39 Gb/s / 133.01 Gb/s total (overall) Backup out: 6.65 Gb/s avg / min 1.64 Gb/s / 133.01 Gb/s total (overall) === Disk Utilization === Effective bandwidth: 226.41 MB/s avg / min 221.30 MB/s / 4528.14 MB/s total Active bandwidth: 230.77 MB/s avg / min 222.62 MB/s / 4615.49 MB/s total Reading: 230.25 MB/s avg / min 219.16 MB/s / 4604.97 MB/s total Writing: 257.32 MB/s avg / min 181.80 MB/s / 4889.07 MB/s total Disk active: 98.12% avg / max 99.54% (of total recovery) Reading: 92.49% avg / max 97.20% (of total recovery) Writing: 5.63% avg / max 10.77% (of total recovery) === Backup Events === Segments read: 37.5 avg / max 39.0 Primary segments loaded: 375.5 avg / max 390.0 Secondary segments loaded: 0.0 avg / max 0.0 === Slowest Servers === Backup opens, writes: server.rc03 (366.2 ms) Stalled reading segs from backups: server.rc03 (585.6 ms) Reading from disk: server.rc02 (219.2 MB/s) Writing to disk: server.rc11 (181.8 MB/s)

60 Masters, 120 Backups: