2011-02-11
All tests used R=1 and quiesced writes. Disk tests used /dev/sdb1.
In memory, large objects
=== Summary ===
Recovery time: 0.817 s
Masters: 1
Backups: 6
Number of objects: 81442
Object size: 8192.0
=== Coordinator Time ===
Idle: 770.6 ms / 94.37% of total recovery
In Recovery constructor: 21.2 ms / 2.59% of total recovery
Recovery::start: 20.2 ms / 2.47% of total recovery
=== Master Time ===
Total: 796.2 ms / 97.51% of total recovery
Inside recoverSegment: 710.6 ms / 87.03% of total recovery
Approx. CPU: 429.1 ms / 52.55% of total recovery (above minus BackupManager time)
Verify checksum: 176.5 ms / 21.62% of total recovery
Segment append copy: 54.6 ms / 6.69% of total recovery
Segment append checksum: 160.4 ms / 19.65% of total recovery
Waiting for backups: 154.0 ms / 18.85% of total recovery
Stalled on segment open: 139.3 ms / 17.07% of total recovery
Stalled on segment write: 8.9 ms / 1.09% of total recovery
Stalled on segment read: 5.7 ms / 0.70% of total recovery
Removing tombstones: 34.1 ms / 4.18% of total recovery
Transmitting in transport: 130.0 ms / 15.92% of total recovery
Copying: 129.5 ms / 15.87% of total recovery
=== Backup Time ===
Total: 794.4 ms avg / stddev 0.9 ms / 97.29% avg of total recovery
Idle: 732.5 ms avg / stddev 10.2 ms / 89.71% avg of total recovery
sRD: 0.2 ms avg / stddev 0.0 ms / 0.02% avg of total recovery
write: 37.5 ms avg / stddev 11.8 ms / 4.59% avg of total recovery
Filtering segments: 61.3 ms avg / stddev 13.4 ms / 7.51% avg of total recovery
Transmitting in transport: 23.3 ms avg / stddev 5.7 ms / 2.86% avg of total recovery
Copying: 23.2 ms avg / stddev 5.7 ms / 2.84% avg of total recovery
=== Network Utilization ===
Master in: 6.11 Gb/s (overall)
Master out: 6.11 Gb/s (overall)
Backup in: 1.02 Gb/s avg / stddev 0.33 Gb/s (overall)
Backup out: 1.02 Gb/s avg / stddev 0.26 Gb/s (overall)
=== Disk Utilization ===
Overall bandwidth: 258.00 MB/s avg / stddev 36.95 MB/s
Active bandwidth: 1898.39 MB/s avg / stddev 95.92 MB/s
Disk active: 13.57% avg / stddev 1.64% (of total recovery)
Reading: 6.92% avg / stddev 1.59% (of total recovery)
Writing: 6.65% avg / stddev 2.09% (of total recovery)
In memory, small objects
=== Summary ===
Recovery time: 2.180 s
Masters: 1
Backups: 6
Number of objects: 3812983
Object size: 128.0
=== Coordinator Time ===
Idle: 2133.7 ms / 97.86% of total recovery
In Recovery constructor: 21.1 ms / 0.97% of total recovery
Recovery::start: 20.2 ms / 0.93% of total recovery
=== Master Time ===
Total: 2160.1 ms / 99.07% of total recovery
Inside recoverSegment: 1923.7 ms / 88.23% of total recovery
Approx. CPU: 1714.8 ms / 78.65% of total recovery (above minus BackupManager time)
Verify checksum: 161.6 ms / 7.41% of total recovery
Segment append copy: 139.1 ms / 6.38% of total recovery
Segment append checksum: 109.9 ms / 5.04% of total recovery
Waiting for backups: 90.1 ms / 4.13% of total recovery
Stalled on segment open: 87.3 ms / 4.01% of total recovery
Stalled on segment write: 2.8 ms / 0.13% of total recovery
Stalled on segment read: 0.1 ms / 0.00% of total recovery
Removing tombstones: 194.6 ms / 8.92% of total recovery
Transmitting in transport: 114.2 ms / 5.24% of total recovery
Copying: 113.7 ms / 5.22% of total recovery
=== Backup Time ===
Total: 2158.2 ms avg / stddev 0.9 ms / 98.99% avg of total recovery
Idle: 2102.4 ms avg / stddev 7.5 ms / 96.42% avg of total recovery
sRD: 0.2 ms avg / stddev 0.0 ms / 0.01% avg of total recovery
write: 34.4 ms avg / stddev 5.5 ms / 1.58% avg of total recovery
Filtering segments: 103.4 ms avg / stddev 29.4 ms / 4.74% avg of total recovery
Transmitting in transport: 20.5 ms avg / stddev 4.6 ms / 0.94% avg of total recovery
Copying: 20.3 ms avg / stddev 4.6 ms / 0.93% avg of total recovery
=== Network Utilization ===
Master in: 2.14 Gb/s (overall)
Master out: 2.14 Gb/s (overall)
Backup in: 0.36 Gb/s avg / stddev 0.07 Gb/s (overall)
Backup out: 0.36 Gb/s avg / stddev 0.09 Gb/s (overall)
=== Disk Utilization ===
Overall bandwidth: 90.51 MB/s avg / stddev 15.37 MB/s
Active bandwidth: 2005.44 MB/s avg / stddev 93.24 MB/s
Disk active: 4.53% avg / stddev 0.84% (of total recovery)
Reading: 2.33% avg / stddev 0.70% (of total recovery)
Writing: 2.20% avg / stddev 0.41% (of total recovery)
Disk, large objects
=== Summary ===
Recovery time: 2.457 s
Masters: 1
Backups: 6
Number of objects: 81442
Object size: 8192.0
=== Coordinator Time ===
Idle: 2411.4 ms / 98.13% of total recovery
In Recovery constructor: 21.2 ms / 0.86% of total recovery
Recovery::start: 20.1 ms / 0.82% of total recovery
=== Master Time ===
Total: 2437.1 ms / 99.17% of total recovery
Inside recoverSegment: 2053.1 ms / 83.54% of total recovery
Approx. CPU: 430.0 ms / 17.50% of total recovery (above minus BackupManager time)
Verify checksum: 177.4 ms / 7.22% of total recovery
Segment append copy: 53.5 ms / 2.18% of total recovery
Segment append checksum: 160.8 ms / 6.54% of total recovery
Waiting for backups: 1806.3 ms / 73.50% of total recovery
Stalled on segment open: 1008.7 ms / 41.05% of total recovery
Stalled on segment write: 486.5 ms / 19.80% of total recovery
Stalled on segment read: 311.1 ms / 12.66% of total recovery
Removing tombstones: 34.2 ms / 1.39% of total recovery
Transmitting in transport: 123.8 ms / 5.04% of total recovery
Copying: 123.4 ms / 5.02% of total recovery
=== Backup Time ===
Total: 2435.3 ms avg / stddev 0.9 ms / 99.10% avg of total recovery
Idle: 1887.7 ms avg / stddev 343.1 ms / 76.82% avg of total recovery
sRD: 0.2 ms avg / stddev 0.0 ms / 0.01% avg of total recovery
write: 57.8 ms avg / stddev 10.0 ms / 2.35% avg of total recovery
Filtering segments: 55.0 ms avg / stddev 13.2 ms / 2.24% avg of total recovery
Transmitting in transport: 21.9 ms avg / stddev 5.1 ms / 0.89% avg of total recovery
Copying: 21.7 ms avg / stddev 5.1 ms / 0.88% avg of total recovery
=== Network Utilization ===
Master in: 2.03 Gb/s (overall)
Master out: 2.03 Gb/s (overall)
Backup in: 0.34 Gb/s avg / stddev 0.05 Gb/s (overall)
Backup out: 0.34 Gb/s avg / stddev 0.08 Gb/s (overall)
=== Disk Utilization ===
Overall bandwidth: 85.73 MB/s avg / stddev 12.70 MB/s
Active bandwidth: 107.90 MB/s avg / stddev 6.38 MB/s
Disk active: 79.82% avg / stddev 12.88% (of total recovery)
Reading: 39.22% avg / stddev 8.80% (of total recovery)
Writing: 40.59% avg / stddev 6.96% (of total recovery)
Disk, small objects
=== Summary ===
Recovery time: 2.609 s
Masters: 1
Backups: 6
Number of objects: 3812983
Object size: 128.0
=== Coordinator Time ===
Idle: 2562.2 ms / 98.20% of total recovery
In Recovery constructor: 21.1 ms / 0.81% of total recovery
Recovery::start: 20.1 ms / 0.77% of total recovery
=== Master Time ===
Total: 2588.7 ms / 99.21% of total recovery
Inside recoverSegment: 2117.6 ms / 81.16% of total recovery
Approx. CPU: 1716.1 ms / 65.77% of total recovery (above minus BackupManager time)
Verify checksum: 163.2 ms / 6.25% of total recovery
Segment append copy: 138.8 ms / 5.32% of total recovery
Segment append checksum: 109.6 ms / 4.20% of total recovery
Waiting for backups: 516.4 ms / 19.79% of total recovery
Stalled on segment open: 252.2 ms / 9.67% of total recovery
Stalled on segment write: 28.0 ms / 1.07% of total recovery
Stalled on segment read: 236.1 ms / 9.05% of total recovery
Removing tombstones: 194.2 ms / 7.44% of total recovery
Transmitting in transport: 114.6 ms / 4.39% of total recovery
Copying: 114.1 ms / 4.37% of total recovery
=== Backup Time ===
Total: 2587.1 ms avg / stddev 0.9 ms / 99.15% avg of total recovery
Idle: 2412.1 ms avg / stddev 182.6 ms / 92.45% avg of total recovery
sRD: 0.2 ms avg / stddev 0.0 ms / 0.01% avg of total recovery
write: 50.3 ms avg / stddev 12.0 ms / 1.93% avg of total recovery
Filtering segments: 97.0 ms avg / stddev 19.1 ms / 3.72% avg of total recovery
Transmitting in transport: 20.7 ms avg / stddev 4.9 ms / 0.79% avg of total recovery
Copying: 20.6 ms avg / stddev 4.8 ms / 0.79% avg of total recovery
=== Network Utilization ===
Master in: 1.79 Gb/s (overall)
Master out: 1.79 Gb/s (overall)
Backup in: 0.30 Gb/s avg / stddev 0.04 Gb/s (overall)
Backup out: 0.30 Gb/s avg / stddev 0.06 Gb/s (overall)
=== Disk Utilization ===
Overall bandwidth: 75.63 MB/s avg / stddev 10.42 MB/s
Active bandwidth: 107.25 MB/s avg / stddev 5.34 MB/s
Disk active: 70.98% avg / stddev 12.59% (of total recovery)
Reading: 35.43% avg / stddev 9.21% (of total recovery)
Writing: 35.55% avg / stddev 5.48% (of total recovery)