These experiments explore the write latencies of several SSD's in our cluster. They are conducted in a similar style to Diego's Intel 530 Performance, but were performed at a later time with a standard unix tool (dd) instead of a custom C program. These tests also extend the original experiments by looking into the effects of different file systems vs. raw drive performance.
We tested 4 SSDs and 1 HDD (for comparison) while varying the drive's file system, TRIM option, and write cache. Each configuration was benchmarked with the following dd command and the output in seconds is interpreted as millisecond latencies:
dd if=/dev/zero bs=512 count=1000 oflag=direct,sync of=$TARGET |
where $TARGET specfies the device (i.e. /dev/sdb if raw) or a file on the device (i.e. /mnt/sdb/testFile).
The drive's file system, write cache, and TRIM options were varied with the following table of commands
Make filesystem ext4 | mkfs.ext4 $TARGET |
Make filesystem xfs | mkfs.xfs $TARGET |
Mount with trim | mount -o discard $TARGET $TARGET_DIR |
Mount without trim | mount $TARGET $TARGET_DIR |
Clear File system | wipefs -t <prevFS> $TARGET |
Disable write cache | hdparm -W 0 $TARGET |
Enable write cache | hdparm -W 1 $TARGET |
The entire experiment can be run with the following script
WARNING: THE DRIVE IS SPECIFIED IN THE SCRIPT, CHANGE IT TO THE CORRECT ONE OR /dev/sdb WILL BE DESTROYED
Machine | Type | Model | Linux Distro/Version | SATA |
---|---|---|---|---|
rc01/rc21 | Server | Colfax CX1180-X4 / Supermicro 5016TI-TF | Debian 8.3, Kernel 3.16.7 | 3Gbps |
rcmonster | Server | Colfax CX1265i-X5 | " | 6Gbps |
rcmaster | Server | Colfax 6016T-NTF | " | 3Gbps |
Drive Name | Firmware | Capacity | Installed On |
---|---|---|---|
Samsung 850 PRO | EXM02B6Q | 256GB | rc01 |
Crucial M4 CT128M4SSD2 | 0309 | 128GB | rc21 |
Intel 5120 SSDSC2CW120A3 | 400i | 120GB | rcmonster |
Crucial M550 Crucial_CT1024M550SSD1 | MU01 | 1TB | rcmonster |
Western Digital 7200RPM WD1002FBYS-02A6B0 | 03.00C06 | 1TB | rcmaster |
Drive Name/Model | Firmware | Write Cache | Filesystem | TRIM | 512B Write Latency (ms) |
---|---|---|---|---|---|
Samsung 850 PRO 256 GB on rc01 | EXM02B6Q | Enabled | RAW | - | 3.10 |
Ext4 | Yes | ~8 | |||
Ext4 | No | ~8 | |||
XFS | Yes | 6.2 | |||
XFS | No | 6.2 | |||
Disabled | RAW | n/a | 2.9 | ||
Ext4 | Yes | 10.5 | |||
Ext4 | No | 10.5 | |||
XFS | Yes | 6.05 | |||
XFS | No | 6.06 | |||
Crucial 128GB M4 - CT128M4SSD2 on rc21 | 0309 | Enabled | RAW | - | 3.76 |
Ext4 | Yes | 7.8 | |||
No | 8.2 | ||||
XFS | Yes | 5.7 | |||
No | 5.4 | ||||
Disabled | RAW | - | 1.3 | ||
Ext4 | Yes | 4.3 | |||
No | 4.5 | ||||
XFS | Yes | 2.8 | |||
No | 2.8 | ||||
Intel 520 120GB SSDSC2CW120A3 on rcmonster | 400i | Enabled | RAW | - | 0.44 |
Ext4 | Yes | 0.96 | |||
No | 0.96 | ||||
XFS | Yes | 0.88 | |||
No | 0.88 | ||||
Disabled | RAW | - | 0.048 | ||
Ext4 | Yes | 0.25 | |||
No | 0.24 | ||||
XFS | Yes | 0.13 | |||
No | 0.13 | ||||
Crucial M550 1TB Crucial_CT1024M550SSD1 on rcmonster | MU01 | Enabled | RAW | - | 4.2 |
Ext4 | Yes | 8.7 | |||
No | 8.7 | ||||
XFS | Yes | 8.6 | |||
No | 8.6 | ||||
Disabled | RAW | - | 2.0 | ||
Ext4 | Yes | 6.1 | |||
No | 6.1 | ||||
XFS | Yes | 4.0 | |||
No | 4.0 | ||||
WD 7200RPM HDD WD1002FBYS-02A6B0 | 03.00C06 | Enabled | RAW | - | 8.5 |
Ext4 | Yes | 40.1 | |||
No | 41.3 | ||||
XFS | Yes | 31 | |||
No | 31 | ||||
Disabled | RAW | - | 8.5 | ||
Ext4 | Yes | 42.5 | |||
No | 41.2 | ||||
XFS | Yes | 30 | |||
No | 30 |
Here is a pivot table summarizing the results. Since the TRIM command didn't seem to make much of a difference, they are not included in the results below.
All times are in milliseconds and +WC means with Write Cached Enabled, -WC means with Write Cached Disabled.
Drive | RAW + WC | Ext4 + WC | XFS + WC | RAW - WC | Ext4 - WC | XFS - WC |
---|---|---|---|---|---|---|
Samsung 850 PRO 256 GB | 3.1 | 8 | 6.2 | 2.9 | 10.5 | 6.05 |
Crucial M4 128GB on rc21 | 3.76 | 8 | 5.5 | 1.3 | 4.4 | 2.8 |
Intel 520 120GB on rcmonster | 0.44 | 0.96 | 0.88 | 0.048 | 0.25 | 0.13 |
Crucial 550 1TB on rcmonster | 4.2 | 8.7 | 8.6 | 2.0 | 6.1 | 4.0 |
Western Digital 7200RPM on rcmaster | 8.5 | 40.1 | 31 | 8.5 | 42 | 30
|
Final comparison between the bandwidth + latency of enabling/disbling the Write Cache on a RAW device
The Maximum Bandwidth calculation was generated from:
dd if=/dev/zero bs=128M count=10 oflag=direct,sync of=$DRIVE |
Drive | Write Cache | Max Bandwidth (MB/s) | Bandwidth Degradation | Small Write Latency (ms) | Latency Improvement |
---|---|---|---|---|---|
Samsung 850 PRO 256 GB on rc01 | Enabled | 234 | 3.1 | ||
Disabled | 61.4 | 3.8x | 2.9 | 1.06x | |
Crucial M4 on rc21 | Enabled | 175 | 3.76 | ||
Disabled | 135 | 1.3x | 1.3 | 2.9x | |
Intel 520 on rcmonster | Enabled | 251 | 0.44 | ||
Disabled | 257 | 0.98x (got faster) | 0.048 | 9.2x | |
Crucial M550 on rcmonster | Enabled | 253 | 4.2 | ||
Disabled | 228 | 1.1x | 2.0 | 2.1x | |
Western Digital 7200RPM on rcmaster | Enabled | 107 | 8.5 | ||
Disabled | 39.7 | 2.7x | 8.5 | 1.0x |