SSD Latency Experiments

Foreword

These experiments explore the write latencies of several SSD's in our cluster. They are conducted in a similar style to Diego's Intel 530 Performance, but were performed at a later time with a standard unix tool (dd) instead of a custom C program. These tests also extend the original experiments by looking into the effects of different file systems vs. raw drive performance. 

Summary

  • Write Cache: Disabling the write cache improves latency, but sacrifices bandwidth in most cases.
  • Drives: Intel drives have the lowest latencies (<1ms) with the fastest configuration being in RAW with Write Cached Disabled (48µs write latency!!)
    • Note: The intel drive also saw a bandwidth increase with write cache disabled
  • File Systems: XFS has lower write latencies than EXT4 (1-2x faster).
  • TRIM: TRIM makes minimal difference in the SSD write latencies (although it may be because the drive is completely empty at the beginning of the benchmark)

Setup

We tested 4 SSDs and 1 HDD (for comparison) while varying the drive's file system, TRIM option, and write cache. Each configuration was benchmarked with the following dd command and the output in seconds is interpreted as millisecond latencies:

 

dd if=/dev/zero bs=512 count=1000 oflag=direct,sync of=$TARGET

 

where $TARGET specfies the device (i.e. /dev/sdb if raw) or a file on the device (i.e. /mnt/sdb/testFile).

 

The drive's file system, write cache, and TRIM options were varied with the following table of commands

Make filesystem ext4mkfs.ext4 $TARGET
Make filesystem xfsmkfs.xfs $TARGET
Mount with trimmount -o discard $TARGET $TARGET_DIR
Mount without trimmount $TARGET $TARGET_DIR
Clear File systemwipefs -t <prevFS> $TARGET
Disable write cachehdparm -W 0 $TARGET
Enable write cachehdparm -W 1 $TARGET

 

The entire experiment can be run with the following script

WARNING: THE DRIVE IS SPECIFIED IN THE SCRIPT, CHANGE IT TO THE CORRECT ONE OR /dev/sdb WILL BE DESTROYED

benchmark.sh

Hardware

MachineTypeModelLinux Distro/VersionSATA
rc01/rc21ServerColfax CX1180-X4 / Supermicro 5016TI-TFDebian 8.3, Kernel 3.16.73Gbps
rcmonsterServerColfax CX1265i-X5"6Gbps
rcmasterServerColfax 6016T-NTF"3Gbps
Drive NameFirmwareCapacityInstalled On
Samsung 850 PROEXM02B6Q256GBrc01
Crucial M4
CT128M4SSD2
0309128GBrc21
Intel 5120
SSDSC2CW120A3 
400i120GBrcmonster
Crucial M550
Crucial_CT1024M550SSD1
MU011TBrcmonster
Western Digital 7200RPM
WD1002FBYS-02A6B0
03.00C061TBrcmaster

 

Results

Raw

Drive Name/ModelFirmwareWrite CacheFilesystemTRIM512B Write Latency (ms)
Samsung 850 PRO 256 GB on rc01

EXM02B6Q

EnabledRAW-3.10
   Ext4Yes~8
   Ext4No~8
   XFSYes6.2
   XFSNo6.2
  DisabledRAWn/a2.9
   Ext4Yes10.5
   Ext4No10.5
   XFSYes6.05
   XFSNo6.06
Crucial 128GB M4 - CT128M4SSD2 on rc21

0309

EnabledRAW-3.76
   Ext4Yes7.8
    No8.2
   XFSYes5.7
    No5.4
  DisabledRAW-1.3
   Ext4Yes4.3
    No4.5
   XFSYes2.8
    No2.8

Intel 520 120GB SSDSC2CW120A3 on rcmonster

400i

EnabledRAW-0.44
   Ext4Yes0.96
    No0.96
   XFSYes0.88
    No0.88
  DisabledRAW-0.048
   Ext4Yes0.25
    No0.24
   XFSYes0.13
    No0.13
Crucial M550 1TB

Crucial_CT1024M550SSD1 on rcmonster

MU01

EnabledRAW-4.2
   Ext4Yes8.7
    No8.7
   XFSYes8.6
    No8.6
  DisabledRAW-2.0
   Ext4Yes6.1
    No6.1
   XFSYes4.0
    No4.0
WD 7200RPM HDD
WD1002FBYS-02A6B0 

03.00C06

EnabledRAW-8.5
   Ext4Yes40.1
    No41.3
   XFSYes31
    No31
  DisabledRAW-8.5
   Ext4Yes42.5
    No41.2
   XFSYes30
    No30

 

Pivot Table

Here is a pivot table summarizing the results. Since the TRIM command didn't seem to make much of a difference, they are not included in the results below.

All times are in milliseconds and +WC means with Write Cached Enabled, -WC means with Write Cached Disabled.

DriveRAW + WCExt4 + WCXFS + WCRAW - WCExt4 - WCXFS - WC

Samsung 850 PRO 256 GB
on rc01 

3.186.22.910.56.05
Crucial M4 128GB
on rc21 
3.7685.51.34.42.8
Intel 520 120GB
on rcmonster 
0.440.960.880.0480.250.13
Crucial 550 1TB
on rcmonster 
4.28.78.62.06.14.0
Western Digital 7200RPM
on rcmaster 
8.540.1318.542

30

 

 

Write Cache Effects

Final comparison between the bandwidth + latency of enabling/disbling the Write Cache on a RAW device

 

The Maximum Bandwidth calculation was generated from: 

dd if=/dev/zero bs=128M count=10 oflag=direct,sync of=$DRIVE
DriveWrite CacheMax Bandwidth (MB/s)Bandwidth DegradationSmall Write
Latency (ms) 
Latency Improvement
Samsung 850 PRO 256 GB on rc01Enabled234 3.1 
 Disabled61.43.8x2.91.06x
Crucial M4 on rc21Enabled175 3.76 
 Disabled1351.3x1.32.9x
Intel 520 on rcmonsterEnabled251 0.44 
 Disabled2570.98x (got faster)0.0489.2x
Crucial M550 on rcmonsterEnabled253 4.2 
 Disabled2281.1x2.02.1x
Western Digital 7200RPM on rcmasterEnabled107 8.5 
 Disabled39.72.7x8.51.0x