SSD Latency Experiments

SSD Latency Experiments

Foreword

These experiments explore the write latencies of several SSD's in our cluster. They are conducted in a similar style to Diego's Intel 530 Performance, but were performed at a later time with a standard unix tool (dd) instead of a custom C program. These tests also extend the original experiments by looking into the effects of different file systems vs. raw drive performance. 

Summary

  • Write Cache: Disabling the write cache improves latency, but sacrifices bandwidth in most cases.

  • Drives: Intel drives have the lowest latencies (<1ms) with the fastest configuration being in RAW with Write Cached Disabled (48µs write latency!!)

    • Note: The intel drive also saw a bandwidth increase with write cache disabled

  • File Systems: XFS has lower write latencies than EXT4 (1-2x faster).

  • TRIM: TRIM makes minimal difference in the SSD write latencies (although it may be because the drive is completely empty at the beginning of the benchmark)

Setup

We tested 4 SSDs and 1 HDD (for comparison) while varying the drive's file system, TRIM option, and write cache. Each configuration was benchmarked with the following dd command and the output in seconds is interpreted as millisecond latencies:

 

dd if=/dev/zero bs=512 count=1000 oflag=direct,sync of=$TARGET

 

where $TARGET specfies the device (i.e. /dev/sdb if raw) or a file on the device (i.e. /mnt/sdb/testFile).

 

The drive's file system, write cache, and TRIM options were varied with the following table of commands

Make filesystem ext4

mkfs.ext4 $TARGET

Make filesystem xfs

mkfs.xfs $TARGET

Mount with trim

mount -o discard $TARGET $TARGET_DIR

Mount without trim

mount $TARGET $TARGET_DIR

Clear File system

wipefs -t <prevFS> $TARGET

Disable write cache

hdparm -W 0 $TARGET

Enable write cache

hdparm -W 1 $TARGET

 

The entire experiment can be run with the following script

WARNING: THE DRIVE IS SPECIFIED IN THE SCRIPT, CHANGE IT TO THE CORRECT ONE OR /dev/sdb WILL BE DESTROYED

Hardware

Machine

Type

Model

Linux Distro/Version

SATA

Machine

Type

Model

Linux Distro/Version

SATA

rc01/rc21

Server

Colfax CX1180-X4 / Supermicro 5016TI-TF

Debian 8.3, Kernel 3.16.7

3Gbps

rcmonster

Server

Colfax CX1265i-X5

"

6Gbps

rcmaster

Server

Colfax 6016T-NTF

"

3Gbps

Drive Name

Firmware

Capacity

Installed On

Drive Name

Firmware

Capacity

Installed On

Samsung 850 PRO

EXM02B6Q

256GB

rc01

Crucial M4
CT128M4SSD2

0309

128GB

rc21

Intel 5120
SSDSC2CW120A3 

400i

120GB

rcmonster

Crucial M550
Crucial_CT1024M550SSD1

MU01

1TB

rcmonster

Western Digital 7200RPM
WD1002FBYS-02A6B0

03.00C06

1TB

rcmaster

 

Results

Raw

Drive Name/Model

Firmware

Write Cache

Filesystem

TRIM

512B Write Latency (ms)

Drive Name/Model

Firmware

Write Cache

Filesystem

TRIM

512B Write Latency (ms)

Samsung 850 PRO 256 GB on rc01

EXM02B6Q

Enabled

RAW

-

3.10

 

 

 

Ext4

Yes

~8

 

 

 

Ext4

No

~8

 

 

 

XFS

Yes

6.2

 

 

 

XFS

No

6.2

 

 

Disabled

RAW

n/a

2.9

 

 

 

Ext4

Yes

10.5

 

 

 

Ext4

No

10.5

 

 

 

XFS

Yes

6.05

 

 

 

XFS

No

6.06

Crucial 128GB M4 - CT128M4SSD2 on rc21

0309

Enabled

RAW

-

3.76

 

 

 

Ext4

Yes

7.8

 

 

 

 

No

8.2

 

 

 

XFS

Yes

5.7

 

 

 

 

No

5.4

 

 

Disabled

RAW

-

1.3

 

 

 

Ext4

Yes

4.3

 

 

 

 

No

4.5

 

 

 

XFS

Yes

2.8

 

 

 

 

No

2.8

Intel 520 120GB SSDSC2CW120A3 on rcmonster

400i

Enabled

RAW

-

0.44

 

 

 

Ext4

Yes

0.96

 

 

 

 

No

0.96

 

 

 

XFS

Yes

0.88

 

 

 

 

No

0.88

 

 

Disabled

RAW

-

0.048

 

 

 

Ext4

Yes

0.25

 

 

 

 

No

0.24

 

 

 

XFS

Yes

0.13

 

 

 

 

No

0.13

Crucial M550 1TB

Crucial_CT1024M550SSD1 on rcmonster

MU01

Enabled

RAW

-

4.2

 

 

 

Ext4

Yes

8.7

 

 

 

 

No

8.7

 

 

 

XFS

Yes

8.6

 

 

 

 

No

8.6

 

 

Disabled

RAW

-

2.0

 

 

 

Ext4

Yes

6.1