Foreword

These experiments explore the write latencies of several SSD's in our cluster. They are conducted in a similar style to Diego's Intel 530 Performance, but were performed at a later time with a standard unix tool (dd) instead of a custom C program. These tests also extend the original experiments by looking into the effects of different file systems vs. raw drive performance.

Summary

Write Cache: Disabling the write cache improves latency, but sacrifices bandwidth in most cases.
Drives: Intel drives have the lowest latencies (<1ms) with the fastest configuration being in RAW with Write Cached Disabled (48µs write latency!!)
- Note: The intel drive also saw a bandwidth increase with write cache disabled
File Systems: XFS has lower write latencies than EXT4 (1-2x faster).
TRIM: ~~TRIM makes minimal difference in the SSD write latencies~~ (although it may be because the drive is completely empty at the beginning of the benchmark)

Setup

We tested 4 SSDs and 1 HDD (for comparison) while varying the drive's file system, TRIM option, and write cache. Each configuration was benchmarked with the following dd command and the output in seconds is interpreted as millisecond latencies:

dd if=/dev/zero bs=512 count=1000 oflag=direct,sync of=$TARGET

where $TARGET specfies the device (i.e. /dev/sdb if raw) or a file on the device (i.e. /mnt/sdb/testFile).

The drive's file system, write cache, and TRIM options were varied with the following table of commands

Make filesystem ext4	mkfs.ext4 $TARGET
Make filesystem xfs	mkfs.xfs $TARGET
Mount with trim	mount -o discard $TARGET $TARGET_DIR
Mount without trim	mount $TARGET $TARGET_DIR
Clear File system	wipefs -t <prevFS> $TARGET
Disable write cache	hdparm -W 0 $TARGET
Enable write cache	hdparm -W 1 $TARGET

The entire experiment can be run with the following script

WARNING: THE DRIVE IS SPECIFIED IN THE SCRIPT, CHANGE IT TO THE CORRECT ONE OR /dev/sdb WILL BE DESTROYED

benchmark.sh

Hardware

Machine	Type	Model	Linux Distro/Version	SATA
rc01/rc21	Server	Colfax CX1180-X4 / Supermicro 5016TI-TF	Debian 8.3, Kernel 3.16.7	3Gbps
rcmonster	Server	Colfax CX1265i-X5	"	6Gbps
rcmaster	Server	Colfax 6016T-NTF	"	3Gbps

Drive Name	Firmware	Capacity	Installed On
Samsung 850 PRO	EXM02B6Q	256GB	rc01
Crucial M4 CT128M4SSD2	0309	128GB	rc21
Intel 5120 SSDSC2CW120A3	400i	120GB	rcmonster
Crucial M550 Crucial_CT1024M550SSD1	MU01	1TB	rcmonster
Western Digital 7200RPM WD1002FBYS-02A6B0	03.00C06	1TB	rcmaster

Results

Raw

Drive Name/Model	Firmware	Write Cache	Filesystem	TRIM	512B Write Latency (ms)
Samsung 850 PRO 256 GB on rc01	EXM02B6Q	Enabled	RAW	-	3.10
			Ext4	Yes	~8
			Ext4	No	~8
			XFS	Yes	6.2
			XFS	No	6.2
		Disabled	RAW	n/a	2.9
			Ext4	Yes	10.5
			Ext4	No	10.5
			XFS	Yes	6.05
			XFS	No	6.06
Crucial 128GB M4 - CT128M4SSD2 on rc21	0309	Enabled	RAW	-	3.76
			Ext4	Yes	7.8
				No	8.2
			XFS	Yes	5.7
				No	5.4
		Disabled	RAW	-	1.3
			Ext4	Yes	4.3
				No	4.5
			XFS	Yes	2.8
				No	2.8
Intel 520 120GB SSDSC2CW120A3 on rcmonster	400i	Enabled	RAW	-	0.44
			Ext4	Yes	0.96
				No	0.96
			XFS	Yes	0.88
				No	0.88
		Disabled	RAW	-	0.048
			Ext4	Yes	0.25
				No	0.24
			XFS	Yes	0.13
				No	0.13
Crucial M550 1TB Crucial_CT1024M550SSD1 on rcmonster	MU01	Enabled	RAW	-	4.2
			Ext4	Yes	8.7
				No	8.7
			XFS	Yes	8.6
				No	8.6
		Disabled	RAW	-	2.0
			Ext4	Yes	6.1
				No	6.1
			XFS	Yes	4.0
				No	4.0
WD 7200RPM HDD WD1002FBYS-02A6B0	03.00C06	Enabled	RAW	-	8.5
			Ext4	Yes	40.1
				No	41.3
			XFS	Yes	31
				No	31
		Disabled	RAW	-	8.5
			Ext4	Yes	42.5
				No	41.2
			XFS	Yes	30
				No	30

Pivot Table

Here is a pivot table summarizing the results. Since the TRIM command didn't seem to make much of a difference, they are not included in the results below.

All times are in milliseconds and +WC means with Write Cached Enabled, -WC means with Write Cached Disabled.

Drive	RAW + WC	Ext4 + WC	XFS + WC	RAW - WC	Ext4 - WC	XFS - WC
Samsung 850 PRO 256 GB on rc01	3.1	8	6.2	2.9	10.5	6.05
Crucial M4 128GB on rc21	3.76	8	5.5	1.3	4.4	2.8
Intel 520 120GB on rcmonster	0.44	0.96	0.88	0.048	0.25	0.13
Crucial 550 1TB on rcmonster	4.2	8.7	8.6	2.0	6.1	4.0
Western Digital 7200RPM on rcmaster	8.5	40.1	31	8.5	42	30

Write Cache Effects

Final comparison between the bandwidth + latency of enabling/disbling the Write Cache on a RAW device

The Maximum Bandwidth calculation was generated from:

dd if=/dev/zero bs=128M count=10 oflag=direct,sync of=$DRIVE

Drive	Write Cache	Max Bandwidth (MB/s)	Bandwidth Degradation	Small Write Latency (ms)	Latency Improvement
Samsung 850 PRO 256 GB on rc01	Enabled	234		3.1
	Disabled	61.4	3.8x	2.9	1.06x
Crucial M4 on rc21	Enabled	175		3.76
	Disabled	135	1.3x	1.3	2.9x
Intel 520 on rcmonster	Enabled	251		0.44
	Disabled	257	0.98x (got faster)	0.048	9.2x
Crucial M550 on rcmonster	Enabled	253		4.2
	Disabled	228	1.1x	2.0	2.1x
Western Digital 7200RPM on rcmaster	Enabled	107		8.5
	Disabled	39.7	2.7x	8.5	1.0x

Browser not supported