Page Comparison

...

Update Tue, 25 Feb 2014 13:12:01 -0800: added "Larger Writes" section.

Update Sat, 01 Mar 2014 19:31:34 -0800: updated "Larger Writes" section with big graph, updated microbenchmark section to link to improved benchmark code.

 Contents: 

Table of Contents
maxLevel 3

 

...

I'm optimizing a system in which synchronous disk writes are a key factor in performance. My current drives are too slow. I have a single Intel 520 drive that performs well enough (<1ms), but I need five drives. I bought five Intel 530 SSDs in hopes they would perform like the Intel 520, but instead they take 10ms to write a single byte to disk under various versions of Linux. Curiously, if I connect the Intel 530 drives over a USB-to-SATA adapter instead of using SATA directly, they're much faster (~200us per write). If I turn off write-caching, on some machines they take 1.6ms per write. What's wrong?

Microbenchmark

Here's the original microbenchmark, which calls write() and fdatasync() on a single byte of data 1000 times: https://gist.github.com/ongardie/9177853 I run this as "time ./bench" and divide the wall time by 1000 to get the approximate average time per write.

A much improved and extended version of the benchmark is at https://github.com/ongardie/diskbenchmark .

I normally run this on ext4. (I've tried it on a raw device as well; see negative results section below).

...

These are the disks I've tried and their performance:

My goal is to get the Intel 530 drives to run fast on rc66 and similar machines.

...

• The first thing I did was update from the DC22 firmware to the current DC33 firmware. No effect.
• I tried a machine with 6gbps SATA (rcmonster). No effect.
• I tried a machine with a newer kernel (flygecko). No effect.
• I tried disabling APM power saving with hdparm -B. No effect.
• I wasn't able to change NCQ sizes with /sys/block/sdc/device, but it's stuck at 1 on flygecko and 31 on rcmonster.
• I tried changing the I/O scheduler. This shouldn't have an effect since there's only one I/O outstanding at a time. No effect.
• I tried running the benchmark on the raw block device rather than an ext4 partition. This helped but only slightly, reducing latency per write from 10ms to about 9ms.
• I tried doing bigger writes to see if they would be faster. See "Larger Writes" section below. Didn't help.

Additional Questions

• One of the differences between the Intel 520 and Intel 530 is that the 530 does more aggressive power saving. Is there a way to turn that off?
• Are the software paths under Linux for the Intel 520 and the Intel 530 identical?
• Would Intel be willing to trade my five 530s in exchange for five 520s or similarly performing drives?

...

Many people have suggested that writing 1 byte is not efficient and writing more bytes should be faster. I tried it both with write-caching on and off using the same string of random a-z characters for each write.

In these tests, no drive is faster at writing more than 1 byte than it is at writing 1 byte. Most drives take about 2-4x as long to write 8KB as they do to write 1 byte. 

Image Added

The writes were done at 0 and 512 byte offsets into the file, and the whole experiment was repeated 5 times. The best time is shown for each point.  Each data point represents only a small number of writes, though, so the error may be high. Also, the "x1" machine is my laptop, so it wasn't always idle.