...
Update Tue, 25 Feb 2014 13:12:01 -0800: added "Larger Writes" section.
Update Sat, 01 Mar 2014 19:31:34 -0800: updated "Larger Writes" section with big graph, updated microbenchmark section to link to improved benchmark code.
Contents:
Table of Contents | ||
---|---|---|
|
...
I'm optimizing a system in which synchronous disk writes are a key factor in performance. My current drives are too slow. I have a single Intel 520 drive that performs well enough (<1ms), but I need five drives. I bought five Intel 530 SSDs in hopes they would perform like the Intel 520, but instead they take 10ms to write a single byte to disk under various versions of Linux. Curiously, if I connect the Intel 530 drives over a USB-to-SATA adapter instead of using SATA directly, they're much faster (~200us per write). If I turn off write-caching, on some machines they take 1.6ms per write. What's wrong?
Microbenchmark
Here's the original microbenchmark, which calls write() and fdatasync() on a single byte of data 1000 times: https://gist.github.com/ongardie/9177853 I run this as "time ./bench" and divide the wall time by 1000 to get the approximate average time per write.
A much improved and extended version of the benchmark is at https://github.com/ongardie/diskbenchmark .
I normally run this on ext4. (I've tried it on a raw device as well; see negative results section below).
...
These are the disks I've tried and their performance:
model | qty | capacitor | performance |
---|---|---|---|
Crucial M4 | 160 | 3.7ms per write on rc66 | |
Intel 320 (SSDSC2CW120A3) | 1 | yes | ~200us per write on rcmonster |
Intel 520 (SSDSC2CW120A3) | 2 | no | <1ms per write on rc66, rcmonster (440us) |
Intel X25-M (SSDSA2M120G2GC) | 1 | 210us per write on flygecko | |
Intel 530 (SSDSC2BW120A4) | 5 | no | 10ms per write on rc66 (9.8ms), rcmonster (10.1ms), and flygecko (9.7ms) |
Intel 530 attached over USB-to-SATA adapter | 1 | no | <1ms per write on rc66, rcmonster, flygecko, and x1 |
SanDisk X100 (SD5SG2128G1052E) | 1 | 830us per write on x1 | |
Cheap USB thumb drive | n | 7.6ms per write on x1 |
My goal is to get the Intel 530 drives to run fast on rc66 and similar machines.
...
Many people have suggested that writing 1 byte is not efficient and writing more bytes should be faster. I tried it both with write-caching on and off using the same string of random a-z characters for each write. The code used in this benchmark is slightly modified to allow specifying the write size (up to 8KB), see: https://gist.github.com/ongardie/9218611 (oops, this code had a bug, it only called posix_fallocate for the first 1000 KBs, so most of the 4KB and 8KB areas were not pre-allocated. I should re-run those).
host | disk | write-caching | 1 byte | 1024 bytes | 4096 bytes | 8192 bytes |
---|---|---|---|---|---|---|
rcmonster | 520 | on | 440us | 570us | 1.0ms | 1.0ms |
rcmonster | 530 | on | 9.7ms | 12.0ms | 19.6ms | 19.8ms |
rc66 | M4 | on | 3.7ms | 5.1ms | 9.2ms | 10.4ms |
rc66 | 530 | on | 9.8ms | 12.5ms | 21.0ms | 21.2ms |
flygecko | X25-M | on | 210us | 370us | 910us | 950us |
flygecko | 530 | on | 9.7ms | 12.1ms | 19.6ms | 19.7ms |
rcmonster | 520 | off | 100us | 160us | 400us | 420us |
rcmonster | 530 | off | 10.5ms | 17.5ms | 55.9ms | 68.9ms |
rc66 | M4 | off | 1.5ms | 2.6ms | 5.6ms | 7.2ms |
rc66 | 530 | off | 1.9ms | 3.0ms | 6.9ms | 8.6ms |
flygecko | X25-M | off | 1.0ms | 1.7ms | 3.5ms | 3.7ms |
flygecko | 530 | off | 1.7ms | 2.4ms | 4.9ms | 5.0ms |
In these tests, no drive is faster at writing more than 1 byte than it is at writing 1 byte. Most drives take about 2-4x as long to write 8KB as they do to write 1 byte.
The writes were done at 0 and 512 byte offsets into the file, and the whole experiment was repeated 5 times. The best time is shown for each point. Each data point represents only a small number of writes, though, so the error may be high. Also, the "x1" machine is my laptop, so it wasn't always idle.