Summary
I'm optimizing a system in which synchronous disk writes are a key factor in performance. My current drives are too slow. I have a single Intel 520 drive that performs well enough (<1ms), but I need five drives. I bought five Intel 530 SSDs in hopes they would perform like the Intel 520, but instead they take 10ms to write a single byte to disk under various versions of Linux. Curiously, if I connect the Intel 530 drives over a USB-to-SATA adapter instead of using SATA directly, they're much faster (~200us per write). What's wrong?
Microbenchmark
Here's my microbenchmark:
I normally run this on ext4. (I've tried it on a raw device as well; see negative results section below).
Machines
These are the machines I've tried. The Intel 530 over SATA takes 10ms per write on each of these, and other SSDs take <1ms per write on each of these.
| name | type | description | model | distro | kernel | SATA | USB | Purchased |
|---|---|---|---|---|---|---|---|---|
| rc66 | server | This is the machine type I'd like to use (we have 80 of them). | Supermicro xx | RHEL6 | 2.6.32 | 3gbps | 2.0 | 2011 |
| rcmonster | server | This is a newer, beefier machine with Intel 520 drives, but also stuck on RHEL6. | Intel xx | RHEL6 | 2.6.32 | 6gbps | 2013 | |
| flygecko | desktop | This is a newer desktop machine with a newer version of Linux. | Supermicro xx? | Arch | 3.12 | 2013 | ||
| x1 | laptop | This is a laptop with a newer version of Linux. | Lenovo Thinkpad X1 Carbon | Debian Jessie | 3.12 | mSATA 6gbps | 3.0 | 2012 |
Disks
These are the disks I've tried:
| model | qty | performance |
|---|---|---|
| Crucial M4 | 160 | |
| Intel 520 | 2 | |
| Intel ?? on flygecko | 1 | |
| Intel 530 | 5 | |
| Intel 530 attached over USB-to-SATA adapter | 1 | |
| Cheap USB thumb drive | 1 |
Negative Results
- The first thing I did was update from the DC22 firmware to the current DC33 firmware. No effect.
- I tried a machine with 6gbps SATA (rcmonster). No effect.
- I tried a machine with a newer kernel (flygecko). No effect.
- I tried disabling APM power saving with hdparm -B. No effect.
- Did I try different NCQ sizes? Doesn't work on some machines.
- I tried changing the I/O scheduler. This shouldn't have an effect since there's only one I/O outstanding at a time. No effect.
- I tried running the benchmark on the raw block device rather than an ext4 partition. This helped but only slightly, reducing latency per write from 10ms to about 9ms.
Additional Questions
- One of the differences between the Intel 520 and Intel 530 is that the 530 does more aggressive power saving. Is there a way to turn that off?
- Are the software paths under Linux for the Intel 520 and the Intel 530 identical?
- Would Intel be willing to trade my five 530s in exchange for five 520s?
blktrace measurements
btt