Purpose
We are evaluating four platforms for our proposed 40-node RAMCloud cluster. Each is very similar in price (approx. $2000-2500/node for 24-32GB of RAM, cpu, and a disk). Three machines are from SuperMicro. The fourth is a Dell box with nearly identical configuration to the Xeon E5620-based SuperMicro.
Machines
All machines are server-class hardware (i.e. support ECC) in 1U units. In the Xeon cases, we have the option of buying twin servers (two independent boards in one case) from SuperMicro. We're looking at the following configurations:
- Xeon E5620 (4 core / 8 thread Westmere at 2.4GHz. Dual socket server. Max. 192GB RAM)
- Xeon X3470 (4 core / 8 thread Nehalem at 2.93GHz. Single socket server. Max. 32GB RAM)
- Opteron 6134 (8 core / 8 thread at 2.3GHz. Single socket server. Max. 128GB RAM)
Misc. Notes:
- The E5620 and AMD 6134 have on-die memory controllers, but they do not have on-die PCIe. Both support 1GB superpages. The AMD chips do not have the CRC32 instruction yet.
- The X3470 has both on-die memory and PCIe controllers. However, it is not a westmere chip, so it doesn't support 1GB superpages. It should have the CRC32 instruction.
CPUs
|
Xeon E5620 |
Opteron 6134 |
Xeon X3470 |
---|---|---|---|
Clock |
2.4GHz |
2.3GHz |
2.93GHz |
Max Turbo |
2.66GHz |
N/A |
3.6GHz |
# Cores |
4 |
8 |
4 |
# Threads |
8 |
8 |
8 |
L1 Cache |
64KB / core |
128KB / core |
64KB / core |
L2 Cache |
256KB / core |
512KB / core |
256KB / core |
L3 Cache |
12MB shared |
12MB shared |
8MB shared |
On-die Memory Controller |
Yes |
Yes |
Yes |
On-die PCIe Controller |
No |
No |
Yes |
Max. CPU Sockets / Motherboard |
2 |
? (>= 4) |
1 |
Max. Memory Channels |
3 |
4 |
2 |
Max. Memory Clock |
1066MHz |
1333MHz |
1333MHz (NB: drops to 800MHz when >= 24GB RAM installed) |
Max. Memory Supported |
288GB |
? (>=128GB) |
32GB |
1GB Superpages |
Yes |
Yes |
No |
CRC32 Instruction |
Yes |
No |
Yes |
|
|
|
|
Systems
|
Dell R410 |
SuperMicro 6016T-NTF (Xeon E5620) |
SuperMicro 1012G-MTF (Opteron 6134) |
SuperMicro 5016I-M6F |
---|---|---|---|---|
Motherboard |
? |
X8DTU-F |
H8SGL-F |
X8SI6-F |
Chipset |
Intel 5520 (Tylersburg) + ICH10R |
Intel 5520 (Tylersburg) + ICH10R |
AMD SR5650 + SP5100 |
Intel 3420 (Ibex Peak) + 3420 PCH |
# CPU Sockets |
2 |
2 |
1 |
1 |
# DIMM Slots |
8 (4 / socket) |
12 (6 / socket) |
8 |
6 |
Max. Memory |
128GB |
192GB |
128GB |
32GB (board supports 48, cpu limits to 32) |
# SATA Ports |
6 (?) |
6 |
6 |
6 |
# Hard Drives |
4 x 3.5" |
4 x 3.5" |
4x 3.5" |
4x 3.5" |
# PCIe slots (electrical width) |
1 (x16) |
2 (x8) |
1 (x16) |
1 (x8) |
Notes: |
|
|
|
|
Evaluation Criteria
RAMCloud is concerned mostly with scale and latency. Since we have a limited budget and limited space, we cannot scale to a huge number of machines or load them with expensive high-density dimms (sweet spot looks like 4 or 8GB dimms). As such, most of our evaluation focuses on latency - infiniband network latency, ramcloud micro-benchmarks, and simple end-to-end ramcloud benchmarks. We will also consider miscellaneous features like number of pcie slots, dimm slots, cpu cores, etc.
RAMCloud Numbers:
Benchmark |
Xeon E5620 |
Opteron 6134 |
Xeon X3470 |
---|---|---|---|
Rabinpoly (min, max) |
206MB/s, 381MB/s |
57MB/s, 438MB/s |
|
VMAC (min, max) |
60MB/s, 2630MB/s |
45MB/s, 2294MB/s |
|
HashTableBench (-h 128) |
84ns lookup, 98ns replace |
147ns lookup, 133ns replace |
|
1000-byte Bench (infiniband, localhost, -m 128 server) |
7.6usec +/- 0.1usec RTT (272.76ns on server) |
7.1usec +/- 0.3usec RTT (338.81ns on server) |
|
100-byte Bench (infiniband, localhost, -m 128 server) |
6.6usec RTT (283.43ns server) |
6.2usec (348ns on server) |
|
Infiniband Numbers (all are one-way, i.e. not RTT values, in microseconds):
Benchmark |
Xeon E5620 |
Opteron 6134 |
Xeon 3470 |
Notes |
---|---|---|---|---|
ib_write_lat -s 128 localhost |
1.40 |
1.16 |
|
|
ib_write_lat -s 1024 localhost |
3.33 |
2.92 |
|
|
ib_send_lat -s 128 localhost |
1.59 |
1.42 |
|
Sends are inlined with the WQE up to 400 bytes |
ib_send_lat -s 1024 localhost |
3.44 |
2.99 |
|
|
ib_send_lat -s 128 -I 0 |
2.38 |
2.04 |
|
-I 0 disables inlining data in the WQE |
ib_send_lat -s 128 between E5620 and AMD 6134 |
1.58 |
1.58 |
N/A |
With 2 AMD machines, est. is ~3.02us RTT |
ib_send_lat -s 128 between two E5620 machines |
1.65 |
N/A |
N/A |
~3.3us RTT |
ib_send_lat -s 128 -I 0 between E5620 and AMD 6134 |
2.30 |
2.30 |
N/A |
With 2 AMD machines, est. is ~4.24us RTT |
ib_send_lat -s 128 -I 0 between two E5620 machines |
2.48 |
N/A |
N/A |
~4.96us RTT |
LM Bench Numbers
Benchmark |
Xeon E5620 |
Opteron 6134 |
Xeon 3470 |
---|---|---|---|
mem_bw (read / write / copy) |
7800 / 3267 / 2095 MB/s |
8300 / 7200 / 3870 MB/s |
|
lat_mem_rd -N 1 -P 1 256M 512 (l1 / l2 / l3 / ram) |
1.5 / 3.8 / 18 / 63 ns |
1.3 / 6 / 19 / 109 ns |
|
|
|
|
|