ZooKeeper Performance

Owned by Diego Ongaro
Last updated: Oct 29, 2010Version comment
2 min read

Setup

  • All machines were dual-core Xeon 3060 machines
  • All machines were hooked up to a single gigabit switch
  • ZooKeeper 3.3.0 was used for both the clients and servers
  • 3 machines made up the ZooKeeper cluster
  • The benchmark utility is a Java client that will connect to a specific ZooKeeper server (always the same) and do n back-to-back reads or writes of a single object of a given length

Latency with benchmark client running on the ZooKeeper server

Observations during reads:
  • ZooKeeper takes about 20-25% of one core
  • The benchmark client uses about 10% of the other
  • The kernel uses about 20-25% of each core
  • The userspace programs spend most of their time waiting on a futex.
  • The replicas are idle.
Observations during writes
  • ZooKeeper takes about 50% of the CPU
  • The kernel uses about 7% of the CPU
  • The first replica uses about 60-80% of its CPU on ZooKeeper and about 12% on the kernel
  • The second replica uses about 40-60% of its CPU on ZooKeeper and about 12% on the kernel

Latency with benchmark client running on a separate machine

Throughput

The maximum read throughput I was able to observe was ~50,000 512Byte-read operations per second. This was using 8 processes (without my own Java threads) on each of 4 client machines. This does not entirely saturate the cores on the server – they are each between 5% and 30% idle at any given time.

The maximum read throughput I was able to observe from a single client machine was only ~29K 512Byte-read operations per second using 8 processes (each of 4 Java threads). I don't understand the behavior of the client library (which seems to spawn about 11 threads of its own), so I can't really explain this.

1 client machine: 5.6Kops/s (512 byte reads)

2 client machines: 10.4Kops/s (512 byte reads)

3 client machies: 15.7Kops/s (512 byte reads)

4 client machines: 20.0Kops/s (512 byte reads)