Benchmarking Ideas

Owned by Ankita Kejriwal
Last updated: Mar 30, 2015

New idea: TOCS-style (done by Henry) timeline for indexed operations

 

  • End-to-end latency measurements
    • End-to-end write latency with x number of indexes, each index entry of size y.
      • Vary from x = 0 to n: Fill up index with a bunch of entries. Then measure the time to write one object. Delete that one. Again, measure time to write that one. Repeat to get mean/variance/min/max; keep y fixed (say 30 B) – basically, write perf as a func of num of indexes to be written.
      • For a given x: Latency as function of index size (as more and more objects are written and indexes get more filled up, increasing time for lookup); keep y fixed (say 30 B)
      • Vary y.
    • End-to-end remove latency: Details same as write.
    • End-to-end lookup+indexedRead latency
      • Details same as write.
      • Varying range of lookup such that different number of objects match.
  • Throughput / bandwidth measurements: Varying number of clients c (0 to n).
    • Writes or reads or removes or mix
    • Same index server vs different index servers
    • One indexlet vs multiple different indexlets
    • One table vs multiple tables
  • Scalability
    • How performance varies as index gets larger and spread across multiple nodes. We can do static partitioning at creation for checking.
  • Nanobenchmarks: (later)
    • Time to insert an entry for:
      • varying current number of nodes in the indexlet tree (mostly a sanity check; slope same, intercept different from end-to-end).
      • varying entry size
      • repeated index entry vs different index entry
    • Time to lookup an entry: Details same as insert.
      • vary range
    • Time to remove an entry: Details same as insert.
    • Break down lookup+indexedRead from end-to-end into the two components
    • Relation between tree fanout and read/write perf
  • Memory footprint
    • We don't want to measure deletion / cleaning – can use from lsm paper. We want to see the overhead for an index. Create a large index, then measure space, divide by num of entries, calculate overhead.
    • Can compare malloc version and ramcloud object alloc version.
  • Recovery times
    • Verify that its the same as recovering table of that size to one recovery master.
  • Compare with other systems
    • Systems:
      • mySQL
      • H-base?
      • Cassandra?
      • Come up with more
    • Benchmarks:
      • See if there are standard benchmarks (or something that one of the above do) that measure indexing perf.
      • Create table, write n million indexed objects with x indexes.
      • Do above, then measure random indexed reads.