/
Benchmarking Ideas

Benchmarking Ideas

Mar 30, 2015

New idea: TOCS-style (done by Henry) timeline for indexed operations

 

  • End-to-end latency measurements

    • End-to-end write latency with x number of indexes, each index entry of size y.

      • Vary from x = 0 to n: Fill up index with a bunch of entries. Then measure the time to write one object. Delete that one. Again, measure time to write that one. Repeat to get mean/variance/min/max; keep y fixed (say 30 B) – basically, write perf as a func of num of indexes to be written.

      • For a given x: Latency as function of index size (as more and more objects are written and indexes get more filled up, increasing time for lookup); keep y fixed (say 30 B)

      • Vary y.

    • End-to-end remove latency: Details same as write.

    • End-to-end lookup+indexedRead latency

      • Details same as write.

      • Varying range of lookup such that different number of objects match.

  • Throughput / bandwidth measurements: Varying number of clients c (0 to n).

    • Writes or reads or removes or mix

    • Same index server vs different index servers

    • One indexlet vs multiple different indexlets

    • One table vs multiple tables

  • Scalability

    • How performance varies as index gets larger and spread across multiple nodes. We can do static partitioning at creation for checking.

  • Nanobenchmarks: (later)

    • Time to insert an entry for:

      • varying current number of nodes in the indexlet tree (mostly a sanity check; slope same, intercept different from end-to-end).

      • varying entry size

      • repeated index entry vs different index entry

    • Time to lookup an entry: Details same as insert.

      • vary range

    • Time to remove an entry: Details same as insert.

    • Break down lookup+indexedRead from end-to-end into the two components

    • Relation between tree fanout and read/write perf

  • Memory footprint

    • We don't want to measure deletion / cleaning – can use from lsm paper. We want to see the overhead for an index. Create a large index, then measure space, divide by num of entries, calculate overhead.

    • Can compare malloc version and ramcloud object alloc version.

  • Recovery times

    • Verify that its the same as recovering table of that size to one recovery master.

  • Compare with other systems

    • Systems:

      • mySQL

      • H-base?

      • Cassandra?

      • Come up with more

    • Benchmarks:

      • See if there are standard benchmarks (or something that one of the above do) that measure indexing perf.

      • Create table, write n million indexed objects with x indexes.

      • Do above, then measure random indexed reads.