...
- I was too scared to edit your wiki page, John, so I'm putting my ideas here. Please integrate them (or not) as you see fit. I'm sure this is worth writing, even if no PC would accept it.
When your graph is so obvious to you that you wonder whether others will find it boring, you're just about done.
Don't just look at averages. Also look at distributions, and check individual samples to make sure there's not a pattern (in RAMCloud, Alex and Mendel found that every other RPC was slower).
Script your entire experiment and graph end-to-end.
Generate your graphs incrementally, so you can see what the entire graph will look like while you're still gathering data. Fill the graph in like a progressively-rendered image, then repeat each data point to add error bars.
Record more data than you plot, so that when you decide you want to graph something else, you may not need to re-run the experiment.
- Find out and show what optimal would be, so we can judge how good your solution is. Ideally, your optimal will be derived from some "fundamental" limit, like the speed of light or the speed of your network.
- Know what would happen if you were to fix your current bottleneck by measuring what the next one would be. If you push your system close to the limit, you may essentially exhaust on multiple resources – the second bottleneck may not be far.
- Never use a pie chart.