This page serves to collect ideas about how to measure performance; these might someday form the basis of a paper on this topic.
Never trust a number generated by a computer
- All performance measurements are initially bad: bugs in the measurements, not measuring what you think you are measuring, and bugs in the system
- Biggest danger: initial measurements look good, so you don't bother to figure out whether they are even correct
- If there is anything about a measurement that seems even a little suspicious, distrust everything until you explain it; it's possible that there is an error that has skewed all of your measurements.
Measure one level deeper
- To understand performance at one level, you must measure the constituent factors that explain that performance; you should only trust a result once you understand clearly why you are getting that result
Use your intuition to ask questions, not to answer them
- Never trust a result if it doesn't make intuitive sense. If you see anything that isn't what you expected, measure more in order to understand exactly what is going on and either fix the system or your intuition
- Common mistake: people make up plausible stories to explain a particular result, rather than measuring to see if that is actually happening. Once you get a hypothesis about why something is happening, measure the system to verify whether that is actually happening.
Separate the factors
- If there are 3 interesting design ideas in your system, it is not sufficient to measure the system in aggregate, with all 3 ideas included; it's possible that one of the ideas is actually a bad idea that has made performance worse, but it is compensated by the other ideas. Find a way to measure each of the ideas independently to see how well they work.