...
- To understand performance at one level, you must measure the constituent factors that explain that performance; you should only trust a result once you understand clearly why you are getting that result.
Use your intuition to ask questions, not to answer them
- Never trust a result if it doesn't make intuitive sense. If you see anything that isn't what you expected, measure more in order to understand exactly what is going on and either fix the system or your intuition
- Common mistake: people make up plausible stories to explain a particular result, rather than measuring to see if that is actually happening. Once you get a hypothesis about why something is happening, measure the system to verify whether that is actually happening.
- A common mistake is to measure performance end-to-end for some operation, then conclude based on intuitive arguments that a particular sub-portion accounts for most of the time. Instead measure one level deeper (i.e. time exactly the portion you think is slow, without anything else around it) to make sure your conclusions are correct.
Separate the factors
- If there are 3 interesting design ideas in your system, it is not sufficient to measure the system in aggregate, with all 3 ideas included; it's possible that one of the ideas is actually a bad idea that has made performance worse, but it is compensated by the other ideas. Find a way to measure each of the ideas independently to see how well they work.
...