Evaluating Metrics

March 31, 2021

Good Continous Delivery podcast where they talked about how to evaluate metrics and their suitability to the situation.

  • If you don't know how to respond to a metric, don't measure it. This is a quote from How to Measure Anything) and is a clever way of asking why we need to measure something. If we're not willing to act when we see a metric, what's the point?
  • If there is additional work needed to create the metric and it isn't a natural byproduct of the work, then we have to ask the question whether the effort of creating an observable metric is worth the opportunity cost?
  • Is the metic at the right level of granularity? If a team metric is shown to management or vice-versa, what effects will it have? Will it send the wrong signals which can snowball into waste?
  • Is the metric measuring output or outcome? Pure ouput metrics like Line of Code are not helpful. Metrics such as subscribers gained or software quality which are outcome-oriented are better.
  • Is the metric being observed producing the right behavours? If people are incentivized to do the wrong thing by the metric then it's harmful.