My Crib Sheet for Tyranny of Metrics

By Endy Tjahjono. Last update 01 Jun 2021.

The last chapter of Jerry Muller’s The Tyranny of Metrics is so good I have to make a crib sheet of it.

(The explanation is of course available in the full book.)

One. What kind of information you are measuring?
Measuring inanimate objects is more reliable. Measuring objects that can react to the measurement, such as a human, is not so reliable. But if the human agrees with the goal of the measurement then the measurement can become reliable.

Two. How useful is the information?
Even if something is measurable, it may not be worth measuring.

Three. How useful are more metrics?
Usually measurement is useful to identify outliers. Not so useful for the middle of distribution curve.

Four. What are the cost of not relying on standardized measurement?
Often we can use non-standardized measurement. Example: to find out the best teacher, how about noticing whether the parents of students request particular teacher for their children. Another example: gathering feedback from beneficiaries to judge the result of charities.

Five. To what purposes will the measurement be put?
Who will consume the info?

  1. Good: the practitioners themselves to do internal monitoring and analysis, recognizing those who excel and offering assistance to those falling behind, discovering best practices. Example: crime data used by police to allocate units.
  2. Bad: the boss to dispense reward and punishment. Example: crime data used to determine precinct commander promotion.

Outsiders who don’t understand the limits of the measurement will make mistakes using the measurement.

Reward and punishment reduce intrinsic motivation, induce gaming of the system and fraud. Low stakes measurement is more effective. Avoid direct pay for performance.

Six. What are the cost of acquiring the metrics?
Remember that collecting, processing, and analysing data take time. People working on the report are not doing the actual activity.

Seven. Ask why the boss is demanding performance metrics.
Maybe the boss has little experience for the domain. That’s why it is preferable to hire / promote from within the domain.

Eight. How and by whom are the measure of performance developed?
Involve people who have a stake at the outcome. Best if developed by practitioners with experience in the domain. Less effective if imposed from above.

How to cope: find the most efficient way to provide the data that meets the minimum standard and won’t harm your unit.

Nine. Remember that the best measures are subject to corruption / goal diversion.
Example is choosing measurement to determine a doctor’s compensation:

  1. Number of procedures performed: will prescribe procedures even if the patient doesn’t need it.
  2. Number of patients seen: will skip time consuming procedures.
  3. Successful patient outcome: will take easy cases.

Ten. Remember that sometimes recognizing the limits of the possible is the beginning of wisdom.
Not all problems have a solution. Fewer can be solved by metrics.


comments powered by Disqus