If you can’t measure it, you can’t manage it. Meh.

The quote is management dogma. It makes sense. How do you measure success? How do you quantify what needs to be improved? What is a metric everyone can work towards?

In many ways, measurement is a proxy for intuition, trust, and complexity. Most of us have crappy intuition (checking my dating history). As for trusting data, “There are three kinds of lies: lies, damned lies, and statistics.” Complexity is increasing at an exponential rate. A hundred years ago, two brothers built an airplane. Nowadays we don’t fully understand how AI makes decisions. So we boil everything down to a few numbers.

That’s why Goodhart’s Law is important. It states “When a measure becomes a metric, it ceases to be a good measure.” More specifically, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

Measurements lose or hide fidelity. There are components we don’t understand or are black boxes (see any ad tech platform) or have secondary or tertiary level interconnectedness we cannot see. Plus we don’t fully understand causal relationships between people (their intentions and behaviors) and the interactions with systems. Yet, we are required to make decisions based on the ambiguity. We align incentives with the direction we want the metric to go.

As a slight tangent, we have the McNamara Fallacy, where one makes decisions based on easily identifiable numbers, and disregarding what cannot be easily measured. That is all of digital marketing. Digital-only marketing proponents keep pushing the idea that it is better because it is measurable. Not true. Easy measurement of digital does not mean it is more efficient. Not being easily measurable does not mean it is not relevant. In online advertising, all we hear about is CTR, CRO, CPM, CPC, CPA, confidence levels, etc., creating an industry of spreadsheet junkies that eventually lead to:

Oh really?

The danger is in choosing the wrong metric.

Let’s take Facebook as an example. Their measurement of success is making sure Facebook stock price goes up constantly. This is driven by increasing the number of clicks (oh oops, “engagement”) with ads or publisher content. The platform needs to predict what content a user will click on. Humans are tribal creatures. We look for ways to re-affirm our beliefs and social structures. The success of Facebook inherently generates and amplifies our reptilian instincts. Is it possible for any social platform to not become a channel for culture wars when the ultimate metric is clicks?

Similarly, if you are YouTube, and creators have to play the SEO game for revenue, you end up with garbage. (We will ignore the fact if PBS had done this, they would have been de-funded already. The consequences for Google? Nada. But, you know, Facebook and Google aren’t media companies, they are technology companies.)

Another tangent, the Streetlight Effect where people look where it is the easiest to look. The joke:

A police officer sees a drunken man intently searching the ground near a lamppost and asks him the goal of his quest. The inebriate replies that he is looking for his car keys, and the officer helps for a few minutes without success then he asks whether the man is certain that he dropped the keys near the lamppost.

“No,” is the reply, “I lost the keys somewhere across the street.” “Why look here?” asks the surprised and irritated officer. “The light is much better here,” the intoxicated man responds with aplomb.