We are victims of Goodhart’s Law. And that’s a good thing.
Goodhart’s Law says that, “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” A generalized way of saying it is, “when a measure becomes a target, it ceases to be a good measure.”
When we recently began an analytics project with a client, without citing this law, they expressed a concern that some of the key metrics we were modeling and putting into a dashboard would become less relevant over time, as they got better. And that is a bit of a paradox: metrics and analytics help us get better, but very often in the process, they become less useful. The attention models put on the metric result in actions that help improve performance on that metric. And in a way, that makes our models become stale, as the underlying phenomenon that we modeled begins to evolve.
Is that such a bad thing? Process improvement is the intent of the models. In a sense, we create our own obsolescence by virtue of the act of building and deploying those models.
Complicating matters is the fact that some metrics are already approaching levels where it is more difficult to exact further improvements. In semi-technical terms, they are asymptotically approaching some theoretical maximum or minimum.
We got our start in safety analytics when a manager approached one of our co-founders, Tom Davenport. As Tom wrote in his book, Competing on Analytics, “The … manager explained that safety was a top priority for the company and that it had improved considerably on this front, but it got harder to keep improving.” That “harder to keep improving” is what we call the “plateau problem.” We were able to help them break through that plateau with our predictive safety analytics solution. But even after a few years, the models were more useful for maintaining low levels of reportable safety incidents than making subsequent substantial reductions.
Where else have we seen this phenomenon in action?
We are not discouraged when we hear “it is harder to get better.” In fact, that is where advanced analytics can help most at the outset. We can measure our success when we sense that Goodhart’s law is starting to take hold. That is an odd kind of validation, but we will take it.