Goodhart's law, credentials and social order - Part 2
--
Click here to read part 1.
I find that when I go in with questions I get a lot more out of them, rather than reading blindly. Below I have written some questions to think about before reading this.Questions to think about:
How does measurement affect me today?
How does it affect me in the long term?
How does this affect society around me?
Why isn’t this a more popular problem?
--
“I have been struck again and again by how important measurement is to improving the human condition” - Bill Gates
What we measure dictates behaviour. We already know this and have discussed it extensively. This is all well and good but as people, we tend to be self centred, so you probably want to know how it affects you and why you should care. I’m hoping this post can answer that for you. Specifically, I am hoping to convince you of these 3 things:
Measurement of people is one of the hardest problems we face as a society
Measurement of people is one of the most important problems we face as a society
Measurement of people is one of the most overlooked problems we face as a society
First things first, why is measurement so hard? For starters most people don't know what they actually want to measure. Take the example of a software engineer being continuously evaluated at her workplace. What sort of things would one measure her against? Most people would say a mix between problem solving and team working skills, but what does that mean? Assuming you knew what you wanted to measure (let’s choose teamwork for now), how would you go about measuring it? What sort of data would you try and capture? If “teamwork” is made up of multiple intangible parameters, which in this case it is, then how would you separate them and convert that into concrete metrics?
Let’s now assume you magically solved these two initial roadblocks. You now face the final hurdle of not knowing whether this is the right thing to measure. In the case of the software engineer, we want to measure her “team working” skills, but how do we know that's the correct measurement to optimise for? Metrics may tell us we are performing well, but that does not mean it matches reality. Considering the fact that no “teamwork” metric comes remotely close to the observations one gets by working with someone, I would say no, the reality does not match.
Another good example of this is the measurement of citations in research. This quote sums it up perfectly.
“What science has failed to notice is that the measurement has become more real than the thing being measured.”― R.A. Delmonico
The number of citations has become one of the primary metrics for researchers. With that has come several negative side effects. Some have even suggested the rate of scientific progress has slowed down over the years, and I suspect this may be one of the reasons.
This brings me onto my next point. Measurement of people often dictates the velocity of societal progress and societal social order. Needless to say these two things are extremely important, not just on the aggregate level, but on an individual level too. Let’s take you for example. You may be thinking “this doesn’t apply to me as I’m measured correctly” or “this doesn’t really apply to me, as I don’t put myself in situations where I need to be measured”. I hear you, I once was you, but you, my friend, are wrong.
Like you, I once held that belief, until I attended an event that completely changed my point of view. I attended this event hoping to connect with other “smart” people. The room was filled with different kinds of decision makers, all with their shiny credentials. I listened to the conversations, I tried to engage, but man, there was no denying the fact that everybody was talking a whole lot of rubbish. It was intellectual masterbation on steroids. I’m not trying to diss these people, to each their own. The important thing to note is that they were key decision makers in their respective fields and society. That’s when it dawned on me that even if I try to avoid this game of bad measurement, the majority of people won’t and this majority will end up leading and influencing how society evolves. Have goals of climbing to the “top” ? Well, have fun. They will be waiting to greet you at the door. Have dreams of changing the world? Goodluck working with and against them.
On a more serious note, this is really a major cause for concern. Let’s look at it from another angle, using Singapore as an example. They pride themselves on being a meritocracy built on testing (which I discussed here), yet this testing has been cited as one of the key drivers behind the high suicide rate of 10-29 year olds. Even if you could escape that, would you want to be surrounded by that?
So if it is such a hard and important problem, then why is it overlooked? Well, it is a problem that comes in a pair, but often takes the back seat. Education is probably the best example. Education and credentials go hand in hand, yet credentials get significantly less attention than education does. These forms of measurement (exams, credentials, interviews etc) dictate what people learn and how they are valued. If measurement systems do not incentivise and recognise new and/or better value, then in the eyes of society one remains valueless and it will continue to incentivise low value. More often than not, people don’t seem to question if our measurement systems itself are wrong. It is taken as some sort of “fact” to work with and build on top of. With that mentality, it isn’t a surprise that it’s overlooked as much as it is.
Given what you now know, it should be clear why the problem of measurement is hard , important and overlooked. That said, being the optimist that I am, I do think we have the tools to change this if we really want to. Software is a super power I feel we haven’t properly exploited. Bret Victor gives a brilliant talk about this. There are things I believe were previously impossible, that software now makes possible. Not to push my own agenda here, but one of the ideas I really believe in is the direct measurement performance and eventually potential. I have spoken about what makes for good metrics in part 1. It’s my belief that the internet and software makes it possible to finally have good, objective metrics on people. Think about it. Given how much data there is on individuals (Slack/Teams, Github, Salesforce, GDocs etc), coupled with more offline data coming online (see fireflies.ai), a lot of the things that were previously unobservable, are now observable. Thanks to data science and NLP we can use this data to better understand and predict how an individual will perform. This is just one idea out of many unexplored ideas.
The timing to solve this problem is also pretty great too. Putting the technical advancements aside, there are a lot of emerging fields that are dependent (to varying degrees) on having this problem solved. Here are some, to name a few:
Charter cities. In the case of charter cities, how will talent be identified and organised?
Global remote working. How does globalisation affect the identification and organisation of talent?
Online education. How does online education change the way we identify and develop talent?
I could go on, but I trust you catch my drift. It’s a hard, important, overlooked problem you should care about and now you know why. I hope I have drummed this into your head.
So.
Are you convinced?
And if you are, what are you going to do about it?
As always, please feel free to reach out to me to bounce ideas, collaborate or whatever frankly. I am currently working on solving this problem (full time) so I am always looking to meet cool people.
If you would like to join the credentials and meritocracy community click here, here is a good guide describing who we are looking for. Or you can follow me on twitter @adaobiadibe_