Quality Metrics: Objective measures for subjective things

By Duncan Anderson. To see all blogs click here.

Reading time: 6 mins

Summary:

I find it’s often easy to have Quantity Metrics (eg Revenue, Monthly Active Users), but hard to have Quality Metrics (eg time well spent on site, improvement to education), but… also sometimes it’s not hard to have a Quality Metric (eg ‘cost per tonne to orbit’, ‘cost per renewable mile driven’).
What gets measured gets managed. To run a company well I think you want to have objective Quantity and Quality Metrics.
I believe you can build objective measures for subjective things.
I believe you can build objective Quality Metric(s) to represent 80%+ of something subjective.
Developing Quality Metric(s) is normally a big win.

Culture happens by default or design. Objectives happen by default or design.

I think it’s unwise to not have Quality Metric(s) as the equal most important metric or the most important metric.
Missionaries vs Mercenary

It’s normally much easier to motivate people to improve the world (Quality Metric) over something like ‘increase revenue’ (Quantity Metric).
Of course a business needs to be economically viable, but hopefully also making the world better!
Missionaries normally stay around longer, push through tough times, etc. Missionaries > Mercenaries.

Missionaries possible = 1. Company has a mission * 2. Company has Quality Metric(s) that show if progress is being made towards the mission.

Without objective Quality Metric(s) it’s often easy for an objective Quantity Metric(s) (eg revenue, monthly active users) to be given too much weight.
Having Quantity Metric(s) as the most important thing often means you get mercenaries.

Jingle: Either you manage culture or culture manages you. Either you have objective Quality Metric(s) or it’s hard to have a culture that leans towards ‘missionary’ (instead of mercenary).

+++++++++++++++

Details

Country Example

Our Gross National Product, now, is over $800 billion dollars a year, but that Gross National Product - if we judge the United States of America by that - that Gross National Product counts air pollution and cigarette advertising, and ambulances to clear our highways of carnage. It counts special locks for our doors and the jails for the people who break them. It counts the destruction of the redwood and the loss of our natural wonder in chaotic sprawl. It counts napalm and counts nuclear warheads and armored cars for the police to fight the riots in our cities. It counts Whitman's rifle and Speck's knife, and the television programs which glorify violence in order to sell toys to our children.
Yet the gross national product does not allow for the health of our children, the quality of their education or the joy of their play. It does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile. And it can tell us everything about America except why we are proud that we are Americans.

1. Net immigration demand for a country as a percentage of the population (if citizens are wanting to leave that is not a good thing, if others from around the world are wanting to come that is likely a net good signal)
2. Broad based decline in crime (eg violent crime, domestic violence, white collar crime, corruption)
3. Minimum wage increasing - faster than CPI
4. Average wage increasing - at above peer averages

Company example 1 - Edrolo - A company making education resources for secondary schools

This is a bit of our secret sauce.
But at a high level: improving on tests that we rate (I think some tests are amazing, and some really bad, find the tests you rate and then use them as part of the picture), conceptual understanding (not procedural understanding), increased portion of student continuing education past Year 10, and more.

Company example 2 - Twitter

How do you measure this?
I’m not aware of an easy way to measure this, so I think you’d need to have team(s) of people who are mapping different segments of users and building a metric for ‘unregretted user-minutes’ likely through speaking to users constantly.
As an example when I was at Google back in 2011/12, Google had a team of people who made an objective measure for search quality, compared Google to competitors and then mapped if search quality was improving over time. It was really quite awesome to see.
On their rating scale at the time Google was ahead of Bing and Yahoo, improving over time (ie year on year) and the gap between Google and Bing was increasing literally in a summative quantitative measure.
Making tests to objectively reflect a large portion education quality is super hard but IMO doable. Making a quantitative metric for search quality is super hard but IMO doable. I’m confident the same could be done for Twitter and it would really be worth spending the time to figure out :).

Personal example

1. Saving money each year
2. Physical health in good place (eg weight in healthy range, not getting sick, sleeping well based on Oura ring)
3. Friendships improving (do I have 3-5 quality friends that I feel a deeper connection to than I did a year ago)
4. Enjoy work (I personally track how much I think about quitting vs how much I’m excited about something at work in spreadsheet)

If you only take away one thing

For a quality life, have Quality Metric(s).
For a quality company, have Quality Metric(s).
No money is normally unhappiness. But having money isn’t happiness, money is freedom, freedom to make decisions. Freedom to make good and bad decisions.
I’d like to have enough money and hopefully to make good decisions!
I find having Quality Metric(s) is a great way to balance tradeoffs and try to make good decisions!

Cloud Streaks