Know the Difference Between Your Data and Your Metrics

How many views make a YouTube video a success? How about 1.5 million? That's how many views a video our organization, DoSomething.org, posted in 2011 got. It featured some well-known YouTube celebrities, who asked young people to donate their used sports equipment to youth in need. It was twice as popular as any video Dosomething.org had posted to date. Success! Then came the data report: only eight viewers had signed up to donate equipment, and zero actually donated.

Zero donations. From 1.5 million views. Suddenly, it was clear that for DoSomething.org, views did not equal success. In terms of donations, the video was a complete failure.

What happened? We were concerned with the wrong metric. A metric contains a single type of data, e.g., video views or equipment donations. A successful organization can only measure so many things well and what it measures ties to its definition of success. For DoSomething.org, that's social change. In the case above, success meant donations, not video views. As we learned, there is a difference between numbers and numbers that matter. This is what separates data from metrics.

You can't pick your data, but you must pick your metrics.

Take baseball. Every team has the same definition of success — winning the World Series. This requires one main asset: good players. But what makes a player good? In baseball, teams used to answer this question with a handful of simple metrics like batting average and runs batted in (RBIs). Then came the statisticians (remember Moneyball?). New metrics provided teams with the ability to slice their data in new ways, find better ways of defining good players, and thus win more games.

Keep in mind that all metrics are proxies for what ultimately matters (in the case of baseball, a combination of championships and profitability), but some are better than others. The data of the game has never changed — there are still RBIs and batting averages; what has changed is how we look at the data. And those teams that slice the data in smarter ways are able to find good players that have been traditionally undervalued.

Organizations become their metrics.

Metrics are what you measure. And what you measure is what you manage to. In baseball, a critical question is how effective is a player when he steps up to the plate? One measure is hits. A better measure turns out to be the sabermetric "OPS" — a combination of on-base percentage (which includes hits and walks) and total bases (slugging). Teams that look only at hitting suffer. Players on these teams walk less, with no offsetting gains in hits. In short, players play to the metrics their management values, even at the cost of the team.

The same happens in workplaces. Measure YouTube views? Your employees will strive for more and more views. Measure downloads of a product? You'll get more of that. But if your actual goal is to boost sales or acquire members, better measures might be return-on-investment (ROI), on-site conversion, or retention. Do people who download the product keep using it, or share it with others? If not, all the downloads in the world won't help your business.

In the business world, we talk about the difference between vanity metrics and meaningful metrics. Vanity metrics are like dandelions - they might look pretty, but to most of us, they're weeds, using up resources, and doing nothing for your property value. Vanity metrics for your organization might include website visitors per month, Twitter followers, Facebook fans, and media impressions. Here's the thing: if these numbers go up, it might drive up sales of your product. But can you prove it? If yes, great. Measure away. But if you can't, they aren't valuable.

Metrics are only valuable if you can manage to them.

Good metrics have three key attributes: their data are consistent, cheap, and quick to collect. A simple rule of thumb: if you can't measure results within a week for free (and if you can't replicate the process), then you're prioritizing the wrong ones. There are exceptions, but they are rare. In baseball, the metrics an organization uses to measure a successful plate appearance will impact player strategy in the short term (do they draw more walks, prioritize home runs, etc.?) and personnel strategy in the mid and long terms. The data to make these decisions is readily available and continuously updated.

Organizations can't control their data, but they do control what they care about. If our metric on the YouTube video had been views, we would have called it a huge success. In fact, we wrote it off as a massive failure. Does that mean no more videos? Not necessarily, but for now, we'll be spending our resources elsewhere, collecting data on metrics that matter. Good data scientists know that analyzing the data is the easy part. The hard part is deciding what data matters.

Know the Difference Between Your Data and Your Metrics
Jeff Bladt and Bob Filbin