In-Depth

Metrics -- The Fuzziness Factor

How do you measure those hard-to-reach functions where bookkeepers don't go? Experts share their advice.

Measuring profit is easy, but how do you measure the intangible?

An old ski lodge in California's Sierra Nevada mountains that I once helped manage made me think about that. During my 10-year stint on the board of volunteer leaders, I couldn't nudge my fellow leaders away from their devotion to tangible work such as carpentry, painting and plumbing. There was more to the lodge, I argued again and again, usually to stone faces that sometimes gave me a nod. Too bad my insight was never as tangible as a leaky roof or a fire marshal's ultimatum.

Still, it seemed obvious to me that the generations of visitors, many of whom had grown up or grown old as frequent guests, didn't do so to admire the roof or dine on often bad spaghetti or sleep in bunks. At this lodge, most visitors felt like they belonged. That was either because they'd made a lot of friends there or because of all of the hours they'd invested as volunteers (or both). To me and many others I knew, it sometimes felt more like home than home did.

Late in my leadership stint, it occurred to me that a metric might have made a difference. I just didn't know how to measure something so intangible. I've stopped visiting the lodge, but now I know two experts on metrics who help clients reach those hard-to-measure effects.

Stacey Barr, "the performance measure specialist" (http://www.staceybarr.com) based in Brisbane, Australia calls such metrics "proxy measures." Zach Gemignani of Juice Analytics http://www.juiceanalytics.com in North Carolina calls them "franken-measures." Their definitions vary a little, but each of them has lively mini-cases and insight.

Barr's quick advice: stop brainstorming and drop the creativity. Metric-making is an analytical exercise. People in search of metrics jump into measuring too soon, she says. One of the first things she does for clients is to stop that premature conversation about measures. She'd restart it to define what effect they were looking for -- and describing the results in "sensory specific language."

She says many people in business or government tend to use "fluffy, vague language." In business "we've gotten into using words that mean seven different things to three different people," she says. Her prime examples: quality, efficient, effective, sustainable, and enhanced." "We have to use words that are more concrete." That ensures that when people sit around talking about goals and results, they have the same images in their heads.

"The first step," says Barr, "is to really understand what results matter most and are worth measuring." It's the step most people leave out, and they're left with poor metrics or none at all.

She once worked with an investment firm's IT team that felt unappreciated. At the first meeting with her, they listed all the things they did, from keeping the network stable to upgrading software. Then the more they talked about it, the more they realized that the truly meaningful things were the results.

"They really thought hard about what it looks like, sounds like, and feels like when 'the wheels fall off.'" That is, when any technical failure forced any internal clients to stop work. They broke events down into three levels and on public displays showed an up-to-the-minute report on the company's technical status.

"There was a big culture change for the team," she says. "They went from the traditional 'this is how much work we do' attitude to 'this how we keep the company working.'" A town council she worked with wanted more public participation in meetings. They had been measuring it by the number of meetings held -- that is, more meetings automatically counted as more participation.

She asked members of the council, "If the community were more engaged, what would people be doing that they're not doing now?" At first they said things like more people showing up, a high proportion speaking up, more ideas proposed, and so on.

Eventually they came up with two measures that worked together: the number of "fresh faces" (those who'd never attended before; people who had decided to give it a try) and the number of familiar faces (those who'd come to at least half of all recent meetings and had decided that showing up was worthwhile).

Those Gray Areas

In some situations, you can't avoid gray areas. Juice Analytics' Gemignani had a client with an online curriculum who wanted to know when to take customers off the active list. Students paid up front and could take one year to finish the curriculum and sometimes longer. If the first year ended, the company might expect the customer to renew -- or not, because they might be still working on the first year's curriculum.

There was no definite answer. Gemignani says, "One way to treat it is not such a binary way, to make the relationship sort of fade to black." The customer starts out as a strong customer right after a purchase, but as he interacts less and less, his customer status diminishes.

The Juice Analytics weblog page on Franken-measures lists helpful guidelines for metrics. It advises, for example, that they be complete, modeling all relevant factors to provide a holistic measurement of the concept. They should be understandable and logical to users, direct measures instead of proxies or subjective measures, and each driven independently of the others. (See the complete explanation at www.juiceanalytics.com/writing.)

These metrics can go wrong, according to Juice. They may be unnecessarily complex. They may be designed to (impossibly) count everything. A changing baseline or a metric so obscure it loses credibility pose other problems. A technology-industry executive I know from that ski lodge offers another caution: "Metrics are critically important," he wrote in email, "yet I find that management types who demand metrics usually haven't a clue about the real, underlying processes under way, and tend to make idiotic decisions because they take the metrics at face value."

At the ski lodge, though, a face is exactly what I wanted to create.

People talk about numbers. When "the Dow" is up or down by 300, we mention it to each other over sandwiches. The ski lodge's set of metrics might have borrowed from Barr's client who counted faces, new and returning. It might also have counted volunteer hours, since that was a proxy for that feeling of ownership. Heeding the Juice Analytics advice, I would not try to combine these two.

It would be transparent and credible. If it drifted downward, we'd know what to do.Best of all, this kind of metric is made to be noticed. Our lives, and our businesses, are defined by what we notice. The metric would say, "Notice this."

Must Read Articles