In-Depth

Judgment versus Analytics

A recent survey made BI stomachs turn when it reported that 40 percent of business decisions relied on judgment instead of analytics.

• By Ted Cuzzillo
• 02/18/2009

Survey results from Accenture made BI-industry stomachs turn late last year when it reported that business users' trust in analytics still had a long way to go. The headline alone was alarming. About 40 percent of surveyed companies said their important business decisions relied mostly on judgment instead of analytics.

"'Oh, now what do we do?'" was how Michael W. Cristiani of Market Intelligence Group imagined the widespread response in the analytics community. "It must have been frustrating. [BI consultants] have tried to introduce some rigor and best practices into the data side of things, only to hear that these [survey respondents] don't trust the data or know where the data comes from. Yikes!"

Below the headline, data broke down the reasons. Of the 40 percent, 61 percent of decisions relied on guts because of a lack of good data. Sixty percent -- apparently overlapping the first group -- were from absence of past data, such as predicting acceptance of innovation. Fifty-five percent relied on qualitative or subjective factors. Only 7 percent of judgment-based decisions were done using "judgment" instead o f analytics.

Interesting data, but my gut told me something was amiss. As I expected, experts I checked with say "guts" and analytics aren't so easy to separate.

Cristiani advises clients to make decisions based on data but to audit them against experience -- that is, to use judgment. He tells about one set of data he presented on point-of-sale wait times to a group of experienced people. The data said the average wait time was about two minutes. The group didn't accept it. Something's wrong with the data, they told him. He says, "They could smell it."

People talk about "trusted data" becoming part of the gut feeling, he says, but what does it take for data to be trusted? "If it really is a lack of confidence in competence," he wrote to me via e-mail, "fix the competence." Lack of institutional memory also undermines trust. Users say to themselves, "'I hope I'm getting this right. Where are the old guys who knew this stuff? Oh, yeah, they took the early-retirement buyout.'"

Can we trust Accenture's data? It seems good enough. Results were based on responses from 254 managers and executives at companies earning $500 million or more in 2007. The question, however, is a problem. Question 7 reads, "Most companies make decisions to some extent based on employing business analytics and to some extent based on judgment gained from experience and knowledge of the business. What percentage of your company's important business decisions do you estimate are made by employing business analytics as opposed to relying mainly on judgment?"

Michael Goodman, an old hand at crafting surveys for a variety of uses and now president of Extra Mile Audience Research, says, "You ask questions like this to get a rise out of people."

Of the several bloggers to comment, Neil Raden, co-author of Smart (Enough) Systems (Prentice-Hall, 2007) wrote, "The proper phrasing of this question is, do they rely on it to the exclusion fact-based reasoning? I don't think you can separate gut from analytics, because all analytics can do is inform your decision and at some point you have to apply your gut to the analytics."

Dave Wells, former education director of TDWI, went further. "If you reduce analytics to just the numbers, then they're pointless," he wrote me. "The judgment to know which data to use, to know which questions to ask, to know where to probe deeper, to go beyond 'knowing what' and get to 'understanding why' … all of this is analytics the way that I see it."

Accenture looks at the flip side. In its press release and from a spokesperson by telephone, the company says the survey shows how much work must be done and what a high priority companies will give it.

Another message also seems to break through: don't quibble over the question's phrasing. If users don't believe they can use analytics with decisions, then they can't.

That portends work that should keep consultants' phones ringing well into the future.