In-Depth
Using BI to Assess BI: The Efficacy Conundrum
Some adopters seem disinclined to use BI technologies to assess the efficacy of their business intelligence and data warehousing practices.
A recent survey from market watcher International Data Corp. (IDC) shines a light on the state of analytic efficacy. Simply put: when it comes to analytics, a surprising percentage of shops don’t know what or how well they’re doing. For example, fully one-third of respondents in IDC’s survey say they lack the means to determine if their analytic projects are successful.
IDC sampled more than 6,400 end users across a variety of industries. Awareness of analytic efficacy was higher in some verticals (finance, communications) and lower in others (education, manufacturing). The upshot, the analyst firm concludes, is that “the potential to gain competitive advantage from an increased investment in business analytics” -- an outcome that owes much to the judicious incorporation of analytic insight into business decision-making -- “is … hampered by a lack of effective measurement methodologies.”
This isn’t news to many BI experts. Take industry luminary (and frequent TDWI presenter) Cindi Howson, who annually assesses the BI tools market for her BIScorecard.com. Howson clearly knows whereof she writes: her seminar-length comparisons of BI and performance management (PM) tools are standing-room-only affairs at TDWI’s quarterly World Conferences. In her most recent BIScorecard.com tally, Howson found that both the usage and the perceived efficacy of BI tools declined in 2009 relative to earlier tallies.
Howson asked organizations to rate the “success” of their BI efforts; practices in which BI tools were used to help drive decision-making were judged “successful.” Just over one-fifth of shops rated their BI practices as “successful” last year, as against almost one quarter in 2007, according to Howson.
IDC and Howson are measuring two very different things, of course: analytic efficacy (in the case of BIScorecard.com usage survey) versus the ability to identify analytic efficacy (in the case of IDC’s sample).
Measuring the How, When, Where, and Why of BI
A larger issue, experts say, is that not all adopters are making an effort to understand how, when, where, or why they’re using BI. For all of the talk about BI ROI, a substantial portion of adoptees -- perhaps as many as one-third, according to IDC -- simply aren’t in a position to assess ROI.
Michael Corcoran, senior vice-president and chief marketing officer with Information Builders Inc. (IBI), says he’s unfailingly surprised by how few companies actively monitor their BI usage.
This is in spite of the fact that IBI started bundling a monitoring facility with its flagship WebFOCUS BI suite more than a decade ago. The not-so-novel idea, Corcoran explains, was that customers could use WebFOCUS to … assess its own performance. The rub? After ten years, Corcoran still has to goad customers into doing so.
“We have to tell them, ‘Turn this on. Use this.’ It can help you understand who’s using [WebFOCUS], when they’re using it, how much utilization it has, what [reports are] most popular. You can get more granular from there. The important thing is to start [monitoring BI usage] in the first place.”
In the age of pervasive analytics, the stakes are higher. A decade ago, BI technologies such as WebFOCUS weren’t so much used to drive -- via the disclosure of new insights -- as to confirm the business decision-making process, chiefly (in the circa-2000 state-of-the-art) by serving up reliable, up-to-date operational or production reports. These reports were sometimes complemented by multidimensional reports generated by then-far-from-mainstream OLAP technologies, such as Oracle Corp.’s Express, Hyperion Solution Corp.’s (nee Arbor’s) Essbase, Applix Inc.’s TM1, and Microsoft’s (then-fledgling) Plato OLAP implementation.
A decade later, OLAP has gone mainstream, advanced analytics -- a category that encompasses data mining, predictive analytics, and statistical analysis -- has come to the fore, and shops are gearing up to integrate analytic capabilities into a growing number of applications.
In IBI’s case, Corcoran says, a lot of users aren’t even aware that they’re consuming analytics, or (for that matter) that they’re interacting with a third-party reporting handler of any kind. That’s great for driving BI consumption, Corcoran notes, but not so great when it comes to determining BI efficacy. “That’s actually a very effective marketing tool. When a customer does that [turns on the built-in WebFOCUS monitoring facility] and sees what they’re doing, sees how WebFOCUS is used by just so many different users,” he concludes. “It’s even better when they’re using [WebFOCUS] to determine how [consumers] are using these [insights] to help them do their jobs or [to] inform some of their decisions.”
Application usage monitoring isn’t a neglected market segment, of course. It’s been a mainstay of the software asset management (SAM) toolkit for decades, for example. More recently (in the last 15 years or so), vendors such as BMC Software Corp., CA Inc., Compuware Corp., and the former Peregrine Systems Inc. (among others) have introduced dedicated application performance management (APM) software tools. One core component of APM is its ability to monitor application telemetry -- e.g., usage, reliability, availability, and other (often environment-specific) metrics.
Particular to the BI segment, Appfluent Technology markets Visibility, a usage and workload analytics offering designed for BI and DW practices. It has partnership agreements with several BI players, including SAP BusinessObjects, the former Cognos Inc. (now an IBM Corp. company), Informatica Corp., and Oracle Corp.
IDC’s survey doesn’t just (or mostly) concern the pervasiveness or usage of enterprise analytics, however. Instead, it addresses the purported efficacy of analytic technologies. In other words, traditional APM metrics -- such as utilization -- yield an incomplete picture of how analytic technologies are being consumed. Usage monitoring tells you how, when, and sometimes why, but not to what end. That’s why IDC urges shops to develop effective measurement methodologies to assess analytic efficacy.
One IDC recommendation: A methodological process that harnesses BI, analytics, and performance management (PM) concepts and methods. “Technology vendors and consultants need to focus on applying business intelligence and analytics on the business analytics projects to ensure that BI groups are able to clearly articulate the benefits to their constituents and ensure ongoing funding of projects,” IDC says.