In-Depth

Revisiting ROI: Is Positive Payback Achieved?

Analysts challenge enterprises to look for, and properly measure, BI project benefits up front and after implementation

• By Stephen Swoyer
• 05/28/2003

As a result, they say, customers are buying into BI with the expectation that the ROI will be there. The problem, some analysts and users suggest, is that customers often are not bothering to see if that’s the case. After all, most businesses never commission ROI studies of their completed projects.

More troubling still is the possibility that customers will forego potentially rewarding BI projects because they believe that ROI—measured exclusively in dollars and cents—isn’t there.

BI Mindshare

Research firm Gartner Inc., for example, has found that although IT organizations shelved many projects in the midst of a protracted economic downturn, they’ve nevertheless kept BI projects alive (http://www.tdwi.org/research/display.asp?id=6631&t=y).

Why? “Because there’s the belief that the ROI is there,” asserts Kevin Strange, a senior VP and research director with Gartner. Adds David Folger, VP of enterprise analytic strategies with consultancy Meta Group: “[BI] has turned the corner. Organizations are feeling that there is a return on the investment.”

Evidence of BI’s mindshare among corporate decision-makers abounds: In a December 2002 survey of 100 CIOs at top companies in the US and the EU, for example, Merrill Lynch found that technology spending for BI was at the top of most CIOs’ lists. In another late 2002 survey, Forrester Research Inc. found that 44 percent of companies were mulling BI purchases in 2003.

Most BI vendors can also point to dramatic success stories. Business supply specialist Staples Inc., for example, uses software from SAS Institute Inc.—and a data model created by Alan Gordon, Staples’ director of sales forecasting—to evaluate potential real estate locations for new stores. Gordon’s model takes into account a variety of factors, including historical sales data and customer demographics from Staples’ existing stores, to select locations most likely to be profitable. The savings can be substantial: Staples can spend anywhere from $500,000 to $1,000,000 to shutter an unprofitable store.

Siemens Medical Solutions U.S.A. Inc., for its part, says it believes it saved $1 million as a result of implementing a reporting solution powered by BI software from Cognos Inc.

ROI Revisited

Such accounts are dramatic, to be sure, and certainly underscore the value of BI as an ROI-generating technology for some companies. But do BI customers—some no doubt lured by the promise of easy ROI—ever revisit their implementation experiences to determine if the ROI is, in fact, there?

Many analysts don’t think so, and a sampling of users with whom we spoke underscores their concerns. The reality, they say, is that the impact of a BI project often can’t be measured in dollars and cents. Instead, they argue, companies should recast ROI in terms of potential business benefits—which often can’t be measured by any tangible metrics. The problem, some say, is that businesses that are too focused on the dollars-and-cents of ROI could pass on some potentially rewarding BI projects.

Abdel Kander is VP of corporate development with Saama Technologies Inc., a Silicon Valley-based BI consultancy and integrator. Kander has seen many BI implementations—Saama has worked with technology giants 3Com and Brocade Communications, among others—and believes that for all but the largest, rip-and-replace BI efforts, business customers typically don’t do post-implementation ROI studies. “A lot of people talk about ROI during planning, but the reality is that the deals usually fly through because they tend to come in smaller chunks, so they tend to fly under the radar of the CFO, and there’s rarely any follow-up, except for the huge deals,” he explains.

There’s another consideration as well, says Kander: ROI studies need to be funded, and—except in very rare cases—this money is almost never included in an ROI model during the project planning phase.

Kander’s experience is seconded by Ted Ledman, director of information management with TCF Bank, a financial institution based in the Midwest. Early in 2002, Ledman was handed a daunting assignment: Create a business intelligence (BI) strategy, with an emphasis in customer relationship management (CRM) functionality, from scratch.

Ledman has a background in BI from work with another company and has been in IT for 35 years. He designed an infrastructure from scratch—specifying an RDBMS, hardware, ETL and data modeling tools, staff, and discrete BI tools. He also created an ROI model for his company’s proposed solution. “This plan included the ROI for starting up this department, purchasing all of the tools required, hiring the people, and the benefits of having all this cool stuff available. So our ROI was established for the whole department,” he comments.

TCF Bank’s goal was to improve its customer service and to support a cross-selling application used by 1,600 of its loan officers. By Ledman’s estimation, its implementation—based on an Oracle RDBMS, along with the PowerAnalyzer and PowerCenter BI and data integration tools from Informatica Corp.—has been a success. Yet he concedes he hasn’t been asked to square the project’s post-implementation ROI with the model that he first created.

That doesn’t surprise him, however. “I've been in this business for 35 years and I have never been asked to go back and prove my original estimates, or ROI,” he explains. At the same time, he points out, TCF Bank has realized business benefits as a result of its implementation. “Once we have our complement of staff using the BI tools we can say things like the business runs smoother, the staff is more efficient, [or] everyone's happier. But those things are hard to put a number on.”

Not Just About Dollars and Cents

Besides, Ledman confesses, “I’m not a big fan of ROI because it is like talking to an accountant … because they can make the numbers come out any way you want.”

His concerns are echoed by Cathy Benko, global e-business practice leader with Deloitte Consulting. “I do not believe that many of those financial measures are any way to assess the contribution of an individual project. You can make numbers say anything you want, and yet [[a sizeable percentage]] of companies never actually go back and see if what was delivered met what you expected to be delivered,” she points out.

Especially in large corporations, Benko says ROI can be extremely difficult to quantify, largely as a result of the inter-relatedness of projects across an enterprise. “Financial measures of a project's success are a hurdle. They make people feel good. They're fairly artificial,” she argues. “For one thing, in a large organization with lots of projects, it's often impossible to know how many of those projects are double-counting the [ROI] benefits to the organization.”

Florida Power and Light (FPL) is one such large organization. Joeri Carty, an HR systems project manager with FPL, admits that ROI wasn’t the most important consideration when his company began searching for a new reporting solution to complement its SAP HR implementation. “The eminent problem was that our whole reporting paradigm was changing. We wanted to change from providing raw data to folks that they kind of crunched through to providing more intelligent and analytical data out of the warehouse,” he explains.

Nevertheless, Carty says, he did the requisite ROI study for company executives, largely to demonstrate that his preferred solution—based on Informatica’s PowerCenter and PowerAnalyzer products—was cheaper than an in-house solution. Carty also had to contend with pressure from company executives who favored a product from another vendor, and used ROI research to demonstrate that the Informatica tools were a better fit for his company’s requirements.

“We just wanted to go with the best solution for our users, so we did just a simple ROI study in terms of what would it take for us to build it in-house, as well as some of the Gartner research that was going on that basically said that the SAP [Business Warehouse] wasn’t really where it needed to be yet,” he comments.

The result, Carty explains, is ROI that can be measured not in financial metrics such as dollars and cents, but, rather, in bona-fide business benefits. “A lot of the executives and directors don’t want to sit there and dig through all of that data, so what the new tool allows them to do is provide them with tools that allow them to move through this data,” he explains. “

What Happens to a Deferred Project?

The concern among some BI professionals, however, is that if there’s not a demonstrable case for financial ROI, executives won’t approve potentially rewarding projects.

That’s the perspective of Kevin Tighe, a report developer with a major payroll services provider. Tighe, who creates and manages reports for a variety of customers in several different states, says that most of the clients with whom he works don’t have plans to expand their BI investments in 2003. “Most that I have been in contact with need reports for their general ledgers or their upper management, but have not been actively adding to their staff or [purchasing] new software. And many have gone through layoffs, too,” he notes.

Although Tighe is persuaded by the argument that BI is an ROI-generating technology, he worries that companies are focusing exclusively on cutting costs and ratcheting up profits. As a result, Tighe says, he’s not convinced that businesses are willing to take chances on BI projects that don’t deliver strict dollars-and-cents ROI. “I just see them contracting and trying to figure out how to get more 'x' for less 'y,’” he concludes.