Return on Investment: The Difficulties of Measuring Technology's Dividends
When faced with a request for hundreds of thousands of dollars or millions of dollars, a manager would be regarded as derelict in his or her fiduciary duties by not asking the return on investment (ROI) question. Considering the impact on the company, it seems like a straightforward question to ask.
Yet when it comes to spending money on technology, the answer to the ROI question is far from straightforward. In truth, the most honest answer would often be we don't know. Or it would cost too much and take too long to find the answer. In some instances, maybe it doesn't matter -- a company must provide a certain capability, whatever the cost, or it will close its doors. The last answer is being heard more often as organizations face steep technology investments in e-commerce.
But the ROI question isn't going away. "We don't let our clients proceed until they do an ROI analysis," says Randal Oulton, senior analyst at Oulton Technology Management. At the least, Oulton insists that clients declare specific goals for the technology project and define and implement measurements that will reveal whether the goals are being met. Using his clients' goals and measurements as the yardstick, Oulton believes technology projects should generate an ROI of 150 to 200 percent over three years.
Rather than focus on a specific ROI number, Larry Marion, president of Competitive Edge Information Services Inc., a firm that specializes in ROI studies, looks at the timeframe. Short of a full-blown ROI analysis, "At the least, you want to show a payback in a reasonable amount of time," he suggests. If an organization can get a technology investment to pay for itself in two years or, better yet, 18 months, it is in pretty good shape from an ROI standpoint.
But for many executives the allure of the complete ROI analysis is compelling. This has been fueled by widely publicized stratospheric ROI results. For example, there was a 1,600 percent ROI associated with a data warehouse project a few years back. Numbers that big, however, must be taken with a certain amount of skepticism. The benefits from data warehousing are generally vague and difficult to quantify. How do you put a dollar value on easier access to better data?
The stunning 1,600 percent ROI was validated by International Data Corp. (IDC, www.idc.com), in a data warehousing study of 62 organizations. Even excluding failed projects and the exceptional performers -- both good and bad -- from the final tally, the IDC study, "The Financial Impact of Data Warehousing," reported an average 401 percent three-year ROI.
Compared with data warehousing, it should be easier to calculate ROI for Enterprise Resource Planning (ERP) solutions. When evaluating ERP an organization is dealing with tangible facets of the business: the number of orders processed, number of inventory turns, days of inventory on hand. These things can be counted, measured, and assigned a value with some confidence.
In a widely publicized ROI study of a company's move to SAP R/3, analysts calculated a 156 percent ROI over three years. At the time of the R/3 implementation, Analog Devices Inc. (www.imsgrp.com/analog) -- a manufacturer of specialized integrated circuits -- faced desperate challenges: to wean itself from a reliance on cost-plus defense contracts and to streamline itself to compete in the commercial marketplace. The ROI resulted from reduced marketing, administrative, and selling costs. A significantly shortened order fulfillment process, which the company attributed to the ERP system, also contributed.
The study, conducted by Competitive Edge’s Marion, evaluated the impact and benefits on individual departments over a four-year period when the software was in production. The ROI study strove to be comprehensive and conservative. The data for the study came from the examination of records and detailed interviews with managers involved in the planning and implementation of R/3. The study included the past performance of each functional area and company and analyst forecasts of future performance. Working capital requirements were analyzed and valued using a 7 percent cost of capital, while revenue projections were based on conservative industry estimates of 20 percent per year annual growth. The total investment included the estimated cost of both external and internal resources and all pre- and postimplementation expenses.
The Analog Devices ROI study was exhaustive. But studies like this don't come cheaply or quickly. ROI studies on this scale or larger can run over $100,000 and include teams of accountants, analysts, and auditors engaged over a number of months, Competitive Edge's Marion says. The study will also require the extensive participation of some of the organization's top decision-makers and key staff.
There are, however, some shortcuts available. A small team of software developers at a major bank in Asia wanted to implement a new high-end software configuration management solution for its large Windows NT development environment. Unable to take the time and not having the budget to undertake a major ROI analysis, the team concocted an informal ROI analysis based on the results of two pilot projects with a new configuration management tool and its on-going experience with Microsoft Corp.'s Visual Source Safe. Until that point, the development group was using Source Safe, a good but simple control tool that is included with Windows NT. The group had experienced some serious problems due to Source Safe's lack of advanced features and needed a more industrial strength tool. That need helped them justify to top management the significant added expense of a new configuration management solution.
The managers collected and tabulated all the numbers they could gather on the existing application development process -- including time spent generating releases, performing merges, correcting corrupted files, and waiting for code to become available.
The study revealed that for every three hours per release that developers had spent on various chores, it took only one hour with the new solution. Managers multiplied the difference across 50 releases each month and found the impact to be substantial. Similarly, they found that merges -- which previously took five hours -- required one hour with the new solution. Again, with eight merges each month, the advantage of the new configuration management solution was significant. Developers had once spent hours each month waiting for access to files or fixing corrupted files, now there was no waiting with the new solution. Valuing each developer hour at $100, the bottom line payback of the new solution quickly mounted. The final calculation determined almost $300,000 each year in savings.
The development managers also noted some intangible benefits of the new solution. "The soft benefits sometimes are more significant than the hard-dollar savings," Marion says.
In the case of the bank, the data from the new configuration management tool were not subject to the corruption and undetected changes that previously plagued the organization. Similarly, for the first time the developers had a solution that provided a solid audit trail; before they had actually lost critical labels, causing some configurations to be irretrievable. The new solution also enabled developers to deliver higher levels of service due to the increased development speed. Finally, developer productivity increased due to the new tool's faster performance and effective support for parallel development. Adding the value of the soft benefits to the equation, the development group concluded the new tool will save a total of $500,000 per year, producing a payback in less than 12 months.
The decision became a no-brainer. Top management quickly approved the purchase and full-scale deployment of the new configuration management solution.
On one level, answering the ROI question is simple. It requires knowing how much the new technology costs and a reasonable estimate of the value the organization will receive after the purchase. That value can be in terms of savings, opportunity, new revenue, or soft benefits -- such as improved customer satisfaction or better decision making.
Sometimes when an organization starts to determine the ROI for a technology project, it finds itself in a quagmire. "Usually, everyone can figure out the cost, but it is the other side of the equation -- the return, the value received -- that is open to question," Marion says. The answers for the return side, he warns, are easily subject to manipulation.
Surprisingly, some organizations don't know the expense side either. They know how much they are paying for the software license or the hardware, but they don't know what it will cost to own and support that software over time. This raises the infamous total cost of ownership (TCO) question; infamous because GartnerGroup (www.gartner.com) once pegged the TCO of a $2,000 PC over five years to be a whopping $12,000 to $14,000 per year. This widely debated figure included such intangibles as a futz factor.
When it comes to technology upgrades, many organizations lack the rudimentary facts to begin measuring a change. One organization was stymied when it tried to figure out the ROI for a new application development tool suite. The new tools greatly accelerated the development process in certain areas, such as porting code to new platforms and packaging and deploying an application. But the organization didn't know how long it took its people to do those tasks with the old tools, leaving it no basis for comparison.
In the end, remember the results of any ROI analysis have to be taken with a certain amount of skepticism. "You want to know the biases and the motivations of the people doing the analysis, what pressures they have been subjected to," Marion advises.
The challenges have not prevented organizations from trying to identify some form of ROI. For example, the fluid technology division of ITT Industries (www.ittfluidtechnology.com), for instance, knew that it took one to two years to complete most of its application development projects, but with the use of Vision JADE, the company was able to cut the application development cycle to as short as three months. Similarly, American Home Products Corp. (www.ahp.com) implemented Janus Technologies (www.janus-tech.com) asset management solution after projecting that it will achieve $1 million in hard-dollar savings over three years.
While it would be ideal if an ROI could be determined for every project, it is not always practical. Companies must make systems Y2K compliant, for example, regardless of the ROI. The ROI for many IT infrastructure projects is tantamount to an ROI for electricity or water -- unnecessary. Some programs must be done if the company is to function. Although managers should insist on some level of ROI analysis, they need to balance the cost and time of a full-scale ROI study against the opportunity and risk. For many projects something less exhaustive and more informal will do.