Data Warehouse Factors to Address for Success: META Group’s Scorecard Report Illuminates Key Factors

The responses in META Group's recent data warehouse survey confirm that team experience, architecture, expectation management and data quality are the prevailing factors influencing the success of DW initiatives. With the emerging knowledge base of DW best practices and a growing skill base of DW practitioners, over 90 percent of DW installations are successful.

• 05/01/2000

The responses and subsequent analysis in META Group’s 1999 Data Warehouse Scorecard Report confirm that team experience, architecture, expectation management and data quality are the prevailing factors influencing the success of data warehouse (DW) initiatives.

More than half of respondents report moderate success or better, although outright, documented return on investment shows up for only one-third of respondents. With the emerging knowledge base of DW best practices and a growing skill base of DW practitioners, fewer than 10 percent of DW installations are now utter disasters.

The availability of hard "in-practice" data, such as this report, gives DW project sponsors, managers and architects a tool to benchmark leading technologies and approaches against DW success rates. With these DW characteristic reference points, existing project teams can rationalize their levels of success, and newly formed project teams can improve their odds of success. By 2000/01 these success factors will be firmly ingrained in the DW community’s consciousness, and by 2003/04 the rate of reported DW installation success will improve to converge with rates for traditional application development initiatives.

Through 2003/04, essential DW success factors will remain unchanged even as new architectures, strategies and technologies emerge. Sets of best practices and DW standards will form around these success factors, leading to the formalization of DW projects well beyond current experimental tendencies. Built-in DW capabilities currently found in some packaged business solutions will be evident in all leading packages by 2001/02 and will reflect these winning architectural principles to a large extent. Notable non-factors in determining DW success include the organization’s industry type and the choice of outside systems integrators.

The traditional DW concept has evolved into a broad set of architectural variations. Data marts (standalone and federated), operational data stores, exploration warehouses and application data stores, in addition to conventional hub-and-spoke DW, have enabled sophisticated information management solutions to meet exacting business requirements. Along with leveraging the volume of data generated by ERP applications, there is a burgeoning need for the DW to sustain customer relationship management (CRM), supply chain management (SCM) and electronic commerce initiatives.

META Group’s 1999 Data Warehouse Scorecard Report is a point-in-time survey of existing data warehouse and data mart environments. It focuses on what is currently working in practice, and addresses the aspects of a maturing technology from the subjective perspectives of those deploying it.

The distribution of DW architectures between data mart-oriented (34 percent) and centralized DW-oriented (34 percent) approaches (including hub-and-spoke) is especially uniform. Though there is little preference for one approach over another, the profiles of organizations adopting them are significant. Organizations reporting failed DW implementation are two or three times more likely to have employed data mart-oriented approaches, while classic DW architectures revolving around a centralized data store enjoy a strong lead among acknowledged successful implementations.

We also find that, because single-subject data marts fail more often than formal multi-subject DWs, an insulated data mart project is no guarantee of success. For central DW environments, incremental cost of each additional subject area is constant. In data mart-oriented environments, however, incremental cost of each additional mart increases. In-production DWs employing data mart-oriented architectures also show a 70-percent higher average cost per subject area than those with centralized architectures. Our qualitative analysis suggests this stems from the exponential administrative overhead of decoupled data marts and the technical burden of ensuring consistency and synchronization of federated data mart environments.

In addition, the low rate of operational data store architectures (12 percent) indicates relative disinterest in tactically oriented information solutions. The higher rate of mixed DW architectures (20 percent) illustrates a level of information supply chain maturity for successful DW projects and confusion/politics for unsuccessful ones.

Unsurprisingly, the longer an organization toils at DW, the more likely success will result (e.g., after 2+ years, projects are likely to have twice the level of success as 6-month-old projects). The earliest adopters tend to rate significantly higher, representing not only brute-force success over time, but also cultural propensity toward innovation and acceptance. Successful shorter-term efforts are characterized by a high degree of reliance on technology and larger team sizes. The high number of failed efforts that have persisted for three years or longer begs deeper quantitative consideration, but we believe these results stem from politically mired projects, over-scoped (non-iterative) efforts, sponsor turnover and DW "re-dos."

Although it is not evident that larger teams are more successful, we find that undersized teams can really hurt a project. Core team sizes of seven to 10 FTEs seem optimal to avoid failure. We also find that management’s common belief that a small data mart implies a small team does not hold true.

A lack of expectation management remains a barrier to DW success. Even successful DW initiatives seem haunted by the ghosts of IT projects past. However, successful efforts show double the propensity to deal directly and continuously with issues like user participation, executive involvement, and overall project promotion. Within IT ranks, positive expectation management includes education on architectural, managerial and usage distinctions of DW versus traditional OLTP systems development projects.

Successful DW efforts are nearly three times more likely to deal effectively with data quality issues compared to failed projects. Organizations that jump with both feet into DW data modeling efforts neglect many critical analysis activities that deal with assessing data source availability, accessibility, suitability, "integratability," completeness and accuracy.

Industry. An organization’s industry type does not drive DW success, though varying adoption rates by industries yielded a range of overall scoring results. For example, retail and education organizations (overall DW laggards) are experiencing slightly lower rates of success, while transportation and the historical technology leader, telecommunications, achieve success more often. Overall, however, success rates are relatively consistent across industries.

Consulting. The study also unearths the realization that DW boutiques and DW vendor consultants are found twice as often on successful DW initiatives than large systems integrators. This finding is as much a reflection of global integrators’ current inability (or lack of desire) to capture, package and share DW best-practices as it is a reflection of the prevalence of targeted competencies throughout the smaller and boutique firms. Until global integrators master the economies of scale they have enjoyed in OLTP systems development, we believe individuals, not firms, will continue to make the difference in DW.

ETL Tools. While extract-transform-load tools did not correlate highly to success or failure, highly successful organizations did tend to persist with these tools rather than giving up on them. This finding illustrates the long-term benefit of ETL tools despite their lingering learning curves.

DW project leaders and LOB sponsors must look closely and objectively at the characteristics of their initiatives. While there is no formula for DW success, attention to fundamental factors can lead teams away from danger and toward success. Organizations should apply these survey results to select partners, coordinate activities and establish DW standards. Architectural success drivers illuminated by this study validate the hub-and-spoke DW approach and data mart-centric promoters. Indeed, although technology has advanced to support various data mart-oriented approaches, these approaches are proving less effective. Pressures for direct user access and extreme scalability are borne out in the survey results.

Further, IT/LOB conflicts are being exacerbated by uncoordinated approaches, poor expectation management and underestimated data quality challenges.

Early DW adopters were encouraged to experiment broadly with various architectures, technologies, team compositions and LOB involvement schemes. But project sponsors now require that DWs be treated as a mission-critical information supply chain component. To mitigate risks of DW failure and elevate chances for success, project teams must be guided by aggregate results and insights from previous DW projects.