In-Depth

Taking Out the (Data) Garbage

CRM and BI struggle with bad data problems

• By Drew Robb
• 04/02/2003

“Lack of attention to data quality issues is one of the biggest reasons for failure of Data Warehousing and Business Intelligence efforts,” says Gartner, Inc. (Stamford, CT) research director Ted Friedman. “Over the next three years, more than fifty percent of data warehousing efforts will fail or not achieve the desired results due to lack of attention to data quality issues, and the same applies to CRM and ERP.”

Friedman cites these principal sources of data errors:

• End users not following procedures designed to ensure accurate data
• Errors made in converting data from one format to another
• Inaccurate data coming from external sources
• Lack of synchronization between distributed data stores
• Rapid growth in data volume

Poor quality customer data costs U.S. businesses an estimated $611 billion per year in postage, printing, and staff overhead according to Gartner. This is just the iceberg’s tip. The real damage comes from revenue lost by losing track of, or alienating, existing and potential customers. Added to this is missing the potential value of mining that data to guide strategic decision-making.

“Business Intelligence is the best example of ‘garbage in - garbage out,’” Friedman continues. “If you try to make important decisions based upon data that is faulty and of poor quality, you cannot hope to get good answers.”

Cleaning Up Your Act

One aspect of ensuring timely and data quality is using data cleansing tools such as Ascential Software Corporation’s (Westboro, MA) Integrity XE, Innovative Systems, Inc’s (Pittsburgh, PA) i/Lytics or Firstlogic, Inc.’s (La Crosse, WI) Information Quality Suite. As a core function, these tools clean up mailing lists or customer databases by searching for and correcting duplicate or incorrect addresses, checking for misspellings or spelling variations. The products also include the ability to identify mis-categorized data (a company is classified as an individual, for example) and apply custom rules to ensure data entered matches the business needs. Data quality software integrates with existing CRM, ERP, BI, and other enterprise software pages.

Implementation times vary depending on the degree of customization needed, and the number of other applications with which the software integrates. A simple installation can be accomplished in a few weeks, but complex modifications might stretch it out beyond six months.

Too few organizations are taking advantage of this software. While it is a growing area, in 2001 companies only spent about $120 million in data cleansing licenses.

Taking Ownership

No matter how good the data quality tools are, however, that is only one aspect of ensuring an organization has the data it needs.

“It requires more than tools,” says Friedman. “Data quality is a strategic business issue that requires the heavy focus and involvement of business units, not just IT, to make an improvement.”

This means creating a business culture that understands the importance of data and sees it as a competitive asset, not just a bunch of database entries.

“The biggest challenge we face is getting people in the business areas to recognize that ‘data’ is an essential part of what they produce, not just a by-product of the process,” explains Brad Bergh. Currently an Information Architect for DoubleStar, Inc. (West Chester PA), Bergh has spent the past three decades helping dozens of firms including as UPS, NBC, Dupont and Citibank improve their information flows.

This involves re-engineering and streamlining business processes to reduce the chance of errors at the source. Then there is the ongoing task of making sure the data stays current and accurate. People have to take responsibility for the accuracy of the data in their areas.

“Taking ownership requires that they know who uses the ‘data’ that springs from their process and what decisions are affected,” says Bergh. “Only then can they take the steps to ensure that the data coming from their process is usable.”

There is no silver bullet that will instantly solve all data quality problems. It takes tools, and it takes work, but once it is firmly in place, your company can start to realize the full potential of Business Intelligence and CRM.