In-Depth

Business Intelligence: Metcalfe’s Law and Data Marts

• By Jeff Gentry
• 05/01/2001

Practically everyone has heard of Moore’s Law, which states that the processing power of CPUs doubles every 18 months. However, there is a less-known law that might soon become popular – Metcalfe’s Law. Simply stated, Metcalfe’s Law says that the potential value of a network is proportional to the square of the number of nodes in the network. Metcalfe’s Law is v=xn², where v=value, n=nodes on the network and x is a constant value.

This law applies to many different things. A few automobiles would not be very valuable if there were not networks of roads to interconnect them with possible destinations. A single telephone without access to the telephone network is worth exactly the scrap value of its components. When the same telephone is placed in a network with many other telephones, it has great value in combination with the rest of the network. Just as telephones and automobiles become increasingly valuable as their networks expand, data marts also grow in value to the enterprise as the size of their "networks" increase.

To apply Metcalfe’s Law to data marts, we must understand what makes up a network of data marts. The characteristics of a data mart network are similar to the characteristics of a computer network: shared topology, shared network protocol, transport protocol and interconnectivity. Data marts must exhibit these network characteristics to reap the benefits of Metcalfe’s Law.

Networked data marts must be connected by a data warehousing topology. The two most prominent data mart topologies are federated data marts and dependent data marts. Though there is much debate concerning the strengths and weaknesses of these two approaches, the key observation relevant to the value of networked data marts is that both topologies prepare and provide data to all data marts in the topology, in a common way. Just as with a network of computers, the hardware and software that support the topology must interoperate to support the movement of data within a network of data marts. Data mart topology includes extract, transform and load (ETL) tools and any other capabilities that mobilize source data.

A layer above the topology, networked data marts must share a common protocol. The common protocol for data marts is dimensions for dimensional data marts and reference data for relational data marts. Data marts must have conforming dimensions or reference data in order to work together. Relational data marts that capture the results of statistical models or economic projections can work seamlessly with dimensional models representing the same business measures and qualifiers, as long as the data is semantically consistent. Shared metadata ensures consistency between network data marts.

From a purely business perspective, business processes must combine information across networked data marts to add value to the business. If two data marts are remotely related within the business processes, any break in the chain creates inconsistency in the information used to manage and lead the enterprise.

In summary, data mart nodes on a shared data warehousing network must have: shared data mobilization, conforming reference data, conforming dimensions, semantically consistent reference data and dimensions, shared metadata and business processes using multiple data marts.

To establish the value of networked data marts, simply apply Metcalfe’s Law. Consider an enterprise that has deployed 50 data marts. For the purposes of this discussion, a data mart is a single star schema, cube or relational schema designed to address a set of related business questions and deployed in a database. Based on Metcalfe’s Law, if these data marts are networked, their value is: xn², or 2500x.

The value of x will vary and is certainly subjective, representing the value of a single data mart with independent data mobilization, nonconforming dimensions and reference data, and no shared metadata. For purposes of demonstration, suppose that the average value of such a data mart is $25,000. Fifty such data marts would be worth $1.25 million to the business. However, if the same 50 data marts are networked, they are worth 50 times as much, or $62.5 million. The next networked $25,000 data mart, the 51st, will add another $252,500 to the value of the enterprise business intelligence capability.

So, is the effort of coordinating data marts across the enterprise worth the $61,250,000 value? The answer to the question is obvious, yet many enterprises continue to build independent data marts outside a shared data mart network. Is there a place for an independent data mart? Certainly, but they are the rare exception for the unprepared organization, not the rule.

The next time the temptation arises to build an independent data mart that is not in the enterprise data mart network, remember Metcalfe’s Law … and do the math.