In-Depth

The BI Revolution: Business Intelligence's Future

We've entered the third major generation of the BI market. Learn how it's fundamentally different than its predecessors.

• 11/10/2010

by Brian Gentile

A new age of business intelligence is here. We've entered the third major generation of the BI market and it is fundamentally different than the first two generations preceding it.

The first generation was characterized by general-purpose, off-the-shelf tools for reporting and ad hoc query and analysis of structured data using SQL and standard database connectors. These desktop-oriented tools gave way to server-based BI functionality and the rise of the data warehouse, which marked the beginning of the second generation of BI. That second generation continued strongly through at least 2003. In that year, Business Objects bought Crystal, a major consolidation of the BI sector began, and all BI players scrambled to add a stronger Web appeal to their then-aging architectures and approaches.

Today the rules and technologies have changed. Can existing BI vendors reinvent themselves a third time, or will this change give rise to new players that can advance the long-held benefits of business intelligence where the leaders of earlier generations left off?

BI Past and Present

The old BI platform wars are over and the winners have emerged. Business Objects, Cognos, and Hyperion grew faster and to greater mass than all the other BI platform companies combined.

The success of these original BI platforms came from capitalizing on the first- and second-generation needs of the early BI customer while building products based on the prevailing technologies and architecture popular at the time. As a result:

• Scaling to accommodate more users and more data meant buying higher performance desktop computers and single CPU server systems with greater memory and more directly-attached storage so that more desktop clients would experience optimized query run times; and
• Data being analyzed was always structured and relational (and probably normalized), the few users were sophisticated (e.g., power users and analysts) and they either made individual decisions to determine "what happened" in the business or created analyses for someone else to do so.

As Geoffrey Moore points out, this data center-centric stack was the design point for these first two generations of BI, as it was for all enterprise applications at the time. Today, however, major forces are transforming the design point and development options in the enterprise, and a new network-centric BI stack is quickly emerging.

BI Market Forces

The backdrop to these major market forces is the significant consolidation that occurred among major BI providers. SAP's acquisition of Business Objects, IBM's acquisition of Cognos, and Oracle's purchase of several BI providers (including Hyperion) consolidated more than half of the BI market into these "mega-vendors." When markets consolidate, a vacuum of innovation results; we've witnessed some of the most interesting BI inventions occurring in smaller, independent vendors during the past few years.

At the same time these large vendors were consolidating, critical technology shifts clearly changed the market. Fundamentally, the Web has become the central design paradigm and so a fully Web-based architecture is necessary to offer simpler installation and faster, more flexible deployment options. Customers today need a BI tool that can be installed on-premises in the morning and running in the cloud in the afternoon.

A modern design should natively allow new features now required to solve the most current BI problems. Such new features include in-memory analytics, collaboration and process-orientation, search capabilities, embedability/extensibility, mobile client access, and the ability to reach well beyond structured relational data sources to Hadoop, Hbase, Hive, and NoSQL, among others.

As if these technology shifts weren't enough, additional market force is applied by the genuinely higher interest in "competing on analytics." Tom Davenport's book, Competing on Analytics: The New Science of Winning (2007, Harvard Business School Press), chronicles the rise of and need for greater analytic regimen in practically every organization. Much recent research reaffirms the continued strategic importance of BI projects to both the business units and CIO.

To reach "everyman analysts" within a business, new reporting and analytic techniques are required that deliver faster, simpler access to data using Web-based, consumer software-like approaches. The new BI end user expects this kind of software behavior and won't settle for anything less. According to recent TDWI research (BI on a Budget), we know that the key cost-related BI drivers to reach this audience include self-service BI improvements, fast and inexpensive dashboards, and capabilities that better enable the success of "Super User" networks within an organization. (Super Users are important because they make IT professionals more efficient BI implementers within business units.)

Ultimately, BI becomes used pervasively by being embedded in business applications, delivering analytic insight from within a process-focused or functionally oriented system. In my last BI Revolution column I highlighted this analytic application trend – which is the new network-centric BI stack in practice.

BI Platform of the Future

The now-known limitations of the first two generations of BI platforms coupled with the market forces yield the requirements for the business intelligence platform of the future. Ideally, a new and necessary BI platform architecture and feature set must solve for all these modern needs. The table below summarizes some of the stark differences between past BI platforms and what is needed today (and tomorrow).

Key BI Platform Characteristics -- Then and Now

Categorically, a new breed of BI platform must satisfy stringent new requirements across six important categories.

1. Open, Web-Based Architecture: Built on Web 2.0 standards and designed purely for the Web world, the architecture must remain lightweight, modular, and simple to enable faster and less-costly deployments while delivering a great end-user experience using the latest Web technologies. Having an open architecture means that the source code can be reviewed and extended whenever necessary to enable a great BI application that reaches end users quickly and easily. Also, because this architecture expects nothing more than a thin client Web browser as its end point, the ability to include a wide variety of client devices is virtually assured.

2. Agile Deployment: Working equally well on-premises, in a virtualized container, serving multiple tenants in a SaaS mode or scaling elastically in the cloud, this new BI platform should deliver agilely. Today, organizations may want to start a new BI solution in one deployment mode and then switch to another. Such movement should not trigger re-design or a substantially new project to accomplish. Instead, the modern architecture (see first category) allows this movement seamlessly.

3. Flexible Data Access: The volume, variety, and velocity of data today requires flexibility in a BI platform to accommodate new reporting and analysis needs of broad user groups. Ideally, older file formats and techniques would be easily supported (non-relational structures such as hierarchical or index) along with typical departmental formats (most importantly MS Excel). Solving for higher-volume, distributed data formats may be among the most important new requirements because of the volume of Web-based data, often referred to as "big data." This means support for Hadoop, HBase, Hive, and other distributed file techniques that are growing quickly. As I've previously written in a BI Revolution column, these big data types are critical for a modern BI platform to support.

4. Timely Decision Focus: Relevant data is accumulating in much greater volumes and the need to understand what's happening in the business (at the moment) has never been greater. It is no longer sufficient to know "what happened" yesterday or last week. Organizations must tap into the real-time clickstream and decipher the information elements that help make better decisions now. Further, these decisions are made collaboratively, by not just power users but experienced end users (business managers) who work together to improve business processes and outcomes. Therefore, the BI platform must support timely data gathering (and information creation), and collaboration processes designed to speed decision making whenever and wherever it must occur.

5. Embeddable, Extensible, and Customizable: Because so many new BI applications are built around the automation and improvement of a business process or business function, reporting and analytics must become part of a bigger application. In this case, the modern BI platform must be built so that it can easily become the "BI inside." Ranging from low-level architectural capabilities (to enable single sign-on security or multi-tenant delivery) to the visual interaction framework at the presentation layer, the BI platform should sport easy-to-access customizable capabilities (modifiable using open standard approaches). This elegant extensibility enables it to transparently deliver sophisticated reporting and analytics without getting in the way.

6. Affordable Scalability: Scaling out is key, and for good reason. Elasticity is important in any type of deployment (cloud or not) because data volumes and business projects are not always predictable. The ability to easily add CPU power in small increments (standard PC servers), memory where and when it is needed, and network-based storage capacity without having to completely reconfigure the BI server is imperative.

Furthermore, the modern BI platform architecture should allow highly scalable end users (and types) because the heavy lifting is done in the network of servers that provide essential infrastructure services for the BI platform. This infrastructure includes compute, memory, and storage services; application and Web servers; as well as security, load balancing, and even back-up/recovery. In total, the BI platform should inherit and take advantage of the full network-centric software stack emerging in the enterprise today, at no extra charge. Done correctly, a single, modern BI server should deliver reports and analysis to hundreds or thousands of users (using a variety of client devices) simultaneously. That's affordable scalability.

The Future is Now

A modern BI platform can solve business problems previously out of reach because of insufficient project ROI. In other words, second-generation BI tools were too expensive and complex to efficiently use for a wide variety of business problems. This equation is radically changed with a third-generation BI platform.

Take, for example, Revol Wireless, which is creating distinct competitive advantage by widely implementing a modern BI platform. Revol's Strategic Data Warehouse enables data consolidated from a variety of application systems to become valuable, fact-based information on which better decisions are made. Revol delivers pure Web-based access to reports, dashboards, and drill-through analyses to enable executives, sales managers, and store managers to improve their management of marketing campaigns, customer churn, and customer service. According to George Mehok, CIO at Revol, "We're visibly moving the company strategy forward. We've built a strategic asset, and we're a better company because of this work."

This type of fast success and assured ROI will become common as the third generation of BI platforms becomes the new standard in organizations. Now that's a BI future we can all look forward to.

Brian Gentile is the chief executive officer of open source business intelligence software company Jaspersoft. You can contact the author at [email protected].