In-Depth
BI: The Year in Review
It wasn't exactly the best of years, but – all things considered -- it was a far cry from the worst.
It wasn't exactly the best of years, but – all things considered -- it was a far cry from the worst. There's a sense, in fact, in which the tumult of 2009 seemed tailor-made to test the very value BI claims to bring to the table.
When the going gets tough, proponents like to claim, it's a safe bet to double down on business intelligence. Although it won't completely inoculate you against the effects of adverse business conditions, BI certainly provides a measure of insulation. The events of this year put that claim to the test.
Hold Fast and Thrive
Given the economic outlook and the abundance -- some might say overabundance -- of vendors in the data warehousing (DW) market, the stage seemed set for a massive bloodletting, particularly in the analytic database arena.
By this time last year, after all, Oracle Corp. had announced its most ambitious high-end DW entry to date (the Oracle Database Machine), Microsoft Corp. had picked up prominent analytic DW player DATAllegro Corp. (announcing plans to roll that technology into a high-end flavor of its SQL Server database), and the analytic database segment was itself teeming with vendors -- Aster Data Systems, Dataupia Inc., Greenplum Software Inc., Infobright, Kognitio, Netezza Inc., ParAccel Inc., and Vertica Inc. were thought to be enmeshed in a Malthusian struggle for resources (i.e., customers), to say nothing of recognition.
On the whole, however, 2009 saw more vendor winners than losers. Winners came in the form of companies that held their own (by treading water or by shoring up their positions in the midst of choppy market conditions) or new players that made solid (if unspectacular) splashes.
The long-awaited bloodletting in the analytic database segment, for example, didn't come to pass. Yes, Dataupia had an especially rocky 2009, but -- in spite of predictions of its soon-and-inevitable demise (reported here and elsewhere) -- it remains a viable, if skeletal, entity. Founder and inaugural CEO Foster Hinshaw even came back (following a successful battle with illness) in November.
Meanwhile, an already-teeming ecosystem of analytic database players buzzed with new claimants: Kickfire Inc. (a developer of DW appliance systems), Calpont Corp. (developer of InfiniDB, another analytic database entrant), Groovy Corp. (an Australia-based analytic database start-up), Exasol AG (an analytic database auf Deutsch), VectorWise (a Dutch analytic database start-up with a formidable academic pedigree), and VoltDB -- the latter a start-up founded by data warehousing luminary Michael Stonebraker -- all burst on the scene this year.
On top of this, BI or DW newcomers such as Compact Solutions, VDO Software, Wherescape Inc., along with a host of new software-as-a-service (SaaS) BI players, also conspired to keep things interesting.
Consolidation of an Altogether Different Kind
All wasn't sweetness and light in 2009, however.
A BI industry long accustomed to unchecked growth a reality check this year. Call it a new (and perhaps unprecedented) spin on BI market consolidation: culling.
To the extent that there was shrinkage of any kind in the combined BI and DW space over the last several years, it occurred largely as a consequence of consolidation. Best-of-breed players such as Crystal Decisions Inc., Firstlogic, and Outlooksoft -- along with BI giants Hyperion Solutions, Business Objects, and Cognos -- were gobbled up by bigger vendors. Probably the single biggest scare faced by any BI vendor in recent memory was that of the former Brio Software, which -- by all accounts -- warmly welcomed a timely buy-out overture from Hyperion. That was six years ago, an eternity in business and technology time.
In 2009, the unsparing economic climate claimed at least one BI industry victim when LucidEra, an ambitious SaaS start-up, shut its doors. The end came quickly, at least in light of LucidEra's near-four-year history. In late spring, founder Ken Rudin started quietly shopping for a white-knight buyer. Finding no takers, LucidEra went belly up in early summer, a victim (insiders confirmed) of an especially chary venture capital (VC) climate.
For a while, it seemed as if Dataupia, too, might succumb to the same VC troubles. Things came to a head in late June, precipitated in part by LucidEra's implosion. Several industry watchers speculated that Dataupia would almost certainly be next. By year's end, however, reports of its demise looked to be premature: founder and CEO Hinshaw retook the leadership role in November; concomitant with Hinshaw's return, the company claimed it had secured new funding, too. Dataupia isn't in the clear: in addition to reassuring jittery customers and keeping pace with aggressive competitors, it must replace both its CTO -- respected DW technologist John O'Brien -- and its veteran VP of marketing, Samantha Stone.
Another player that surprised observers was Microsoft Corp., which pulled the plug on its PerformancePoint Server product in late January. At that point, PerformancePoint had been shipping for just 16 months.
To be fair, Redmond didn't completely pull the plug: PerformancePoint's scorecarding features will get a new lease on life in the upcoming SharePoint Server release. Still, it was a surprising setback for Microsoft, which had once waxed enthusiastically about taking budgeting, forecasting, and planning mainstream. Moreover, Redmond had been tremendous successful with its BI-oriented offerings: SQL Server Analysis Services, SQL Server Integration Services (nee Data Transformation Services), and SQL Server Reporting Services were all smash hits. With two SQL Server-oriented offerings on tap in 2010 -- viz., Projects Gemini and Madison --Microsoft hopes to resume its winning streak.
Consolidation remains a hugely important force in the combined BI and DW segment. This year, for example, the industry said goodbye to Sun Microsystems Inc., steward of the former MySQL AB and prominent partner to a pair of analytic database players (Greenplum and ParAccel). Other casualties of merger and/or acquisition include SPSS Inc., GoldenGate Software, and HyperRoll Software.
Sun was both the first to fall and -- as of press time -- is the last to close: the European Commission (EC) still hasn't approved Oracle's acquisition bid.
SPSS fell next, gobbled up in July by IBM Corp. Both SPSS and rival SAS Institute Inc. had achieved prominence by leveraging their strengths in statistical analysis and data mining. In just the last decade, both had emerged as creditable BI and analytics players too. IBM's acquisition of SPSS leaves SAS -- with its more than $2 billion in annual revenues -- as the Last of the Independent Statisticians.
On the whole, it was another extremely busy year, acquisition-wise, for Oracle – especially on the BI front. In addition to Sun and data integration specialist GoldenGate, Sun also tied up a legacy loose end, nabbing HyperRoll, a vendor that first gained fame (and invited litigation) by marketing an OLAP accelerator technology for Hyperion's Essbase engine.
Today's Trends
A number of notable trends either came to the fore or stayed in the fore in 2009. Here are some of them:
A Dream (Still) Deferred
BI vendors have been talking up "pervasive" business intelligence for about a long time. To be truly pervasive, BI must achieve widespread adoption outside of its traditional silos and -- more often than not -- deliver measurable business impact. BI has consistently failed to make the grade in both respects.
In 2009, laments BI tools expert Cindi Howson, a principal with BIScorecard.com, BI didn't gain much, if any, ground.
"Since our initial survey in 2007, success rates and BI usage are largely unchanged," writes Howson in the 2009 edition of her Successful BI Survey. "While there are some very successful BI deployments, the majority are stuck in the middle, with only slight to moderate success and business impact."
In fact, Howson concedes, the percentage of employees using BI actually dipped this year, slipping by 1 point.
More Data, More Problems
In 2008, DW vendors started waxing enthusiastically about the mainstreaming of multi-terabyte data warehouses. It sounded far-fetched -- the sweet spot for data warehouse configurations is in the double- or triple-digit gigabyte range, after all -- but 2009 saw an undeniable uptick in multi-terabyte (or "Big Data") activity.
This year we had Big Data pushes on the parts of both Microsoft -- which announced a Fast Track SQL Server DW program in February -- and Oracle, which unveiled its mammoth Exadata Version 2 in September.
Similarly, 2009 saw a surge of interest in technologies such as MapReduce and Hadoop (to say nothing of Google Inc.'s and Microsoft's next-gen takes on MapReduce, dubbed Pregel and Dryad, respectively), which proponents claim are tailor-made for Big Data and Big Problems of Scale. To wit: analytic database player Aster Data Systems sponsored a Big Data Summit in New York; Cloudera -- a commercial Hadoop firm -- likewise convened the first ever Hadoop World, also in New York. Meanwhile, a pair of established heavyweights (Netezza and Teradata) seemed to warm up to MapReduce, too.
Next year promises more of the same. On tap: SQL Server R2 Parallel Data Warehouse, otherwise known as Project Madison, Microsoft's effort to retrofit SQL Server 2008 with massively parallel processing (MPP) capabilities. It's slated to ship sometime next year.
Assuming it completes its acquisition of Sun, Oracle could deliver still another Exadata refresh; Teradata, Netezza, and other pure-play analytic competitors have promised to deliver much more in terms of both speed (chiefly via solid state disk drives and flash cache modules) and capacity (thanks to ever-expanding aerial densities).
Finally, new entrants will emerge to push the Big Data envelope still further.
Columnar is Hot
This year, players seemed to warm up to the idea of the columnar store as the repository of choice for analytic requirements. This actually cut both ways, with several row-based vendors touting come-to-columnar deliverables, even as at least one columnar player (Vertica) trumpeted the availability of a row-based capability in its flagship product.
Oracle is the most prominent columnar convert. It introduced Exadata-only support for column-based compression in Oracle 11g R2 this September. Elsewhere, analytic database stalwart Greenplum unveiled a columnar implementation of its own, the dauntingly dubbed Polymorphic Data Storage. Finally, columnar player Vertica debuted a new row-based capability called FlexStore as part of its Vertica 3.5 platform refresh this summer.
Expect more on the columnar front in 2010. Industry watcher Curt Monash, who actively tracks the analytic database space, notes that columnar specialist VectorWise has teamed up with Ingres Corp. (the commercial steward of the open source Ingres database) to develop what amounts to a row/column hybrid. It's slated to appear sometime next year, according to Monash.
Analytics Reloaded: BI and the Post-Analytic Age
Tired of plain old analytics? This year, vendors started talking up "advanced analytics" as the latest, greatest, and most promising spin on analytic technology to date.
Forget about the data warehouse-driven reporting or OLAP applications to which your boss has long been partial. Advanced analytics prescribes the use of extremely complex (often SQL-driven) queries or best-of-breed predictive analytic tools. IBM says advanced analytics also entails the top-to-bottom reorganization of a company's existing business processes; such was Big Blue's pitch with the Business Analytic Optimization (BAO) service it unveiled in April. IBM, as a matter of fact, bet huge on analytics in 2009: all told, it launched BAO, spent $1.2 billion for SPSS, and unveiled a "Smart Analytic" appliance based on RISC/Unix hardware and Cognos software, with middleware assists from DB2 and WebSphere.
There's a sense in which BI has already entered the post-analytic age. According to TDWI Research, nearly 40 percent of shops are currently practicing advanced analytics. They're just getting started: by 2012, says TDWI research analyst and veteran industry watcher Philip Russom, fully 85 percent of organizations will be doing as much.
Vendors are already brushing up on their advanced analytic talking points. Netezza, for example, says it's preparing a big advanced analytics push for next year; and at its Partners user conference this October, Teradata Corp. discussed advanced analytics in tandem with analytics powerhouse SAS. Look for more on the advanced analytic tip in 2010. Microsoft hasn't yet weighed in.
Final Observations
Although some BI and DW players had previously flirted with cloud computing, this was the year in which BI-in-the-Clouds really soared.
This year we saw a downpour -- a cloudburst? -- of cloud-related offerings, with enhanced packages from bleeding-edge adopter Vertica and brand new offerings from Aster Data Systems, IBM, Oracle, Teradata, and others.
Also this year, software-as-a-service players such as Birst, Good Data, Oco, and PivotLink started talking up the inevitability of SaaS BI. Most of these vendors also argued that the demise of SaaS BI pioneer LucidEra -- which went out of business in July -- was more a function of a brutal economic climate than a referendum on the SaaS BI model; industry watchers noted.
Finally, this was the year in which free and/or open source software (F/OSS) arguably went mainstream, at least with BI buyers. About 25 percent of all companies were using F/OSS BI tools in 2009 according to BI and DW consultant Mark Madsen, who conducted a survey of F/OSS BI usage.
What's more, Madsen found, fully 40 percent of small shops and nearly one-third of large organizations were in the process of evaluating F/OSS BI offerings.
The inescapable upshot, according to Madsen, is that F/OSS BI has at long last crossed a threshold -- into respectability.