In-Depth
BI: The Year in Review
For BI professionals, it was a year in which several long-simmering trends seemed to coalesce and boil over.
Despite the tumult in the global financial markets, 2008 was relatively calm for BI professionals. In contrast to 2007, when the three biggest BI pure-play vendors (Hyperion, Business Objects, and Cognos) were acquired by larger, non-BI vendors, the year past was a sleepy one. There were the requisite acquisitions, to be sure, but nothing comparable to the domino-toppling wave of consolidation that swept the industry in 2007.
There was a hot and fractious DW appliance market that, almost from the beginning of the year, was whipped into a frenzy by a new batch of upstart players, including Aster Data, ParAccel, Vertica, InfoBright, and Kognitio. These companies were founded prior to 2008 -- Kognitio, in fact, can credibly claim to be a DW appliance veteran -- but all had an eventful year, announcing prominent partnerships and, in some cases, customer wins. The DW appliance segment gained newfound legitimacy this year, too, thanks to interest from established powers, including Sybase Inc., Teradata Inc., and Oracle Corp., all of which launched dedicated DW systems.
That helped keep things interesting. So, too, did the aggressive maneuverings of Microsoft Corp., which -- never one to neglect a burgeoning market -- acquired one of the most prominent DW appliance players (the former DATAllegro Corp.), launched a BI-friendly SQL Server 2008 database, and disclosed plans to develop a version of SQL Server optimized for data warehousing and analytics.
Elsewhere this year, Microsoft, Oracle, IBM, and a host of new players -- particularly purveyors of columnar analytic technologies -- were more serious about tackling a not-so-new problem: poor ad hoc query performance. This year might well be remembered as the year in which established DW and BI powers, DW appliance players, and analytics vendors maneuvered to address analytic issues of a new and almost staggering scale. Indeed, a growing desire to use analytic technology against ever-greater volumes of data, by ever-growing numbers of users, helped fuel much of the activity in the DW appliance segment, the traditional DW space, and the BI tools arena.
All in all, it was an interesting and blessedly tumult-free year.
Interest in DW Appliances Grows
It was a year of surprising events, including acquisition activity (Microsoft buying DATAllegro), new product entries from non-traditional players (e.g., Sybase and Teradata), and partnerships between appliance vendors and BI players (e.g., Sybase and MicroStrategy; Dataupia and Tableau Software).
It was also the year in which interest in data warehouse appliances finally took off. It’s been a slow time coming. Consider DW appliance specialist DATAllegro, which Microsoft acquired in July. According to Philip Russom, a senior manager with TDWI Research, DATAllegro had only a handful of customers -- probably fewer than five. That’s in spite of the fact that DATAllegro had been shipping product for more than three years at the time of its acquisition this summer.
The performance of appliance pioneer Netezza -- which became a publicly-traded company just 17 months ago -- underscores the point. When it first went public, Netezza was an unprofitable entity. It was so unprofitable, in fact, that it had lost money for four years running. That changed in 2008, when -- with slightly more than six months of public-trading under its belt -- Netezza posted its first-ever profit. (That was for Netezza’s fiscal year 2008, which ended in January of this year. Because of other charges, Netezza still lost money.)
Since then, Netezza posted profits in the first three quarters of fiscal 2008.
That span coincides with the emergence of a second wave of DW players -- companies such as ParAccel, Vertica, Aster Data, and InfoBright, which (along with established players Netezza, Dataupia, Kognitio, Greenplum, and others) helped make 2008 a particularly interesting data warehousing year.
Relational database powers Oracle and Microsoft were worth watching this year, too.
In September, Oracle announced its own DW appliance-like deliverable, the Oracle Database Machine, a combined hardware and software solution, sold by Oracle, serviced by Hewlett-Packard Co. (HP). The Oracle Database Machine is powered by a grid of Oracle database servers that distribute queries to Oracle’s HP-powered Exadata storage systems. In this sense, it was unlike any other appliance or DW hardware system on the market. (Industry watcher Curt Monash, a principal with Monash Research, described Oracle’s approach as “node heterogeneity.”)
Regardless of how you describe it, the Oracle Database Machine is undeniably intriguing, although being an Oracle-powered offering that runs only on HP hardware will probably limit its adoption. For combined Oracle and HP customers, however, the performance boost promises to be nothing short of jaw-dropping: a fully-outfitted Oracle Database Machine configuration can deliver as much as 72 times the performance of a vanilla Oracle data warehouse, according to officials
Microsoft Flexes Its Muscles
In 2007, Microsoft took a tentative step into higher-end data warehousing, announcing prepackaged SQL Server systems in tandem with hardware partners HP and Dell Computer Corp. This July, Redmond took the high-end data warehousing plunge, snapping up DATAllegro for an undisclosed sum.
It’s expected that DATAllegro will help shore up SQL Server where it’s weakest -- in DW configurations over 20 TB.
Microsoft flexed its DW muscles in other ways this year. Just two months ago, for example, Redmond announced an upcoming version of SQL Server -- dubbed “Kilimanjaro” -- that it plans to deliver sometime in the first half of 2010. The software giant also announced plans to develop a new in-memory capability for its Excel spreadsheet (dubbed “Project Gemini”), which will effectively put a hot-rodded in-memory analytic workbench on a user’s desktop. Finally, Microsoft announced “Project Madison,” its attempt to ingest the assets of the former DATAllegro.
All of what Microsoft disclosed is still hypothetical. But Redmond’s announcements suggest that it is newly (and perhaps even deadly) serious about high-end data warehousing. SQL Server Kilimanjaro, Project Gemini, and Project Madison were the biggest news items to come out of Microsoft’s second annual Business Intelligence Conference, which convened in early October.
Kilimanjaro, for example, is designed to deliver on Microsoft’s (not-entirely-visionary) notion of “pervasive business intelligence,” and it calls for a heaping helping of user self-serviceability. Concomitant with this, Microsoft officials outlined a vision of what they called “grab-and-go” reporting. “In Kilimanjaro, we're going to take [reporting] much further, and focus on taking [Report Builder] 2.0 and adding new capabilities to it so that it's really an end-user tool in … [that] it's easy to use and it's intuitive, but also in terms of … people [can get] up and running with it quickly,” said Herain Oberoi, group product manager for SQL Server with Microsoft, in October. "Instead of having to build their own charts and data, [end users] could just search for a repository. They could just drag and drop that chart on to their report. If I'm the end user, I don't have to worry about binding the data set -- it's all done for me.” Project Madison, like SQL Server Kilimanjaro, has a relatively lengthy lead time. It should ultimately produce a massively parallel version of SQL Server, which -- theoretically, at least -- will let Microsoft take on not just Oracle and its Database Machine, but Teradata, Netezza, Dataupia, and other MPP players. It’s of a piece with an emphasis on bigger and more scalable data warehousing -- chiefly by means of massive parallelization -- that came to the fore this year.
Analytics Everywhere
If Kilimanjaro focuses on usability and self serviceability, Project Gemini is all about analytics. It, too, is of a piece with another key 2008 trend -- what might be called “ambitious” analytics.
At several TDWI events this year, for example, vendors talked up analytic problems of a new and hitherto undreamt scale: no longer were organizations content to put ad hoc capabilities at the fingertips of a half-dozen, dozen, or even several dozen users. Increasingly, customers were asking about the possibility of putting analytics within the reach of rank-and-file users. For this reason, DW systems that were designed to support several dozen business power users quickly ran out of steam once they're exposed to several hundred (or more) rank-and-file users, very few of whom are concerned with the niceties of query syntax, efficiency, or elegance.
Even as organizations open up their DW systems to more users, they're also bringing ever large volumes (and a greater variety) of data into the DW itself. Historical analyses that once involved months or, at most, a year, now consume three, five, or even ten years' worth of data. Moreover, users may need to mix and match data from the usual suspects (inventories, purchase reports, etc.) with non-traditional sources (such as pricing information from competitors). The combinations can get downright dizzying.
Viewed in this context, Project Gemini -- which Microsoft says will take the form of an in-memory, columnar data store add-in for the Excel spreadsheet -- is just one of several ambitious analytic offerings that were unveiled this year. SAS and Teradata kicked things off back in February when they announced an effort to embed SAS analytics directly inside the Teradata database. Moreover, SAS officials stressed, the SAS/Teradata arrangement was by no means exclusive: SAS hoped to partner with other vendors to deliver in-database analytic technology.
A couple of months later, RDBMS stalwart Sybase got into the act. Its venerable Sybase IQ database experienced something of a renaissance over the last 18 months, thanks -- again -- to a growing emphasis on fast analytic performance. Sybase IQ is a columnar database, of course, and columnar databases -- a category that includes full-fledged DW systems from ParAccel and Vertica, as well as Redmond’s forthcoming Project Gemini deliverable (albeit on a much smaller scale) -- have emerged as a popular means to supercharge analytic performance. In May, Sybase announced its first-ever Analytic appliance, a bundle that includes Sybase IQ running in tandem with BI software from MicroStrategy. Its impetus, Sybase officials said, is to tackle the Big Analytic problems faced by a growing number of organizations.
Other ambitious analytic entries include InfoAssist, a new guided ad hoc query tool from Information Builders Inc., and Lyza, a new deliverable from BI start-up that mimics -- or, more accurately, anticipates -- Microsoft’s Project Gemini: Lyza, too, puts a full-fledged columnar store on the end-user desktop.
A more general response to the problem of making analytic technology more scalable and pervasive can be seen in the rise of specialty DW systems. In this respect, the irruption of upstart DW players, the ambitious expansion of Microsoft’s and Oracle’s DW efforts, and even Teradata’s long-awaited entry into the DW appliance segment are all responses to the same problem. 2008 was just the year in which this long-simmering need finally boiled over. What still-simmering trends will coalesce and boil over in 2009? Stay tuned.