In-Depth

Analysis: IBM Rediscovers Itself

TDWI Research's Wayne Eckerson examines news from IBM's briefing on Smart Analytics.

• By Wayne Eckerson
• 07/30/2009

Editor's note: This article previously appeared (in a slightly different form) at Wayne’s World, TDWI Research director Wayne Eckerson’s blog.

On Tuesday, I had doubts about the value of driving from Boston to New York (eight hours roundtrip) to attend a short IBM briefing on Smart Analytics, but thankfully IBM didn’t disappoint, at least in the end.

SPSS Acquisition

The Non-Announcement

The briefing consisted of two announcements and one non-announcement.

The non-announcement was that IBM acquired leading analytics vendor SPSS for $1.2 billion (see separate story at http://www.tdwi.org/News/display.aspx?ID=9548). Oddly, the acquisition wasn’t the focus of the Smart Analytics briefing I was attending, as I had assumed once I saw the press release. In fact, as I learned later, it was a coincidence that the SPSS announcement occurred on the same day as the Smart Analytics briefing.

This was reinforced by the fact that the IBM software executives (Steve Mills and Ambuj Goyal) didn’t say much about the acquisition other than it would “embed the technology across our platform.” What’s strange about that statement is that IBM had a great data mining product called Intelligent Miner which it discontinued as a standalone product several years ago and embedded its functionality inside DB2 and other applications, so they basically bought what they already had or still have. Odd.

Anyway, after five minutes discussing its head-turning non-announcement, IBM then turned to the real focus of the briefing, which was to announce the Smart Analytics System and the Smart Analytics Optimizer. The first is a new twist on an old yarn, but the second is potentially groundbreaking.

IBM Smart Analytics System

InfoSphere Redux?

To understand the Smart Analytics System, you have to know what IBM means by analytics. To IBM, analytics is Cognos 8 -- which in reality is query, reporting, dashboarding, and some OLAP. It’s not data mining or predictive analytics (things that SPSS does), although I suspect that will evolve in short order.

What I gleaned in one-on-one meetings after the briefing (IBM briefings are traditionally long on concepts and short on details) is that the IBM Smart Analytics System is IBM Infosphere Balanced Warehouse done right, with Cognos 8 embedded as an optional add on. At least this is the first instantiation of the Smart Analytics System. Eventually, IBM will embed other components from its vast array of ever expanding software products into the Smart Analytics System, including presumably SPSS, as well as ETL, data quality, archiving, and cubing.

It will also expand the platforms supported (beyond the IBM Power platform currently) and the number of out-of-box configurations (currently, based on data volumes: 4TB, 12TB, 25TB, 50TB, 100TB, and 200TB), so from a systems perspective, it’s not all that new.

Radical Reworking

From a customer and IBM perspective, Smart Analytics System represents a fundamental overhaul of how IBM goes to market and meets customer needs. For the first time, customers can purchase a vertically integrated IBM system -- from hardware and software to tools and applications to consulting and support services -- with a single SKU and have the system configured, tested, and shipped in as little as two weeks with a single point of contact for support. IBM will then periodically perform health checks to ensure performance still meets customer requirements.

Back to the Future

If you have some history in the high-tech market, you might remember that this is similar to the way IBM operated prior to the early 1990s. If you were an IBM customer then, you bought vertically integrated systems to run your core applications and were a “big Blue” shop through and through.

With the advent of Unix and open computing, IBM began losing market share, and to ward off financial ruin, it brought in CEO Lou Gerstner. Gerstner decided to blow apart IBM’s business and systems models. To overcome the stigma of selling closed, proprietary systems in an open world, Gerstner created autonomous divisions for hardware, software, tools, and services and gave them permission -- in fact, exhorted them -- to partner with all players, including fierce competitors of the other divisions.

In short order, IBM consultants began recommending non-IBM hardware and software, DB2 began running on non-IBM hardware, and IBM hardware began running non-IBM chips, operating systems, and databases. It worked financially, saving the company, but it came at a price.

Making IBM Easy to Do Business With

The price is that IBM has become increasingly difficult to do business with. In fact, from a customer perspective, IBM looks and feels like multiple companies rather than one company. This increases the costs, complexity, and time to value when deploying IBM solutions. To keep legendary IBM executive Tom Watson from rolling in his grave, IBM now is rediscovering its roots and providing vertically integrated systems and a single face to the customer. To do this, IBM is more closely aligning its product divisions, which is perhaps the biggest upshot of the announcement.

IBM’s attempts to reduce costs, complexity, and time to value through the Smart Analytics System resonated with a panel of IBM customers at the briefing. Ironically, none had implemented a Smart Analytics system or had even heard of it until the night before, but all said that it would have saved them a lot of time, money, and headaches.

IBM has rediscovered what worked prior to the 1990s (vertical integration) but without losing what has worked since (horizontal integration). IBM will need this new hybrid model to compete in an increasingly competitive BI marketplace where established vendors and upstarts are selling vertically integrated solutions, some using open source and the cloud to radically alter the rules of the game.

Smart Analytics Optimizer

Purpose-Built Analytical Database?

Perhaps the more interesting announcement was the Smart Analytics Optimizer, whose details I teased out in a one-on-one meeting with Arvind Krishna, vice president of enterprise information management products. I wanted to ask Arvind when IBM will get analytics religion and build or buy a purpose-built database optimized for query processing. Before I could, he rattled off a long list of limitations of traditional transaction databases when performing queries (e.g., ACID properties, logging, indexes, etc.) and the tradeoffs of new query-processing architectures (e.g., in memory, columnar, cubes, parallelization, etc.) . He finished by saying, “What if we do it all? Only better?”

That got my attention.

In Memory Sandbox

It turns out that Smart Analytics Optimizer, which will ship in Q4 for IBM System z mainframes, provides a super-fast query database inside a transaction processing system. The Optimizer lets you store a copy of highly trafficked query data in up to 1 TB of main memory and uses compression, vector processing, and a proprietary IBM data format to accelerate query performance by a factor of 10 or more, Krishna said. From there, IBM’s cost-based optimizer knows whether to direct queries to the in-memory database or the transaction database to get the best performance. Today, IBM consultants will help companies figure out which data to allocate to memory but in the future this task will be automated.

Best of Both Worlds

If I heard Krishna right, the Smart Analytics Optimizer gives customers the reliability and availability of a transaction system and the performance of a purpose-built query processing system -- but all in one system. In other words, customers don’t need to offload queries from a transaction system or offload analytics from a data warehouse. They might only need one system for transactions and analytics! This saves plenty of dollars compared to offloading queries to a stand alone analytical system. Eventually, the Smart Analytics Optimizer will be ported to other IBM systems and support upwards of 50TB of memory, Krishna says.

Last But Not Least

If what Krishna says is true, then IBM has figured out how to make an end-run around the spate of new analytic database vendors in the market today and finally has a response to Oracle’s Database Machine and Exadata Storage Server as well as Microsoft’s Project Gemini, both of which turbocharge their flagship database management systems with purpose-built analytical add ons.