In-Depth
1010data Makes Big Splash with Big Data
The NYSE uses 1010data as the back-end driver for NYXdata, a customer-facing site that gives subscribers access to historic NYSE data
There's been an explosion in the number and variety of Big Fast Data solutions, starting with data warehouse (DW) appliances -- such as Netezza Inc., the former DATAllegro Corp., and Dataupia Inc. -- continuing with quasi-appliances (such as products from ParAccel Inc., Kognitio, and other vendors that market either software- or software-and-hardware-based offerings), and even encompassing unconventional software-only offerings, such as the "correlational" data store marketed by illuminate Inc.
It's a teeming segment, and one in which new vendors are continually making splashes of their own.
Consider 1010data, a provider of high-performance analytics for Big Fast Data customers in the finance, retail, and pharmaceutical segments. The company isn't by any means an unknown, but it certainly made a big splash last month when it announced the expansion of a relationship that it first notched -- six-and-a-half years ago, according to company officials -- with the New York Stock Exchange. 1010data's columnar database back-end -- which is available in either software-as-a-service (SaaS) and on-premises (as a "managed service") flavors -- powers an NYSE warehouse that adds about 750 million new records daily. The NYSE today exposes several revenue-generating services based on its 1010data innards -- including both raw data feeds and Web-based applications. What's more, it uses 1010data as the back-end power plant for its NYXdata Web site, a customer-facing site that gives subscribers access to historic NYSE data.
In the NYSE's case, says David Frankel, a vice-president at 1010data, it's a question of facilitating fast access to an enormous (even mind-boggling) amount of historical data. 1010data, which was founded eight years ago, has been working with the NYSE since about 2001, and launched a production version of its SaaS offering -- which consists of Tenbase, a columnar database backend, and 1010Vision, its ad hoc query and analysis front-end tool -- six years ago.
"We started off with a small project [with the NYSE] and have been continually growing from there to where we are today. It [1010data's technology] was not purpose-built for the New York Stock Exchange. It was purpose-built for large scale data sets of the sort that the NYSE does have, but was not purpose-built for the NYSE in particular," Frankel says.
"From our inception, we've been delivering this [SaaS] model, and we've had a full-featured Web client application [on the front-end] for analysis, so we were probably a little ahead of the curve," he continues. It's a Big Fast Data curve that Frankel feels is trending 1010data's way: "Nowadays, with the acceptance of new data warehousing technologies in general, and … a lot more buzz about accepting software-as-a-service -- these are two fundamental shifts in the industry that have worked to our advantage."
Frankel cites a teeming DW marketplace -- comprising traditional relational database management systems, appliances, quasi-appliances, and non-traditional software solutions -- that he says grew up around one problem: how to rapidly and efficiently query enormous volumes of data. It's the result of several interrelated trends -- including enormous advances in processing power and storage capacities; regulatory compliance requirements and/or data retention (or data safeguarding) practices; and a growing push among business users to be able to access and query bigger and bigger volumes of historical data.
It's here to stay, according to Frankel. "Folks have tremendous amounts of data, billions of records of data, and what we're saying is, simply place it on the 1010data platform," he explains. "What we're advising [our customers] to do is to actually take their broad transactional information and load it into the [Tenbase] system -- to actually translate that as a practice."
He cites the example of big retailers, which typically rely on summary data (in place of voluminous detail data). In the 1010data scheme, Frankel says, retailers not only can but should bring all of that detail data into the warehouse.
"[Retailers] built these data marts and data warehouses with [summaries such as] sales by SKU by store by day, sales by SKU by store by week, and they have this information in summarized form. It's summarized a million different ways; it's summarized for different kinds of user communities. This is the way they've been going about building their data mart and data warehouse platforms," he says.
"What we advise you to do is to take your raw TLog -- your transaction log, the output of your point-of-sale systems -- and feed that into the warehouse. It has what every person purchased, at the most granular level of detail. Every [retailer] has that data, but most of them are not able to effectively do much with it. If you put that the 1010data platform from there you can go anywhere."
Big Data is just one problem. Facilitating pervasive access to Big Data is quite another issue. It's a problem most organizations -- and almost all of 1010data's customers -- are grappling with, according to Frankel. The rub, he says, isn't just facilitating pervasive access -- i.e., embedding analytics or exposing analytic insight to custom-built or third-party applications that are not nominally "BI" or performance management (PM) tools -- but instantiating and exposing analytic insights in the first place. That's just what the NYSE is doing -- exploiting 1010data's XML query capability to expose analytic functionality in portals, Web applications, and other not-strictly-BI-or-PM tools.
This isn't a case of NYSE exploiting any integration "special sauce," Frankel stresses: instead, the NYSE's programming teams are doing what they do best: exploiting the Tenbase warehouse's well-defined XML interface to actively embed analytics in their own in-house development efforts. As a result, the NYSE has been able to launch several new revenue-generating services.
"At the NYSE, they have people who know the business metrics analyze the data and they arrive at some Eureka! moment where they say 'Here are the numbers people need to look at.' At that point, they're able as business users to tell their Web portal, 'Put this out, publish this to our portal users' -- and they can even attach a price tag to it. For example, anyone can run this for $15, $30, or $100 a shot, however their pricing model works internally. Then it's published to their community of external users that can now have access to this analysis," he explains.
"The idea is that the creator of a program exposes certain parameters … that you can change -- say, you can make adjustments from 15-minute increments to 30- or even 45-minute increments -- or you can substitute a group of stocks or, of course, a [given] time frame. As far as the user is concerned, you might expose these [parameters] via a familiar interface -- say, certain 'knobs' or 'tuning knobs' -- but the complexity is hidden from them."
A hosted service such as 1010data's would seem to appeal to business users, who -- owing chiefly to IT's inherent inertia -- value the ability to quickly implement and get started. For most of 1010data's existence, that has been the case, Frankel acknowledges. At the same time, he stresses, IT isn't deaf to the promise of what 1010data has to offer.
Thanks to Big Fast Data and other trends, IT is increasingly warming up to 1010data and its managed (i.e., on-premises) flavor, too.
"We always thought that we would be talking to business people … and when we created the company, that was the thought process. Our strategy was to talk to the business people, and since they can get running without a lot of capital expenses, we figured they'd be quick to adopt it," he says. "We expected that we would get resistance from IT, because they may view this as a threat, but we've encountered actually a number of scenarios where … we've actually worked with the IT folks directly, and in some cases have been brought in by IT," Frankel concludes.
"That's a more recent phenomenon … due to this gradual building of the acceptance of the SaaS kind of delivery method in the market place. The market forces out there, the direction [in which] the industry is headed … is very helpful to us, and it's opening the eyes of IT."