In-Depth

New Spin on an ETL Best Practice

New data extraction tool boasts .NET underpinnings, OLE DB, RSS support

• By Stephen Swoyer
• 07/21/2004

There’s more than one way to get data out of mainframe, minicomputer, and other so-called “legacy” systems, and while direct connection—via ETL or some other data integration technology—seems to be all the rage, Datawatch Corp. offers a new take on an old best practice.

For several years, Datawatch has marketed Monarch Data Pump, a tool that exports data from reports generated by ERP systems and other mainframe applications to Excel, Access, or relational databases.

Last week, however, Monarch Data Pump got a next-generation overhaul, with new .NET underpinnings, support for OLE DB, and Really Simple Syndication (RSS) information dissemination. The new version, Monarch Data Pump 7.0, is available now.

“The purpose of Data Pump is really to help a lot of our customers who have large amounts of desktop user licenses, to really help pull data out of legacy systems [such as] mainframes, or J.D. Edwards—anything that generates these huge reports—and really bring that into a controllable workflow,” explains Gareth Horton, product manager for Datawatch’s desktop and server division.

Horton says that Monarch Data Pump Microsoft’s has been completely retooled using Microsoft’s .NET framework. “The reason for doing that is to move on to a new tech, but also to provide a far better grounding” for the product going forward, Horton confirms. For example, he says, Datawatch can exploit .NET as a scripting engine, which “allows us to provide far more power in the pre- and post-exporting steps.”

Similarly, the transition to a .NET basis made it easier to incorporate RSS support into the Monarch Data Pump product: “We could generate an RSS feed which could have certain bottom-line information that would let people skim all of the stuff that they wanted in the RSS feed, rather than have” the complete report “blasted to them.”

Elsewhere, Horton says, the new product’s OLE DB connectivity is a must-have for many customers. “The old product was capable of loading via ODBC to SQL [Server] and Oracle, and also e-mailing reports around, but really, that’s certainly not enough for our customers,” he comments.

So why use a report-parsing tool like Monarch Data Pump when more direct methods are available? Horton argues that it’s much easier to go right to the line-item in the report itself than to search for it a data warehouse. “If you just want to know how many widgets have been sold in a certain zip code, it’s easier to find that in a report,” he says.

For that reason, directly importing data into Excel seems to be the most popular option for most of Datawatch’s customers—although that’s changing, Horton points out. “Excel is really the target of choice, but more and more people are obviously going to the relational database, to database servers or data marts. That’s why the old version used ODBC, but the new one supports OLE DB, which is considerably better for this.”

Reports are imported to a central server, even though end users initiate the import process at their desktops. Once a report is successfully imported, its location can be published (via RSS or e-mail) to all users. From there, entire reports can be broadly disseminated, says Horton: “We have a distribution action that allows you to move things around on the network or over the Internet.”

Users can schedule imports for mainframe or legacy reports that are frequently refreshed, he confirms.

So what’s next? Horton says that Datawatch may tighten ties between Monarch Data Pump and Microsoft’s new SQL Server 2000 Reporting Services. “That’s something that we’re thinking about [for the] future, where we could use Monarch when we create a summary view (which is almost like another report in Monarch); we could maybe write that out to the RSS file, and then we could use the SQL Server we’re using internally as a staging post and integrate it with SQL Reporting Services,” he says.