Main-Memory Databases
You're building an application that needs to move many transactions quickly between the DBMS and the end user. The application might be an interactive Web site or an e-commerce application that can support thousands of simultaneous users. Initial benchmark testing indicates that traditional databases choke on the transaction load you expect. What are you going to do? One option is the newest twist on database management systems: main-memory databases.
The big difference between a main-memory database and a traditional RDBMS is that the former keeps its data completely memory-resident. The primary DBMS files are not stored on disk. This eliminates the performance drag caused by the long time -- for a computer -- it takes to locate a specific block on a disk, seek the head to that track and wait for the disk to rotate under the head.
The main-memory database has no need for buffers, read-ahead algorithms or any of the other optimizations that DBMS designers created to work around the slow pace of hard drives. In addition, this type of database has no need for B-tree structures to search for data. Instead, the main-memory database can be optimized for direct access to data using a specialized memory structure called a T-tree. These systems are blindingly fast compared with disk-based databases, and memory is increasingly inexpensive. So why isn't everyone using one today?
There are a number of potential problems with this approach that must be considered. One is the issue of data persistence. When a traditional DBMS crashes, database files are still physically intact -- unless the disk itself crashes -- and the system can often automatically recover the logical integrity of the database using a variety of before-image, after-image and transaction log files. The recovery process ensures that all committed transactions are available once the database is brought back up.
If a main-memory database system crashes, however, you've completely lost the database. But many vendors have come up with a solution. They have created a disk-based data store and have added the ability to capture only the transaction-related data in log files. This transaction-specific information can be saved to disk in serial fashion before a transaction is regarded as committed. If the system crashes, the database can be read in from the data store and the transactions reapplied until it is back to its previous state. There is some overhead associated with writing the log files, but it pales in comparison with the overhead involved with two or three disk seeks for every transactions. To minimize recovery time in the event of a crash, the system can be configured to write a main memory checkpoint image to disk at regular intervals.
Another big question has to do with the size of the database. If a database is confined to main memory, it can be no bigger than the amount of memory in the machine. In fact, it should be less than that since some memory has to be allocated for other uses, such as the operating system and client connections.
Both Angara Database Systems (www.angara.com) and TimesTen Software (www.timesten.com ) deliver 32- and 64-bit versions of their main-memory databases. The 32-bit version is limited to a maximum of about 2 GB of memory. For many applications, such as an interactive Web site, that may be sufficient. If your application will store larger volumes of data, these vendors also supply 64-bit versions that are capable of storing essentially unlimited amounts of data. But if you need the 64-bit version, you'll have to use a Unix platform. With the advent of Windows 2000 and the Merced chip from the Intel/Hewlett-Packard consortium, we may see this type of product available in the NT marketplace within the year or two.
The bottom line is if you have an application that has high database throughput requirements -- thousands of transactions per second -- you should definitely consider one of the main-memory database products that are out there today. --Robert Craig is vice president of marketing at WebXi Inc. (Burlington, Mass.), and a former director at the Hurwitz Group Inc. Contact him at rcraig@webxi.com.