In-Depth

Creating the Storage Utility – The Ultimate in Enterprise Storage

• 11/01/1999

Imagine storage that is always available, on demand, like the water that comes from the faucet or the electric power that flows from any wall socket. When an application needs more storage, an administrator need only allocate more with a few simple keystrokes. Better yet, how about a system that recognizes when the application needs more storage and automatically allocates the additional capacity? When additional storage capacity is added, it need only be plugged into the pool; the system discovers its presence and allocates it as necessary. High-availability failover and backup occur automatically, transparently.

This is the enterprise storage utility – storage that is delivered throughout the enterprise as transparently, as seamlessly, as reliably as the municipal utilities that we generally take for granted. The enterprise storage utility doesn’t exist today, but many of the pieces necessary to create a storage utility are here, and more are in the development pipeline. It will not be long before the enterprise storage utility emerges as a practical reality.

The enterprise storage utility is a self-diagnosing, self-healing, policy-based, intelligent storage system that dynamically allocates and re-allocates a virtual pool of storage as needed and automatically handles saving and backup. The actual physical storage may be distributed or centralized, or a combination of both. From the standpoint of the storage administrators and users, the storage will appear local, regardless of where any given piece of it is physically located.

Standards are a key to the enterprise storage utility. Servers and storage devices will use a standard network interface connection (SNIC) card and standard cable (fiber or copper). Standards will enable organizations to add storage devices and servers, mixing and matching components from multiple vendors, without the interoperability problems encountered today. The enterprise storage utility will recognize new servers and storage devices and incorporate them into the environment.

Policy-based, intelligent, automated storage management will lie at the heart of the enterprise storage utility. The management system will recognize applications and data that have been designated through an administrator-defined policy for high-availability, and transparently set up and manage the appropriate RAID, clustering and mirroring within the virtual storage pool. It will automatically perform backup and archiving based on policies established by the administrators.

The enterprise storage utility will allocate storage capacity, again based on policies and rules defined by the administrators. It will automatically re-allocate storage as necessary to ensure sufficient capacity where it is needed. As such, it will optimize the organization’s storage resources, eliminating situations where one server or application is constrained by the lack of storage, while other servers or applications have surplus storage capacity. Through policy-based management combined with electronic commerce, the storage utility could even be directed to automatically purchase additional storage electronically under certain pre-defined conditions and within specified guidelines when more storage capacity is required.

The pieces of the enterprise storage utility are starting to come together. Specifically, high-availability clusters, fibre channel and storage area networks (SANs) provide the core storage and communications components. High-speed, high-capacity RAID storage arrays, tape drives and automated tape libraries already are here, proven and widely accepted as storage elements.

High-availability clusters, for both UNIX and Windows NT platforms, ensure access to stored data even in the event of a failure of one server in the cluster. Server clusters are likely to play an increasing role in enterprise computing and storage.

Fibre channel provides a high-speed, long-distance standard communications connection. It allows organizations to connect servers to storage and connect distributed storage into virtual storage pools while providing high performance. Fibre channel will become the standard interconnect for the SAN.

The SAN is emerging as the storage pool that will form the basis for the enterprise storage utility. SANs decouple storage from specific servers, allowing for the creation of virtual storage pools and the dynamic allocation of storage among multiple servers and applications. In the past, storage has been accessed through a specific server.

With the SAN’s decoupling of storage from the server, any server or application can connect with any storage device within the SAN. Data can be transferred, copied or mirrored between storage devices directly. Backup can occur between disk arrays and tape drives without an intervening server. In addition, backup occurs within the SAN without impacting the corporate LAN with heavy backup traffic.

As these components mature, the goal of the enterprise storage utility comes tantalizingly close. Within the next year, the first rudimentary storage utility implementations should be practical with enhancements rapidly following.

The biggest challenges at this point are vendor interoperability, scalability and the need for sophisticated policy-based management software to direct the system. There are also a number of cultural and organizational challenges to address.

Interoperability. In the ideal enterprise storage utility, organizations can simply plug in servers, disk arrays, tape devices and other storage components without concerning themselves with the particular vendor. They can mix and match servers, NIC cards, controllers and disk arrays from multiple vendors in plug-and-play fashion.

Such plug-and-play interoperability, however, is not possible in today’s storage environment. Customers are rightfully wary about trying to combine components from different vendors. Even where standards exist, differences in vendor implementations of the standard can interfere with interoperability.

The problem is compounded in today’s multi-vendor (heterogeneous) system environments. Organizations need to leverage the investment they have made in existing storage, which means being able to incorporate existing storage capacity into the enterprise storage utility, as well as adding new storage capacity.

Interoperability, however, is arriving. The industry is coalescing around standards for fibre channel and SANs. Vendors are offering fibre channel adapters for their hardware. Differences between vendor implementations of standards are being resolved through alliances and vendor-initiated certification programs. In such cases, leading vendors typically work with third-party partners to deliver interoperable multi-vendor storage solutions. Over time, these efforts and the public standards initiatives will resolve the interoperability issues for fibre channel and SAN. SCSI interoperability, now a given, went through similar growing pains in its earliest years.

Scalability. Scalability is a matter of configuring hubs and switches with increasing numbers of ports. Today, the largest fibre channel hubs and switches sport 16 ports. They must be daisy-chained together when more ports are required.

The addition of each new switch complicates the configuration and testing. Again, switch and hub vendors will introduce larger switches and hubs as the demand increases.

Management Software. Maybe the biggest challenge remaining is effective policy-based storage management software that can handle an all-encompassing enterprise storage utility, including both primary and secondary storage. This software must be policy-based, which enables management by exception. The software, in effect, will handle all routine events based on policies specified by the administrators, who intervene only to handle the exceptions. In practice, the software will monitor all storage-related events and performance and automatically perform the necessary administrative tasks and make appropriate adjustments transparently to both users and administrators.

The storage utility management software will also be predictive. This means that it will monitor storage performance and usage for the purpose of anticipating failures. The software will identify disk drives and other components likely to fail, based on their performance or behavior and alert administrators of the need for maintenance or replacement before the component in question actually does fail. Such a system may automatically generate trouble tickets and even order replacement parts.

Cultural/Organizational Issues. The enterprise storage utility changes the way organizations plan for and distribute storage.

While IT has traditionally owned and operated storage for the central applications and databases, each department has owned and managed its own servers and their attached storage. With the storage utility, departments will continue to own and manage their servers, but the storage will be handled by the central storage utility group, most likely a part of the IT organization.

For the departments to give up control of their storage, the managers of the enterprise storage utility will have to convince departmental managers of the reliability of the storage utility. This is a classic early-adopter situation. Over time, departmental managers will see the reliability and benefits of the storage utility experienced by other departments and join in.

In the central data center, the storage utility will change the nature of the workload. It will reduce or even eliminate many of the routine administrative tasks associated with storage, such as backup, enabling IT staffers to focus on more value-enhancing activities.

For the CIO, the storage utility solves the vexing problem of forecasting, planning and budgeting storage costs. It provides a mechanism that will eventually enable the CIO to understand and present storage as a metered per-unit service, much like municipal utilities forecast, meter and bill for water or electricity.

As organizations look beyond the challenges facing the storage utility, they will begin to see the benefits for both the users and administrators.

Maximum optimization of the storage resources. Allows the dynamic allocation and re-allocation of storage capacity where needed, ensuring that storage capacity is not sitting idle, while elsewhere the enterprise is experiencing shortages of storage capacity.

Maximum leverage of storage investments. Flexibility and openness ensure that organizations can incorporate both existing storage resources and future storage resources in the enterprise storage utility.

Efficient storage management and administration. Intelligent policy-based management automates routine storage management tasks and enables management by exception, allowing a small number of administrators to handle increasingly large volumes of storage by focusing only on those problems that demand their direct attention.

Ease of scalability. Allows organizations to scale storage easily and quickly through plug-and-play capabilities.

Transparent high-availability and disaster recovery. Policy-based management automates the appropriate levels of high-availability, backup and archiving to ensure the necessary levels of high-availability and disaster protection.

Efficient purchase of open storage components. Through standards and vendor-certified interoperability, organizations can mix and match storage components from multiple vendors, allowing them to take full advantage of a robust, competitive storage components market.

Ease of use. Users of storage experience storage-on-demand, as transparently as they receive electricity, without concern for the details of where storage resides or how backup and other functions are handled. Storage performs as if it were locally attached.

Predictable, measurable costs. Managers are able to better calculate, forecast and meter storage usage and costs as the enterprise’s storage resource is managed as a single, logical pool.

In the end, the storage utility delivers the best of both centralized and distributed storage. Organizations experience the performance benefits of physically distributed storage and the management efficiencies of logically centralized storage management.

Organizations cannot and need not implement an enterprise storage utility overnight. Rather, it will emerge in stages. Initially, organizations will partner with leading storage vendors who have the resources both to deliver major components of the storage utility on their own and partner with others to provide the remaining components.

Organizations will lay the foundation for the enterprise storage utility by moving toward a fibre channel storage infrastructure and begin a phased deployment of SAN technology. As SAN technology proves itself, more servers and their attached storage can be incorporated into the SAN, creating a growing storage pool.

As policy-based storage management software emerges, organizations will be able to logically centralize storage and automate most storage operations through intelligent, policy-based management utilities and applications. This will give storage administrators the ability to ensure high reliability of the storage resource.

About the Author: Mark Lewis is Director of Engineering, Storage Products Division at Compaq Computer Corporation.