In-Depth
A Tea Party in Storage?
The V-party may be a grassroots movement energized by widespread anger against the predatory behavior of storage and system vendors, but it seems to have followed the pattern of many movements -- technological and political.
Take this any way you want: For corporate computing, VMware fans are beginning to resemble what the Tea Party has become for U.S. politics. Whether you are favorably disposed toward the Tea Party movement and its impact on elections or governance or you don't like what they represent, the analogy holds -- especially following the announcements last month at VMworld concerning the strategic direction that the hypervisor vendor's engineers are envisioning for storage.
Let me be clear. Fundamentally, the "V-partiers" are giving voice to some fundamental truths about storage. The first of these is that the present storage model is a mess and has been for a long time. The storage industry status quo, as discussed (and decried) here in great detail over the years, has mostly seen the triumph of marketecture over architecture.
At VMworld, V-party spokespersons called out the storage industry for fomenting a silly debate between NAS and SAN, among other things. I quite agree with them on this score, though probably for different reasons. As I see it, NAS and SAN are the same thing: direct attached storage. Neither topology is actually networked at all -- SANs being more or less direct-attached storage with a physical layer switch, and NAS being storage again directly attached to a server, albeit one optimized for delivering data in the form of shared files.
That said, the "pointless NAS/SAN debate" that the VMware engineers held up as an example of storage industry dysfunction is really a non-issue. It is rhetorical "ear candy" -- and has been for almost a decade. I know of no IT planners who care about the phony distinctions that the industry has set forth through their own marketing, the insights of their paid "industry analysts," or the ruminations of their echo chamber in the press or blogosphere. For the majority of consumers, the whole SAN vs. NAS discussion harkens back to a time when we had more money than common sense to throw at storage requirements -- and paid way too much for gear that sported a certain kind of three-letter descriptor.
In most of my conversations with CIOs and planners today, the real challenge is to service I/O and to store an increasing data glut with fewer administrators, fewer hard-to-come-by watts, a smaller capital budget, and, ideally, using a build-out model that has predictable performance and cost.
Bottom line: the V-partiers can invent slogans around NAS vs. SAN debates all they want, but the root causes of storage woes are more nuanced. Vendors have gone out of their way to bolster revenues from otherwise commodity hardware by "adding value" to array controllers in the form of services or functions contextualized as beneficial to the consumer.
Although advances have most certainly occurred in array controller functionality in terms of services that need to be handled close to disk (RAID and RAGS, and perhaps support for sub-LUN tiering, just to name two), most value-add claims are truly questionable. This is especially the case when the value-add service offered by the vendor is isolated to its own hardware rigs rather than delivered across infrastructure as a whole. Thin provisioning is such a value-add service: it might make sense if you have only one array, but its value case fails the smell test when the number of arrays increases or your infrastructure is provided by more than one vendor.
In these more common situations, doing thin provisioning the way that the folks who invented it, DataCore Software, deliver it -- across virtualized pools of storage via a storage hypervisor -- makes more sense from an economic and technical perspective. Forget about HP/3PAR's latest thin-provisioning-in-ASICs approach, it does little more than build in obsolescence into their platform while enabling them to charge a premium for the commodity components of the rig. Software processes should be done in software, abstracted away from the controller, and delivered to all of the capacity you have deployed rather than to a single stand of disk.
It is especially dissembling of the hardware vendors to tout benefits of value-add software embedded on array controllers when the very presence of the software interferes with the ability to manage the rigs in common with all of the other gear deployed in an infrastructure. XIO gets this, with its notion of universal management via open standards-based REST APIs. Although VMware also uses RESTful management, it hides it from direct access by infrastructure technology providers, requiring them instead to use proprietary API front ends. This is a very important factoid that V-partiers conveniently ignore.
A second big issue with storage that V-party advocates do not address is the cost for value-add software licenses and warranty/maintenance agreements. These charges are, in a word, insane. Renewing a brand-name storage vendor's warranty and maintenance agreement after year three typically costs as much as paying for the entire rig a second time -- and that's for gear that was "end-of-life'd" by the vendor within 17 months of selling it to you.
Contemporary storage follows a classic "razor/razorblade" model: you think you are getting a great deal on your hardware, looking at the price cuts achieved through vigorous bargaining with your sales reps, but the hardware is actually just the razor: cheap to buy. The real cost comes in the form of the razor blades that must be purchased after the hardware has been bought: the razor blades in storage include annual software license costs, warranty and maintenance agreement renewals, the cost of overpriced "signed" drives that must be added when expanded capacity is needed, downtime owing to failed controller software patches and firmware updates, etc.
Most IT planners must deal with this situation, which the V-partiers, whose idolized company is 80 percent owned by the biggest storage hardware vendor on the planet, never mention or even seem to contemplate in much detail. In fact, they seem downright disinterested in this practical problem.
Their "movement," as the V-party evangelists have taken to describing the VMware sales and marketing blitzkrieg, is guided by "common sense" and quasi-constitutional principles such as "improved resource utilization efficiency," "application portability," and "holistic manageability." These are all great goals and may well be part of a greater vision for storage sanity. What the evangelists are really delivering, however, is a return to a past computing model in which a single vendor determined the rules for all others.
They might be wise to review the literature of the late 1970s and the anger of an earlier "tea party" that coalesced against centralized mainframe computing. The dominant vendor in those days, IBM, basically told any vendor of "peripheral devices" who wanted a connection to their backplane that the price to play was compliance with a set of proprietary de facto rules. The result, many argued back then, was a very reliable compute platform but one purchased at the expense of healthy competition and innovation.
Today, the V-partiers are pursuing a model that is very similar to the single vendor dominance model of 40 years ago. Truth be told, their vision of building a "storage hypervisor" in their server hypervisor software seems to reflect nostalgia for a time that never actually existed ("the era of perfect mainframe computing platform"). More to the point, they want to base their new order on a technology stack that in no way delivers the stability, security, or manageability that once existed (and continues to exist today) in mainframes.
IBM mainframes had PRISM and other technologies, developed over a 30-year period, to create highly stable logical partitions that were well-suited for application multi-tenancy and that insulated each guest machine and workload from all others. These are capabilities that VMware lacks. Simply put, those who invested in mainframes, for all their warts, got a more robust and resilient platform than those who buy VMware today.
This observation may be debated by the V-party aficionados, but it is not debatable that the same voices who are arguing that VMware must bring order and goodness to the world of storage are also arguing for VMware to become the king of the technology stack overall. They telegraphed this strategy last year, when VMware released APIs that modified the standard SCSI command stack, without going through the ANSI vetting process.
Last year, VMware announced that it had arbitrarily modified the ANSI standard SCSI language by inserting additional and arbitrary commands that would offload some I/O functions to select storage arrays beneath the server layer ("select" being those who conformed their own offerings to comply with the new and unvetted SCSI primitives). Their goal was to address the I/O congestion seen in many VMware environments by 20 percent by offloading certain data-movement tasks to arrays.
They released the first version of these primitives with no advance warning to the storage industry, creating a frantic scramble among many vendors to engineer compliance into their controllers so as not to alienate a small but growing customer base that had embraced the hypervisor. A few months later, VMware repeated this debacle with the release of version 2.0 of its VAAI API. Again, there was no advance notification given to storage vendors about the release, which had the effect of triggering yet another frantic engineering push in companies that wanted to represent their storage as "VMware ready."
The problem was not just the hassle of revising storage firmware twice to agree with ad hoc changes to a foundational ANSI protocol, though this drove storage engineers crazy, but also the grindingly slow process for obtaining approval and certification from VMware for the engineering work that had been done.
Said one storage vendor's chief technologist in a recent interview, "Don't get me started about VMware. We have been in a queue for more than six months waiting for a VMware engineer to be assigned to us in order to get our hardware certified with the new APIs. They don't have any idea about what they are doing or what it costs the storage industry."
Even 1970s IBM did not cause such pain for peripheral equipment vendors who wanted to plug and play with their mainframes back then. More to the point, some industry observers have asserted that EMC, VMware's majority shareholder, was not briefed in advance about the ad hoc changes that its stepchild was making to SCSI. So much for any temporary advantage that might have accrued to their hardware products!
If this example is any guide, it suggests that the V-party isn't trying to fix the problems of storage so much as it is trying to become the "alpha dog" in the enterprise computing stack -- the new IBM in the old mainframe model. The pursuit of such a self-serving goal is certainly their prerogative, but it doesn't quite match their rhetoric.
The mantra originally advanced by VMware was improved resource utilization efficiency, but their wares actually made I/O less efficient. They wanted to break the application/server hardware relationship, abstracting software away from hardware, then their VAAI API, which offloaded certain functionality to select hardware providers -- established certain vendors as most favored while queuing up others for review and approval. Now, VMware seems determined to drive the storage industry to its knees by taking everything away from the array controller, even those functions that seem to need to be done close to the disk, and fielding it instead as an additional set of microkernels in an already microkernel-intensive software hypervisor.
Absent is not only a concern about what this does to the storage industry but what it does to the consumer. Assuming that everyone does not embrace VMware for every workload -- perhaps sticking with non-virtual servers for some workload, or choosing Citrix Xenserver or Microsoft Hyper-V for other workloads -- the new and improved VMware compute environment represents a terrible inconvenience. Customers with mixed workloads will need to set up a separate network and storage infrastructure for VMware operations, separate and isolated from their non-VMware environment. That is rather like the situation that existed for almost 30 years in which distributed computing was separately managed and operated from the mainframe data center.
The question is: does any of this actually make things better?
The V-party may be a grassroots movement energized by widespread anger against the predatory behavior of storage and system vendors, but it seems to have followed the pattern of many movements -- technological and political. What begins as a movement with a valid critique to offer of the status quo is too often institutionalized into something far worse than the problems that led to its creation.
Your comments are weclcome: jtoigo@toigopartners.com