In-Depth
Files, Files Everywhere
To help IT manage its file deluge, vendors are trying to develop a platform that delivers respectable performance for file access vendors. They're really just building a bigger junk drawer.
Over 56 million hard disks were sold last year, according to those who count such things. Assuming that IDC and other analysts have this one correct, most of these drives will be used to host files.
Files. Some might call them "unstructured data." They are those assemblages of bits used by most companies to conduct the bulk of their business processes. The corporate crown jewels remain databases ("structured data" in industry doublespeak -- file systems that are structured, too), but files are the key drivers of both storage growth and IT pain in the current decade.
File proliferation is partly a consequence of distributed computing and the rise of "office productivity software," and partly a result of everyone's desire to download a private copy of the Internet for fear that something they read, heard, or saw won't be there the next time they go looking. Also contributing to the exponential growth of files is e-mail, which has become the dominant communications vehicle of the corporate world.
With files comes pain. If memory serves, I wrote a column some 20 years ago about one pain point introduced by file proliferation. At the time, I had a problem sharing files between Microsoft/Intel and Sun Microsystems platforms -- an issue I never had in the mainframe environment where I began. I called Redmond for help and the best solution offered was to get rid of all that Sun UNIX stuff. Turning to Sun for help, I got basically the same advice, but in reverse: "Solaris rules, Windows drools."
Since that time, of course, advances have been made in cross-platform file sharing (or, more appropriately, file copy sharing, since no one actually shares original files). In addition, the block storage vendors stepped up to the plate with OS-agnostic storage solutions. The FC SAN folks and sellers of large high-performance rigs optimized for hosting databases advanced technology that enables us to map file systems to their volumes and shares so we can host files in the most expensive ways ever conceived by mankind. These have been joined by a host of file server vendors who represent their wares as "Network-Attached Storage" (NAS), who want you to target their rigs to host the flood of files. Good to know that as the number of files increases in our businesses, storage vendors will be there to help us spend our budget.
Sharing was only one issue with files, as it turned out. Truth be told, it may come back to haunt us as proprietary stacks of hardware and software -- the "mini-me" mainframes offered by VMware-Cisco Systems-NetApp, Oracle-Sun-STK/Pillar, Microsoft-QLogic-HP, etc. -- begin to take hold in shops where planners outsource their brains to their love brands. I can't wait to hear from users of these platforms when they start deploying one from the Cisco team and another from the HP team, then try to share files between them.
Today, another issue is how to host the file deluge that won't bankrupt our companies in the process. I have been watching the industry to see what will be this year's catch-all phrase and I think it might be "horizontally-scalable NAS." In less than a month, we have seen several hints of this. It first appeared on my radar when Exanet filed Chapter 7 in February.
Product Announcements Proliferate with Files
Exanet was developing software to cluster together NAS boxes and deliver an extensible global namespace that would treat the growing physical repository as a coherent whole. Investors had pumped roughly $70M into the firm, but they needed another infusion of funds to achieve their goal. They reached out for support, difficult in an economy that has seen venture capital and investment banking support for storage start-ups fall from approximately one billion dollars in 2005 to about a quarter of that last year, and found Dell. Dell did its due diligence when reviewing the investment opportunity, lacking a NAS clustering/global namespace option of its own, but ultimately decided to let the company fail, then pick over the corpse to buy the intellectual property outright for a paltry $12M.
This wasn't the only acquisition of clustering/namespace assets we've seen in recent years, of course. NetApp acquired clustering companies such as Spinnaker Software a few years ago (and of global namespace software vendors since), but never delivered a successful product to market. Silicon Graphics (SGI) built a very good cluster-with-global-namespace play at about the same time, but management was still licking its wounds from the company's forced downsizing at the start of the millennium and was too timid to compete with NetApp to aggressively push their new wares into the market. Other players in the clustered NAS space were either too poorly funded to have their message heard over the din of the brand name OEMs or too engineering top heavy to market effectively at all.
Bottom line: although the failure of Exanet came as no real surprise, the interest of Dell in buying their IP meant something. Why would a medium- to low-end product purveyor ""be so interested in clustered NAS?
On the heels of the Exanet news came several announcements that are still making my head spin. One was from Big Blue.
In February, IBM announced SONAS (Scale Out NAS, get it?) that is part product, part architecture aimed at providing horizontally-scalable file hosting -- even for z/Linux on mainframes! Mainframes traditionally hosted block database data and treated files as "anonymous data sets" when the mainframe was used to back them up or otherwise manage their storage. The IBM offering seemed to suggest that even in the mainframe world, file storage might be a burden.
Of added interest with this announcement was that IBM was side-stepping its partner, NetApp, to sell a different solution based on Infiniband interconnects and IP networking-based management protocols. (NetApp had announced that most of its largely unsuccessful GX NAS clustering solution was soon to be embedded into its highly successful ONTAP operating system.)
Around the time of 'IBM recent announcement, Isilon announced the latest revision of its clustered file storage platform. This intrigued me as the first intelligent use I had seen of Flash SSD technology. Isilon has added Flash SSD purposefully to its product lines to support file access across its horizontally-scaling file hosting platforms -- not, as many other vendors have done, just to appear fashionable or trendy.
In the Isilon world, mostly unchanging file metadata is placed into a Flash SSD to expedite "file tree walking" and access to discrete files placed anywhere in the growing repository. Their announcement drew criticism from other NAS vendors, including a comment from a NetApp evangelist, "The use of flash as a cache for metadata was also the goal of the NetApp PAM version I cards introduced in 2008. These were the 16GB DRAM variety and acted as a 2nd level CPU cache for ONTAP metadata and user data. The big ticket item -- the biggest performance boost -- came from the caching of metadata. File data storage continues to grow, hence the 512GB flash-based PAM vII cards out now. Flash for metadata caching turns out to be a great use case. For NetApp, a good first step was investment in flash as cache."
Isilon quickly clarified, "To be clear, Isilon's use of Flash is not as a cache for metadata. It differs from NetApp's PAM implementation in that it is focused on accelerating UNCACHED metadata access. The metadata for OneFS lives permanently on the SSDs."
The company spokesperson went on to say, "Caching metadata operations is a no-brainer and is enabled in an Isilon system by its globally coherent DRAM-based cache, which expands as you add nodes into a system. DRAM is far lower latency than flash for cache."
He continued, "The problem that we are specifically solving relates to uncached access, which occurs during tree walks, arbitrary directory/file access, first-access latency, and random I/O -- all of which involve uncached metadata lookups. The only solution to this problem is putting the metadata on a faster medium."
Since file system metadata cached on Flash SSD by Isilon in its latest products is rarely changing data once written, it minimizes the issue inherent with all Flash SSD: memory wear. I have been increasingly concerned that Flash SSD was being added to systems by storage array vendors to capitalize more on the "cool factor" than on any meaningful contribution to array architecture. Let's face it: even with the advances in multi-layer cell recording in Flash memory, the actual useful life of a product is a function of workload. In a high-performance, transaction-oriented environment, deploying Flash SSD to expedite workload may actually find organizations changing out SSDs every few days or weeks because of the volatility of the data being written and overwritten. Not so with the metadata that Isilon is storing on its Flash SSD drives. The low volatility of file system metadata means that the Flash SSD should last longer, maybe even approaching the useful life estimates promised by Flash SSD vendors.
From Cache-Cows to Cash-Hogs
While the industry is working to come up with a decent platform for hosting the file deluge that delivers respectable performance in terms of access on demand to file data, vendors are really just building a bigger junk drawer. Costs to store files are accelerating, partly as a consequence of vendor hardware warranty and maintenance agreements which add obscene after-sale costs to initial hardware purchases, and partly because of a lack of discipline in managing file data itself.
In Dallas recently, I was a guest of storage engineering company Zerowait at a meeting of storage planners for about 20 large enterprise users of brand-name storage gear. To a one, the key question being asked was how to get out from under OEM maintenance agreements.
Zerowait, whose company offers third-party maintenance and support, talked about leveraging qualified third-party support contracts to replace high-dollar agreements purchased at the time of equipment sale then renewed every year thereafter. Even where storage hardware had been deemed "end-of-life" by the original equipment manufacturer, maintenance contract prices remained the same as for new equipment, or grew in cost. Using third-party services from qualified vendors were the best solution, the Zerowait spokesperson argued to a largely receptive audience, suggesting that we may be poised for an uptick in sales of such services.
In Austin, later in the same week, I met with several dozen IT executives at FujiFilm's Global IT Summit. Their interest seemed to go to the real root cause of storage cost: keeping data on spinning rust forever.
Many were at the Summit to hear about advances in tape technology, which they said was their preferred strategy for dealing with the cost of burgeoning file storage. Given the precipitous drop in rates of file re-reference -- accesses to files once written to disk tend to drop off to zero within 30 days, often sooner -- the question repeatedly being raised was simple: why store data on disk at all after that time?
Others winced at a challenge that they see coming in the near term: the need to classify data to manage its cost. This story had two dimensions. One speaker for a large telco said that its prior strategy to de-duplicate data and store it on a large installation of Data Domain hardware had proven misguided and expensive. The Data Domain equipment, the speaker said, had been "gifted" to his company as a drag-along to the EMC storage they were buying for other uses. They took advantage of this gift to host massive amounts of file data but quickly discovered that not all files avail themselves of the same de-duplication ratios. He had to revisit the strategy and to classify data better to send only the data to the platform that would reduce well using the technology. The alternative was to stand up many more de-duplicating arrays with each one having its own separate management requirement -- a labor cost accelerator.
This fellow and others said that archiving files to tape was proving to be a more compelling approach (good news for FujiFilm, which makes LTO tape), but what they went on to say made even more sense. Prior to platforming data at all, the speakers concluded, it was necessary to establish the policies and mechanisms to classify that data at a more granular level -- not just to optimize storage technology investments such as de-duplication or media but to facilitate its inclusion in retention and deletion schemes that would bend the curve of data massing.
Smart folks, those FujiFilm Summit participants.
Your comments are welcome: jtoigo@toigopartners.com.