In-Depth

The Staggering Scope of Big Data

The amount of data we're all generating at any given time is astronomical. Jump on the big-data train or risk getting left behind.

• By Lee Pender
• 04/29/2014

So you've been hearing about storage lately, maybe a lot. Well, that shouldn't come as too much of a surprise. Storage is a hot category for a lot of reasons, but the main driver of demand for storage is something that has now become a familiar buzz phrase: big data.

Simply put (and it's scary to think that anybody in IT wouldn't know this) massive data growth is driving a panic scramble for storage solutions. But how massive is this massive growth in data volume? It's incomprehensible actually, or nearly so. 

Let's take a look at some of the numbers that make up big data. A summary from Villanova University offers a few numbers:

• Users create 2.5 quintillion bytes of data every day. Essentially, this means that 90 percent of the data in the world today has been created in the last two years alone.
• Retailer Walmart alone controls more than 1 million customer transactions every hour -- and then transfers them all to a database that stores more than 2.5 petabytes of information
• There are 45 billion photographs (and counting) in Facebook's database. That's more than six photos for every human being on earth.

But wait...there's more. According to Domo, a business-intelligence firm, every 60 seconds, technology users:

• Send 204 million e-mails;
• Upload 3000 videos to YouTube;
• Tweet 100,000 times;
• Download 47,000 applications from an app store.

And those numbers are already old. They were old the first time somebody typed them. So, are we humans just that much smarter than we used to be? Do we just know that much more? Maybe, but one of the drivers of big data is the storage of increasingly massive file types -- think MRIs and sonograms.

Then there are government regulations, social media, the proliferation of online video, streaming-video services, blogs about storage... The stuff we're storing is just bigger, broader and greater in volume than it has ever been before. That's all there is to it.

So, what does this all mean? Well, for one thing, now would be a good time to acquire skills in working with big data. There will soon -- within the next few years -- be significant shortages of IT people who know how to handle big data. And then there's the investment that big data will require. That brings us full circle to our discussion of storage. All of this stuff has to go somewhere and be accessible and recoverable -- and soon.

It's not that IT professionals don't know all this. It's just that the numbers provide a stark reminder of the onslaught of big data that's happening right now and a harbinger of what's to come. Get ready.