Three Hidden Costs of Backup
Backup is a necessary evil whose hard costs (including disk, bandwidth, and software) are easy to calculate. Beware these three not-so-obvious costs that can cost IT thousands of dollars.
By Kelly J. Lipp
Backup -- the necessary evil. Present in every IT shop in the world in some form, this necessary activity consumes, in some cases, a disproportionate amount of resources. Human resources and computer resources are expended all in an effort to protect against the unexpected. Then, the unexpected happens and we're wondering if we've done all we can to prepare, and in the worst case, realize that we haven't.
Fundamentally, backup as a concept is simple: make a copy of all of your important data so that in the event the original is lost or destroyed it can be recovered. Add the requirements of recovery time and recovery points and the definition is complete. How we get to the end result varies widely. What doesn't is how its cost impacts its business. There are hard costs and soft costs, but many of the soft costs are often overlooked, downplayed, or ignored. These are the hidden costs.
Hard costs are easy to figure: the cost of the hardware itself, servers, disks, tape drives and libraries, bandwidth to offsite locations, vaulting services, etc. Add the software costs and maintenance on all of this and you know your hard costs.
Some soft costs are more easily determined than others. The ongoing cost of maintaining the backup infrastructure is relatively simple. There are usually a fixed number of people deployed to cover these tasks, which include daily inspection of the backup process to ensure completeness and remediation when something goes awry. Although not as easy to calculate as the hard costs, by using sample burden rates and estimating the amount of time consumed, one can arrive at a reasonable number.
The hidden costs that are not generally counted have to do with architecting a new system or an update to an old one, installing a new or updating the old system, and maintaining a system that includes hardware and software from different vendors.
I will not discuss the costs of application downtime as these are very hard to predict and vary widely by industry. These are actually the most important "costs" but are usually factored into the decision to implement backup in the first place. I believe the hidden costs have a greater impact on the value of the overall system because they often remain hidden!
The hidden costs of backup creep in from a number of areas.
The Hidden Costs of Design
Let's look at the case where we've exceeded the capability of the existing backup product and we've begun the search for a replacement. This happens at some point to most IT shops as we grow or as we add different technology. This is particularly prevalent today with virtualization. As we consolidate from physical to virtual, our backup needs change. At this point we begin a design exercise.
If we consider that most of the backup problems in the world have already been solved at least once, it is not imperative that we re-invent that wheel again. However, our natural tendency is to believe that we can do it better or that our environment is so different that our backup needs require a whole new idea. It is here that significant hidden costs are incurred and ignored! While we are engaged in the design of a new backup infrastructure, we're ringing up costs in two areas: our people involved in the re-architecture and the opportunity costs of not having the new backup system in place sooner rather than later. Again, the re-architecture cost is easier to figure: burden rate multiplied by time. However, the opportunity cost is much more difficult since it involves the potential loss of data.
In the case of a badly broken infrastructure, our risk of an application outage is much higher. If we are unable to restore an application in a timely fashion while we're re-architecting, the damage to the business can be great. It is imperative to replace the broken infrastructure as quickly as possible. The design process is our enemy. Evaluating technology, vendor demonstrations, issuance of RFPs, etc., consume time that we do not have. Our objective should be to upgrade/replace as quickly as possible. Remember, most (if not all) backup problems have already been solved. You don't need to re-invent a solution and contribute to the hidden costs.
As our computing architecture changes, our backup needs change as well. It may well be that our current backup architecture will not support these changes without significant changes or additions. Again, we can waste valuable time here. It may be better to gut the existing system and start from scratch. Our tendency, though, is to simply add another patch to the current system hoping that will carry us through. Like re-inventing a wheel, patching a fundamentally flawed architecture is costly. The thought process is understandable: it will be easier to add technology XYZ than to replace the entire system. This appears logical. However, your past experiences should be your guide. Have other patches you've applied to your system been met with resounding success? Probably not.
Architecting a replacement or an upgrade can contribute to your hidden costs. Beware these costs and minimize them.
The Hidden Costs of Implementation
If we undertook the re-invention of the wheel or decided the patching approach will work, that process is followed by the implementation of our newly designed system or the application of the patch. Significant hidden costs are often ignored here, too, and these costs have the same two cost components as the design phase.
Most of us are provided the opportunity to implement a new backup system once or twice in our lives (with any luck at all). Therefore, we're not necessarily as good at it as we think. While we're attempting to put together our newly designed and acquired hardware and software, burden rate multiplied by time plus the risk factor are adding to our hidden costs. Thus, a lengthy implementation of a new system is dangerous. Remember: the goal is to rapidly implement the system and reduce these hidden costs, especially the risk of an application outage.
Thinking about and implementing either a new system or patching an old one have significant hidden costs that are often not counted. Beware the costs of these endeavors. Fight the tendency to re-invent as much as possible. Fight the tendency of arrogance in believing that you can build a better mousetrap than has already been built.
The Hidden Costs of Maintenance
As we re-architect a replacement system or an upgrade to an old one, we may add another hidden cost factor to our environment. I'll call this the finger-pointing cost.
In the case of a newly designed replacement system consisting of hardware from a couple of different vendors and software from another, the potential for finger pointing creeps in. We'll probably witness this in the implementation phase, and the maintenance phase. Again, there are two components to these hidden costs. Burden rate times time plus risk. While we're arguing with our vendors trying to solve a problem, we're wasting valuable time. If an application is down while we're having this argument, the damage to our business can be significant.
Minimizing these hidden costs is difficult. The very nature of the blended solution using the best-in-class products from a variety of vendors encourages this problem. Our natural tendency to want all of the best working together to solve our problem is our enemy. This circles back to the design phase but has a dramatic impact on the cost of the solution in the long run.
Tried and true would be a better objective. Stay away from the bleeding edge of technology. In fact, choosing a solution from one vendor that provides all the hardware, software, installation, training, and support is likely the best alternative -- especially considering that they have probably implemented hundreds of backup solutions for all sorts of industries and customers. It's likely they've seen a site almost identical to yours.
Do your due diligence. Challenge your vendors to provide multiple references that match your environment as closely as possible. Follow up. Dig into those solutions. If you are pursuing the best-of-breed product mix from multiple vendors, find a customer doing the same thing. Understand the issues related to maintaining this type of environment. Know the players. Know the issues. Know the pitfalls.
Minimizing the Hidden Costs
If you know where the hidden costs are, they're no longer hidden. You can accurately count them. A prolonged design phase, a prolonged implementation including finger-pointing and long-term maintenance issues can add significantly to the apparent costs of the overall backup solution. These hidden costs can easily add tens of thousands of dollars to the cost of your solution. The impact of these costs cannot be minimized, as none of them is actually contributing to providing the best backup solution possible. These are truly wasted dollars.
Kelly J. Lipp is the CTO of STORServer, Inc. You can contact the author at kelly.lipp@storserver.com.