In-Depth
Data Center Emergency Communications Key to Disaster Survival
The 4 Ps for staying informed when disaster strikes.
By Scott Madsen
As consumers and businesses shift more of their data and applications into the cloud, the pressure is on data centers now, more than ever, to stay up and running at all times. Any service outage could lead to major consequences. Although most organizations today have systems in place to notify their customers of planned network outages, they are often unprepared when it comes to unplanned events, such as natural disasters -- whether it’s a hurricane, earthquake, or tsunami.
This past August, Hurricane Irene put data center operators in the Northeast to the test, forcing them to scramble to put together a disaster communications plan on the fly as a near-historic storm churned its way up the coast. Though most data centers and their customers emerged unscathed, the storm had a surprising, and in some cases catastrophic, impact on communities located inland -- and served as a warning bell.
Enterprise customers of data center services everywhere need to ensure their service providers have a thorough emergency communications plan in place that can be implemented immediately in the event of a disaster. The plan should proactively keep customers informed of detailed business continuity and disaster recovery efforts at regular intervals and across multiple communications channels.
We've developed an easy-to-remember set of four steps -- to take what we call the 4 Ps to stay informed about data center operations during and after a disaster.
P #1: Plan Ahead
If you’re a customer of a colocation, managed service, or cloud service provider, you rely on these organizations to keep your business up and running. Beyond physical data protection and recovery plans, one of the most critical elements of selecting a provider is making sure they have a step-by-step emergency communications plan in place so that you can keep your customers, employees, and other constituents informed during a disaster. In fact, you should make sure this plan is thoroughly outlined in your service-level agreement (SLA) before signing up for services.
P #2: Proactive Communications
Data center communications should be frequent, detailed, and span multiple channels -- but above all, they should be proactive. Your provider should alert you to a potential issue before you are forced to spend critical time reaching out to them for a status update. That way, in the event of an emergency, you can keep your energy focused where it needs to be -- on your business and your customers.
Although Hurricane Irene had a surprising impact, there was sufficient warning that potentially dangerous weather was on its way. Unfortunately, not all disasters come with advance notice. As a customer of data center services, you need to ensure your service provider has an established communications plan for both scenarios. When it’s something that you can see coming, the plan should call for ample communications leading up to, during, and after any potentially hazardous weather pattern or natural disaster. Get specific about timing of updates -- ask how often you’ll receive them. Will it be in 15-minute intervals, hourly, or daily? Also, be sure there is an automated system in place for distributing updates.
P #3: Precise Communications
The devil is in the details, and that’s also true in an emergency scenario. You need comprehensive information, but you also need the ability to easily share it with your key audiences -- including those outside the disaster zone -- to outline the steps that are being taken to prevent any impact on business.
Although your service provider should have redundancy and contingency plans in place to prevent outages, they still need to be prepared for those instances when Mother Nature will not be deterred. Providers should distribute specific information, including where the outage has occurred, what steps are being taken to correct it, and an estimated date and time that the issue will be resolved. If there isn’t a quick fix, your service provider should continue to provide thorough updates throughout the course of the issue -- to resolution and beyond -- when appropriate.
Also, if you have staff onsite at a data center, the service provider needs to keep your entire team apprised of details, such as how they are working with building management, fuel vendors, and security, assessing and procuring supplies and equipment, scheduling shifts for onsite personnel, providing travel arrangements, and finalizing evacuation procedures.
You will also want to ensure there are plans to distribute a formal outage report that summarizes continuity and recovery strategy, efforts, and key milestones. For example, some service providers offer a Reason for Outage (RFO) report to impacted customers that includes a full summary and timeline of events, root cause analysis, and a review of preventive/corrective actions taken or to be taken. This report should provide a technical breakdown of events that is digestible by a wide audience, including non-technical personnel.
P #4: Prevent Lost Time
In today’s increasingly global and mobile world, it’s also important to make sure your service provider is able to distribute updates across multiple mediums. Since you’ll receive critical information on all relevant channels -- no matter where you are or what device you have access to -- this ensures that you’re not wasting valuable time searching for information or trying to contact your provider for a status update.
E-mail updates are a must, and you should also be able to call the network operations center (NOC) for updates. (Though the ability to speak with a live person is ideal, it is not always possible during a disaster. Be sure your service provider intends to post regular, pre-recorded updates at the beginning of the recording.)
Beyond these two communications channels, you’ll want make sure you can check your provider’s customer portal for critical service alerts. For security reasons, make sure this page is protected rather than open to the public. Also consider this simple but often-overlooked tip: make sure you keep your log-in and passcode to the critical service alerts page easily accessible so you can find it in the event of a disaster.
You should also ensure your service provider has an opt-in SMS (text messaging) emergency alert service. SMS is quickly becoming the preferred method for communicating emergency information in public school districts, the private enterprise, and even 911 services. Make sure you have the ability to tailor updates to suit your needs -- for example, the ability to opt-in to receive updates on specific facilities so that you are only receiving information essential to your business.
There’s No Time like the Present
Officials estimate that Hurricane Irene generated between $8 and $13 billion in property damages, caused five million power outages between North Carolina and Maine, and resulted in at least 29 fatalities, not to mention millions of people uprooted over the course of the event. Unfortunately, hurricane season isn’t over yet -- the Climate Prediction Center expects a total of seven Atlantic hurricanes this season, which is expected to last well into November.
Given shifting weather patterns shifting, there is no time like the present to assess your data center service providers’ emergency communications plans to ensure that you can protect your business in the event of another disaster. Even if you’re dealing with a trusted service provider that prides itself on network availability, there is still no excuse not to be prepared for the worst. Reach out to your provider today to ensure that the priorities and methodologies described in this article are included in your provider’s SLAs, and then make sure your own organization has an emergency communication plan in place. The cost, time, and effort of such considerations can help to offset the enormous risk and financial burden of data lost permanently to Mother Nature.
Scott Madsen is the director of network operations at global IT infrastructure services provider Internap Network Services Corporation. You can contact the author at smadsen@internap.com.