Disaster Recovery made simple? Now that’s an oxymoron if I’ve ever seen one!
Anyone who has had the responsibility of preparing a Disaster Recovery Plan knows it is far from simple. Having had the responsibility for building and testing DR plans in several organizations, I can attest to the fact that preparing a plan can be a daunting task. In this article, I hope I can impart to you some basic knowledge that gets you started, or at least headed in the right direction.
With Hurricane Season just around the corner, this takes on additional significance for those of us living in South Florida, or anyone affected by the Atlantic Hurricane Season, which officially begins June 1 every year and goes through November.
While the information contained herein is geared towards Information Technology, the same principles can be applied to any business, or even any home.
One of the first and most vital components of a DR plan is documentation. Not just documenting your plan, but what documentation do you need to keep, and where do you keep it?
A proper Disaster Recovery/Business Continuity plan is one of the most vital documents in an organization. The lack of having this document has been the ruin of many a business over the years. What value can one place on the ability to stay in business after a disaster?
Like they say in real estate sales, the three most important things in selling a home are location, location, location. The same can be said for choosing a DR site.
Organizations need to carefully consider where their Disaster Recovery site should be. If you live in Tornado Alley, or, in my case, in an area prone to hurricanes, it would be wise for your DR site to not be located in that same geography.
In the event of a disaster, what is they key functionality or processes required to keep your business running? And, how do you plan to make that happen? Do you have adequate resources at your DR site to assure that those functions will continue?
Do you know what resources are dependent on other resources? Do you know in what order those resources need to be brought online? Bottom line – Know your business, and understand your business needs.
Along the line of knowing your business are some key things which must be understood as you build your plan, and as you build your infrastructure. A plan needs to include some key items, which essentially define your success criteria. Service Level Agreements (SLA), Recovery Time Objectives (RTO), and Recovery Point Objectives (RPO) must be understood, defined, and documented. As these criteria are understood, the infrastructure required to support them is better understood and implemented.
In the event of a disaster, who will perform the functions that keep your business running? If you have multiple office locations, this becomes much simpler. But if your primary office location is where the disaster occurs, consider that the people who perform those critical functions have also likely experienced a disaster in their personal lives.
When was the last time you did a Disaster Recovery test? Was it successful? Did your resources come online properly, and were your applications accessible from all expected resources?
I remember back to a couple of organizations I worked with in the past, and their DR tests.
At one of the companies, we were “preparing” for the DR test, and I asked the System Administrator how they planned to do the test (I was new to the organization – been there maybe 2 days at that point). He said, in a straight face, “We make a tape from this system, and we send it to our DR site, and if they can restore from it, we call it successful.” Now, if this system was the only one in the organization, that would be closer to OK. However… this was not even a production system, was not for a critical application, was only one tier of a 3 tier architecture… I asked how many times they had a successful test, and found out that even this rudimentary test had failed multiple years in a row. Needless to say, when I took over the DR plan, that process was the first thing scrapped. We, as a team, then built a plan which accomplished the organizational objectives, tested then plan, and were successful. Our team won an award that year from the company for our teamwork, not only within our team, but in partnering with other teams throughout the organization.
At another company, I was brought in primarily to be the DR Coordinator. Upon assuming my responsibilities, I began discussing the plan with the lead System Administrators for each technology area, and reviewing the DR plan provided by their DR site provider. I quickly found that the actual plan was 99% boiler plate, and 1o0% incorrect and inadequate. I asked the Director who I reported to when was the last time they had a successful DR test. Guess what the answer was…. “Never!”
So, we set out to do the work required to make a DR test successful – trashed the old document, interviewed the appropriate stakeholders, reviewed the business processes and application inter-relationships and dependencies, and built the new plan based on reality. We then set a date for our DR test and gathered the appropriate team members. We went to the DR site, executed our plan, and VOILA, we had a SUCCESSFUL test, the company’s first EVER!
So, the point of this article is not to blow my own horn. Yes, I’ve had much success in the area of Disaster Recovery planning and testing. But it has been the result of having a great team, all of whom bought in to the mission, and executed the plan to perfection. The credit goes to them, not to me.
You may notice that I touched on some fundamental things in this article in regard to DR plans, but didn’t give you explicit details, or all the details. There are 2 reasons for that – one, I would be writing for several months and far exceed the limits of this forum; and two, being brutally honest, we do this to earn a living.
So, if your company is in need of a Disaster Recovery/Business Continuity plan, or an update to your existing plan, hopefully I have stimulated your thought process, and you’ll reach out to a Disaster Recovery specialist and allow them the honor of working with you to accomplish your goals. Thank you for allowing me the opportunity to share my thoughts with you.
Note: This article may also be found on my blog at https://www.jamieadennis.com/blog
About the author
I’m just a Southern Ohio boy, born and raised in Chillicothe, OH (about 45 miles south of Columbus on US Route 23). I now reside in Delaware, Ohio, and love it here. I’ve traveled widely, worked in many organizations of varying sizes, and learned something every step of the way.
I do not claim to be an expert on anything – I reserve that title for those who create and improve technology, but I’ve been successful throughout my career by applying common sense to complex problems, whether those be technology problems or people problems. I continue to learn on a daily basis, which is the nature of a career in Information Technology.