For Seth Copeland, taking the job of IT director for Tanner & Haley Resorts
in Kansas City, Missouri, meant jumping in to the development of a disaster
recovery plan to support the company's 160 users. "When I first took over this
position in October 2005, there was really no plan; we simply used tape backup,"
Seth said. Adding to the complexity of implementing such a plan, Tanner &
Haley filed for Chapter 11 bankruptcy in July 2006 (and is in the process of
being acquired by Ultimate Resort)—just as the newly developed disaster
recovery plan was going into effect. Taking advantage of cost and space savings,
Seth decided to implement a virtualized disaster recovery solution. I spoke
with Seth about how he used virtualization to provide a budget-friendly, two-tiered
approach to high-availability and recovery.
What triggered the development of a disaster recovery plan for the company
at this time?
For me, the motivator event was when a tornado touched down not too far from
the office. Most disasters in Kansas City are the local ones, tornados, that
type of thing. I think there may be a fault line somewhere, but I don't know.
The Missouri River might flood, but we're pretty far from the river. If somebody
nukes Kansas City, then we're probably out, but I don't worry too much about
that.
Explain your current disaster recovery solution. Does
the plan define procedures according to a disaster's severity?
Our plan uses VMware products and Double-Take Software's Double-Take for Virtual Systems. I have a server here in my data center that runs VMware GSX Server,
and I replicate all my individual servers onto it locally. The GSX Server machine
serves as my local, high-availability server, in case I lose a single box. If
my mail server dies or my database server goes down, the local high-availability
server takes over. Additionally, all those virtual servers on the local high-availability
box then replicate over to our Overland Park, Kansas, site via the Double-Take
software, to four boxes there, on which we put about four or five servers each
using VMware GSX Server. The virtualized server here at my main data center
takes care of a single server failure. The Overland Park site takes care of
site failure.
My two main concerns were how fast could I get the system back up and whether
the virtual site could handle everybody coming into it. Could it handle the
load? And it did. For our recovery time objective (RTO), we generally shoot
for about two hours, losing no more than the last 10 minutes of data. That's
what I set up, looking at the business needs.
Losing one disk, losing the mail system, the reservation system—that's
a small disaster. The plan takes care of that by just failing the systems over
to the local server.
The plan also takes into account an event that wipes out this building, where
employees can't even come here and people have to work from home. In that event,
we'll need to get the call center back as quickly as possible, so we'll have
employees take home their office phones, reroute our phone lines to corporate
headquarters in Connecticut, and connect to the Mitel Networks VoIP phone system
to run the call center out of their homes.
Why did you make virtualization the basis for your solution?
The main justification was the cost savings. After we filed Chapter 11, every
dollar counted. Virtualization software has come far enough and the hardware
is powerful enough that I have no worries about running our recovery servers
in virtual machines. I mean, if I had to build my data center from scratch,
I'd definitely do so by virtualizing more. So that's why we opted to go with
virtualization, really—to save on hardware costs and save on rack space
in our disaster recovery site.
In addition to restoring call-center services, what are some other components
of your business continuity plan?
I've given the individual managers telephone plans—that is, how I'm going
to communicate with them in the event our systems go down, because it's going
to be a little while before our BlackBerries and email come back online.
I also ask the managers to prioritize a few items: Who are their most important
people— whose service should be restored, and in what order—and
what are their most important systems? If on the first day of the disaster,
I can get 10 of your people back on, who are your most important 10?
How's the solution running so far? have you had the opportunity to use it in
an actual disaster yet?
We haven't had any issues with the solution. We test it every few months. In
testing, my two main concerns have been how fast can I get the system back up
and can the virtual site in Overland Park handle the load of all our users switching
over to it. And it performed well on both counts. We haven't had to use it in
real life yet. I hope we never have to.