Catastrophic Disaster Planning and Recovery

I state that Hurricane Katrina missed the city because she passed just east
of New Orleans as a Category 3 hurricane with winds of 125 mph. For years, those
of us living in the metropolitan area have been warned about “The Big
One” that would fill the city with flood waters, taking years to recover.
However, the worst case scenario was to have the strongest of all hurricanes,
a Category 5 storm, pass to the west of the city. In fact, to be totally correct,
New Orleans was spared the brunt of Katrina. This leads us to a huge lesson
learned: In Southeast Louisiana, we live in a very fragile environment.

Due to the fact that the worst may be lurking in the future, we have added
the word Catastrophic to our Disaster Recovery Plan. We now know that disaster
recovery planning must include the possibility of relocating the firm to an
entirely new city for over two months. We also recognize that a real disaster
recovery plan addresses much more than just technology or business continuity;
it addresses staff retention, reconstruction issues and post-disaster communications.

Bourgeois Bennett is a traditional CPA and Consulting firm established 85
years ago, maintaining two offices. Our New Orleans office houses 80 staff,
and we maintain an office in Houma, La., with 10 staff. Houma is located about
60 miles southwest of New Orleans, and, as you’ll see, it eventually became
our new home for months after Katrina. Our 14 partners have worked hard to provide
a professional yet family oriented environment for our staff. Two years ago,
we moved into the entire 17th floor of an office building located directly on
the 17th Street Canal (this canal was one of those that breached, flooding the
city during the storm).

In baseball, a good fastball sets up a curveball. For years, New Orleans has
been thrown fastballs by our local media, warning of the catastrophic flooding
that could occur if a Category 5 hurricane slammed into the city. In September
2004, Hurricane Ivan approached the city with a 25 percent chance of being “The
Big One.” Local government called for an evacuation, and over 600,000
people left the metropolitan area. Our disaster recovery plan was put in effect
just as Ivan turned north and hit Gulf Shores, Alabama. Although the evacuation
was painful, the recovery plan implementation went smoothly.

On Thursday, August 25, 2005, Katrina hit Florida as a Category 1 storm. Since
it was the eleventh named storm of the season and the eighth storm of the year
to hit the Gulf of Mexico, not much attention was paid to Katrina. Weather forecast
computer models all pointed to a curve to the North and a second landfall in
the Florida panhandle.

By Friday afternoon, computer models had concentrated on New Orleans. Our
disaster recovery plan mandates that laptop computers, which make up 50 percent
of our inventory, are evacuated with staff. This gives us a good start in case
we need to rebuild our practice in a new location with no access to our existing
offices. Unfortunately, a good many of our staff had already taken off early
to enjoy the summer weekend, leaving some laptops at the office.

However, the other parts of our plan were put into effect. According to our
plan, three full tape backups of our system are evacuated with our Managing
Member, Senior IT Staff and me. On Saturday morning, as citywide evacuations
began, I initiated a full system backup. Our plan also calls for us to shut
down our servers as late as possible, to unplug and remove all electronics from
exterior windows and to download a listing of all staff cellular phone numbers.
All of these steps were followed, and by 10:00 p.m. Saturday night, when a hurricane
warning was issued for the area, our office building shut down for the storm.

On Sunday morning, the day before Katrina made landfall, she was a Category
5 storm with sustained winds of 175 mph. One million people in the metropolitan
were evacuated, and at 9:00 a.m. I join them, heading for Houston. Although
it usually takes six hours to get to Houston, I arrived in 14 hours due to evacuation
traffic problems.

On Monday, August 29, at 8:00 a.m., Katrina passed just east of New Orleans
with winds of 125 mph. Tidal flooding of 12 feet occurred in St. Bernard Parish,
just east of New Orleans with 35,000 homes and businesses destroyed. The surge
in Lake Pontchartrain, located to the north of the city, reached over 10 feet
and breached the levee system in three places, pouring the lake into New Orleans.
Eighty percent of the city would flood while the water poured in for the next
three days. (For those of you unfamiliar with the geography of New Orleans,
the city lies in a bowl, surrounded by water that is kept out by a large series
of levees. All water, whether it is from rain or flood, must be pumped out of
the city by large pumps located on canals that run throughout the city. It was
these canals that breached, leaving the pumping systems virtually useless since
the pumps would only pump the floodwaters back into the already breached canals.)

On Tuesday morning, the news coverage showed the total devastation of the
city. It would be weeks before the floodwaters could be pumped out, there was
no electricity or water, and government officials were saying it could be up
to a month before residents would be allowed to return home. My main concern
was for my property. My next-door neighbor stayed for the storm and was able
to relay information regarding my house for two days via landline phones (they
never went down). Although I had three feet of water surrounding my house, it
did not flood because it is built three and a half feet off the ground.

My next concern was for my friends. I knew that many of my coworkers lived
in areas that were under 12 feet of water. I pulled out my cell phone list and
tried to make calls, but all cells were dead. The cellular towers in the New
Orleans area code were in floodwaters, and the generators running them during
the power failure had not been refueled with diesel. Any attempt at cellular
communication was in vain. This was our first lesson learned from Katrina. Although
we had created a listing of key staff cell phone numbers, this listing became
useless. This problem was further compounded by the fact that our firm had evacuated
across the country to unknown locations. Each of us became an island due to
our inability to communicate with each other.

It was at this time that I realized a major flaw in our Disaster Recovery
Plan: We had no “meeting place” to which we could report and get
information on the firm. I like to compare this to your home’s fire plan.
You plan your escape routes from the house and then decide on a meeting place
away from the house. Although our escape was successful, we had failed to come
up with a meeting place in case communications became unfeasible.

I shot off an e-mail to my technology associates at AGN International, our
CPA association, informing them that I was safe. For the next few days AGN posted
my messages on the front page of its website. A few of our staff, unable to
make contact with anyone from the firm, went to the AGN site and found my messages.
Eventually, they made contact with me via AGN. We’ve now added AGN’s
site address to all of our emergency contact information.

On Wednesday, I received a text message on my phone from an Audit Manager
friend. It seemed that, although voice activity was dead, cell phones were still
able to intermittently send and receive text messages. Using this method, I
was able to contact a handful of staff, acquire personal e-mail addresses and
began to put together an e-mail list. Most of us had evacuated to locations
where Internet access was available. However, communications regarding the firm
were still irregular at best. We needed to come up with a more complete solution.

Then I remembered that a year before Katrina, we started leasing an online
bulletin board system to facilitate the questions and answers that arose from
the implementation of new tax preparation software. I created a section on that
site specifically for the hurricane and set up topics such as “Firm Information”
and “Where Am I?” We immediately sent e-mails and text messages
to each of our staff, giving them the URL of the site along with their username
and password. By Friday, September 2, we were in contact with 60 staff. We used
the bulletin board system to develop an emergency contact listing, containing
the personal e-mail, current address and phone numbers of all staff.

Our management team also used the board to call for a staff meeting in Lake
Charles, La., for the Sunday after Katrina. As it turns out, we had 15 staff
located in Lake Charles working a class action disbursement the weeks before
and after Katrina. Many others drove in from around the area. After a closed
door meeting, the Members sat down with the staff and discussed the options
facing the firm. It was decided that the New Orleans office could be back online
in two weeks. During that time, staff would be paid for an additional two weeks
of Emergency Time Off (the firm already paid us for a week of ETO after the
storm).

At that time, local governments were not allowing anyone into the city or
surrounding parishes, so the only other option would have been to set up shop
in another city. This decision to wait and see, in my opinion, was the catalyst
for the strong comeback our firm has made. Fifteen of our New Orleans staff
lost everything they owned. Many more of us had suffered some damage to our
property. By waiting an additional two weeks, our staff had time to get over
the shock and awe of the event. Also, due to the location of our office, the
east and north facing offices overlooked a flooded area of the city. My first
trip into our offices two weeks after the storm was shocking. As far as the
eye could see, rooftops were sticking out of the water, as if an entire city
had been built in the middle of a lake.

Other local firms made more hasty decisions. Some farmed out their staff to
firms in other states. Others decided to let staff go, thinking that their clients
and practices would not recover. Of all the firms I am familiar with, Bourgeois
Bennett was the most generous with its handling of the problem. As I said before,
we’re a family, and management treated us like family.

“How do you prepare for and recover from a disaster that destroys the
entire city in which you do business?” I’ll never forget this quote
from Ralph Cox, our Managing Member. It encompassed the soul of the business
community of New Orleans after Katrina. Although massive residential problems
and communications issues were still brewing at the surface, our offices appeared
to be in relatively good shape. Two weeks after Katrina, power had been restored
to our firm, and the major damage appeared to be limited to a single office
that had a shattered window. Some minor rain seepage had occurred in a few of
the offices, but our servers were running and we appeared to be ready to reoccupy
our space in the following week.

Unfortunately, the news came that our building had mold problems, and until
those were resolved, tenants would not be able to occupy their space. It was
decided that we would rent office space in Houma on the floor directly below
our existing 10-person office. We put 45 staff into 2,500 square feet of office
space using six-foot folding tables that seated two and sometimes three staff
members. We ran a data line connection up to our existing office and tied into
that network. Just five months before, we had installed a point-to-point T-1
line between New Orleans and Houma. Using this line, and the fact that although
our offices were uninhabitable they still had electricity, we did not have to
move any of the equipment in our server room to our remote location. This also
gave us the ability to allow staff to work from other remote locations through
Citrix and Outlook Web Access.

Less than a year before Katrina, we purchased a document imaging system and
began to scan our individual tax returns and supporting documentation. Although
we could get back into our offices to retrieve manual workpapers when necessary,
having the scanned documents to work from was a real timesaver. If you want
to guarantee access to information in a disaster, scan it.

For the next 76 days, we worked out of the Houma office, commuting one to
two hours each way. It was cramped and noisy, but our family was together. We
sat next to each other, offered hugs when needed, and cried together about our
losses. I have never been more honored to come to work than during that time.

In November, we finally moved back into our offices … well, almost all
of them. The north-facing offices had to be totally gutted, so it was just before
Christmas when our
entire floor of office space was ready to be occupied.

In response to Katrina, we have added some items to our Catastrophic Disaster
Recovery Plan. Bourgeois Bennett has purchased its last desktop. All future
computers purchased will be notebooks. Under our current plan, the staff will
evacuate with their notebooks. Also, we now maintain a listing of everyone’s
personal e-mail address. If our Exchange Server is down, we can now e-mail our
staff. We also maintain a listing of emergency out-of-town contact numbers,
which is kept on our externally maintained Bulletin Board system and can be
modified to reflect where each staff member has evacuated. Finally, we developed
laminated business card-sized informational cards with key management’s
personal e-mail and cell phone numbers. It also tells staff how to access our
Bulletin Board if power is down at our main office.

Our firm has also enrolled in an Employee Assistance Program, in which staff
can call 24/7 and get stress or grief counseling. As time passes and rebuilding
progress continues to be slow, the suicide rate in the New Orleans area has
risen significantly. In April, Bourgeois Bennett faced this issue first-hand.
We now listen to each other a bit more closely, hug a bit more tightly and pull
together as a family even more than we did before. The tragedy of Katrina continues
many months after the storm.

In spite of everything that Katrina could throw our way, business here is
good. In fact, it is great. Staffing continues to be an issue, and we are always
looking for senior-level staff. The tax filing deadline was extended to October
15, so we have had an extended tax season. Our clients are coming back, and
we’re here to help them.

We learned lots of lessons due to Katrina. Probably the most important lesson
is to never say, “It can’t happen here.” Everyone needs to
have a plan of action in case the unexpected happens. And plan for the unexpected.
If you do, the lessons learned will be educational and not destructive. And
remember, clients come first, but staff comes a close second. 

——————————

Les Nettleton is Director of Information Technology at Bourgeois Bennett,
LLC., a large locally owned, regional accounting firm in Louisiana.