It was a Sunday afternoon but Channa Jayasinha was in the Wellington City Council (WCC) complex when the earthquake struck.
“The first thing that came to mind was my own personal safety,” says Jayasinha, manager, information technology of the WCC. “It lasted 20 seconds but felt a lot longer.”
His team had earlier worked on moving their data centre to the Revera facility in Trentham.
He was there to “tidy up for Monday morning” and was just about to finish the work when the earthquake struck.
“As soon as the earthquake stopped, I came out of the building and from there the emergency activity started to take action.”
A day after the incident, Jayasinha says the team responded “extremely well” to the situation.
Because of the data centre move, “We actually had to test our BCP and DR already as part of that move,” says Jayasinha.
“We already had the BCP DR aspect top of mind for past three or four months,” he says, and this leads to another key lesson.
“Don’t write a BCP and then put in on a shelf,” he says. Make sure it is discussed in CIO management meetings and maintained and updated, and planned for any testing activities.
He says the council also has a building in the suburb of Tawa where business people can be relocated and they don’t have to travel to Wellington to work.
Once the ICT team activated the BCP, they started doing checks on production systems. We also did some testing on our DR to make sure things like replication has happened.
We have made sure the core members of BCP team access to physical and electronic versions of the BCP. “They didn’t have to come to work to activate it,” says Jayasinha. “We want to make sure we are ready at any time.”
“We have BCP copies in various locations in the car as well as at home for emergencies."
They then informed the people that everyone who wanted to work from home can work from home as long as the local ISPs are functioning. “Since then, today, we have opportunity to do a number of relocations of facilities to get the critical functions working.”
At the time of the interview, some of the offices were being checked for structural damage, and only one building was cleared for use.
But even before this, he says, the council has already set up an emergency response centre for the public walking to the council building and wanting some information.
Because it was payroll week, making sure the payroll system can function from tomorrow (Tuesday) was important so people can get paid on time.
“The key thing at the moment is making sure the regular communications are going to staff,” he says. Aside from email, they use bulk text through Fronde which can access 2degrees, Vodafone and Telecom numbers.
They are also using social networks like Facebook and Twitter to provide information for the public. Radio advertisements and news broadcasts inform the public where to get the latest information.
Jayasinha shares the top lessons he can share to CIOs who may be facing a similar challenge.
First is, “have a practical working current BCP and DR plan and they need to be practised well in advance of an emergency.”
Second is to have a core team who can support you in the BCP and DR. “Don’t rely on one person.”
Jayasinha had a core team of six to eight staff for this. “Make sure the core team can take responsibility to activate it. Don’t wait for the CIO to be the one to active it.”
The most important one is to be able to activate the BCP remotely. “You may not have access to the building,” he says. “That is the way we have set it up so we can access or production systems, our DR systems remotely.”
He says this was important as in the case of the incident on Sunday when most of the complex were closed and in locked down mode.
He says in case of an emergency, people have to work from home. For us the key was enabling them and making sure email was active and working. The council is able to send email and bulk texts to everyone. The bulk texts are useful as not all staff have smartphones.
“Find different channels of communication,” he says. “Don’t rely on one channel, have multiple channels.”
Social networks are important during emergencies because people not only get the latest information from these sites but also use them to keep in touch with friends and neighboyrs who they rely on when disaster happens.
He says the staff has made allowances to relocate some of the critical teams so tomorrow (Tuesday) they will be shifting PCs and printers.
Once things settle down, we will debrief and look at what lessons will be learned and what they need to do to update the DR plan. “Is there anything unusual that might have happened?”
Tales of business continuity from the trenches
David Roberts, general manager, marketing and alliances at Fronde says having majority of their business applications running in the cloud meant the quakes that hit the Capital on Sunday did not have a huge impact in terms of business continuity.
He says their Wellington office has a staff of nearly 300 but a great number are working remotely and on customer sites.
They received an SMS message that the office will be shut down on Monday morning and to contact their manager directly. The organisation also sent emails and messages on social media to staff.
“We won’t say we were unaffected,” he says, but “the impact was minimal and we were fully operational, fully available for customers.”
Roberts says Fronde tests its disaster recovery plans for operations every six months, and also participate in similar exercises run by their clients.
Bruce Aylward, managing director of PSODA says the software company’s 10 member team – five in New Zealand and five in Australia – check their business continuity programme every three months.
We simulate an incident then check all their communications, and do rollover testing backups and making sure these backups can be restored, says Aylward. The exercise extends to social networks, where messages are posted on Twitter, LinkedIn and Facebook.
Aylward explains their regular tests paid off on Sunday when a series of earthquakes hit Wellington, where PSODA is based.
But they were able to continue servicing customers as the staff were working from home. The backup plan includes a listing of other locations in case a staff member’s home has to be evacuated.
Aylward says their servers in Wellington were not impacted by the quakes but if these were, they would be able to run the business using the facilities in Auckland.
Aylward says SMS takes precedence over phone calls in case of an emergency as phone networks will be busy when disaster strikes. This will be followed by an email.
“Then we just monitored the emergency channels to see any additional problems that came up,” he says.
The lessons from their experience points to having in place a business continuity plan, he says. “Make decisions around if certain failures happen, what actions are you going to take? Who is going to be responsible, and then after that event, what kind of monitoring do you need to put in place?”
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.