CIO

Grace under pressure

When the Ministry of Education moved ‘the heart of the organisation’ – its datacentre - to the cloud, the ICT team prepared for every possible scenario.
At the Ministry of Education in Wellington: Geoff Major and  Aaron Kumove.

At the Ministry of Education in Wellington: Geoff Major and Aaron Kumove.

When the Ministry of Education moved its production data centre to a cloud facility early this year, its then acting CIO (and current chair of its IT Infrastructure Services Board) Aaron Kumove likened the project to undergoing a major medical surgery.

“You are, basically, moving the heart of the organisation from one place to another,” says Kumove, who led the project that was completed during the four-day Easter weekend this year. It involved moving the ministry’s St Paul’s Square (SPS) data centre to Datacom’s Kapua, Hamilton facility.

“What a lot of people don't realise is that this ministry runs not just obviously its own IT systems, but also key systems which enable service delivery and transactional activities for the sector as a whole,” explains Kumove. Production systems of the NZ Qualifications Authority and the Tertiary Education Commission, among others, depend on the ministry. “That is key stuff to keep the business of education running for the whole country.”

There was a “high degree of interest and some nervousness” around the project, says Kumove. This was one of the major projects of the ministry following Novopay, the web-based payroll system that led to cost overruns and continuing complaints of under and over payments for teachers in the state and state-integrated schools.

Thus, he and program manager Geoff Major, took a “very risk adverse approach” to the implementation, which was completed successfully, on schedule and within budget.

The key was not to take away people's ability to communicate in case of a crisis situation.

Aaron Kumove

Object lessons on change management

The prime purposes of the move were to increase reliability of the ICT infrastructure supporting the education sector and to mitigate risk says Kumove. It was initially planned to be completed by the end of 2014, prompted by the directive from the Government CIO (GCIO) to move to an All of Government (AoG) selected data centre for public agencies.

But the earthquake that struck Wellington in August 2013 pushed the timeline and priorities forward.

Kristine Kilkelly, deputy secretary corporate and governance, whom Kumove reported to, directed that the top ICT priorities were firstly to accelerate the disaster recovery stage 2 project, ensure that fit for purpose disaster recovery capability existed for all production systems, and then start the move out of the SPS datacentre “as soon as possible”.

“While we had very strong support from the top on down, they also made sure that we did our homework and grilled us quite tightly on it,” says Kumove. The Treasury Major Project Monitoring Unit and the GCIO kept an “active eye” on the project.

The Easter weekend was the only “change window” to do it, explains Kumove. “Each of those other agencies which rely on our systems has their own change windows and getting them all to align didn't leave a whole lot of opportunity, other than that weekend to do that in.”

Major was then working on another project at the ministry but after the earthquake, he was told the datacentre shift was going to be his new focus. Major then brought in a project manager, Brian Orman, who worked with him on a previous data centre move.

He says the IT team at the education ministry was also extremely experienced. “It meant our foundation was very strong to begin with, both in people and in the documentation.”

At the Ministry of Education in Wellington: Geoff Major and Aaron Kumove.
At the Ministry of Education in Wellington: Geoff Major and Aaron Kumove.

“I don't think we can overstate enough that extremely competent and able people made this happen,” adds Kumove. “It involved quite a number of people obviously in the ministry's IT shop, but also wider across the ministry as a whole, across the sector and with our vendor partners.

“Because of the impact on sector agencies due to their reliance on some key systems, the change management around this was also quite critical,” he stresses. “We held workshops with other agencies to ensure we did not miss anything that could leave them in the lurch, but also to get their buy-in.

On top of this, the ministry was also working on other business projects, including the move to Windows 7. This was completed before the April 8 cut-off date from Microsoft’s support for Windows XP.

“We had to visit every office in the country for the Windows 7 move,” says Kumove. “We also put in place a new identity and access management system for the sector as a whole at the same time that the datacentre and Windows 7 projects were underway.”

Another transition was happening with the ministry. CIO Leanne Gibson had left in August 2013 to take the same role at the Wellington Airport. Kumove, an independent board member on the ministry’s IT Governance Board and former NZ Post CIO, took up the role of as interim CIO upon Gibson’s departure.. In March, Stuart Wakefield was appointed as CIO. Kumove remained chair of the infrastructure services board, and Major as the program manager, as they proceeded with the datacentre migration.

The primary focus was to drive out all risks, or prepare for any eventuality during the migration.

“We put in some redundant network links from a second provider from our standard provider,” explains Major. “We used those secondary links to move data across the wire. We pre-provisioned some hardware and then we moved data across the wire onto that pre-provisioned hardware.

“We saw very early that the biggest risk was physically moving critical devices,” says Major. “So wherever possible, we pre-provisioned infrastructure at the destination datacentre and moved the data over the network links. Wherever possible, we eliminated or reduced the risk of physical damage.”

The migration was done in two phases. “The first phase was to move our test environment only and not the production environment,” says Kumove. “That allowed us to prove that our process was good, that we had the roles all lined up properly.”

We saw very early that the biggest risk was physically moving critical devices. So wherever possible, we pre-provisioned infrastructure at the destination datacentre and moved the data over the network links.

Geoff Major

This was done four weeks ahead of the migration. “Basically, it allowed us to dry run the production move and allowed us to test all our processes, our infrastructure, everything from A to Z,” expounds Kumove.

Major says they also tested the new infrastructure that was put into the destination datacentre at Datacom. “We were running real applications on it in our test environment.”

“And being a test environment, if it had not gone quite so well, it would not have been the end of the world,” adds Kumove.

“After we had done the shared test environment, we then replicated all the production data across to the new datacentre and that enabled us to have all the data there already and have it all ready for us to flick the switch over Easter,” adds Major.

Another risk mitigation strategy was using separate vehicles to transport the equipment. “We used two trucks, so there was redundancy if one truck had an accident. They even took different routes, just in case.”

Kumove says the voice and data were on the same network in Wellington. So Major and his team did something “clever”.

“They separated the voice and data networks prior to the move. That way, even if the datacentre move had not gone well, the phones would still be working,” says Kumove.

A storm was forecast for the Easter weekend, so ministry staff were flown in early to make sure they had staff in the new datacentre during the move.

“Every possible bad thing we could think of, we had catered for,” says Kumove, and credits Major and his team for this.

“The key was not to take away people's ability to communicate in case of a crisis situation,” says Kumove. “If it had all turned to custard, and none of the systems were available on Tuesday morning, at least they would have email, Internet, and have phones working. As it turned out, they had everything up and running that morning.”

Major explains this approach was tempered by his previous experiences in datacentre migrations.

Having an immovable deadline was a big help, says Major. “It was a matter of working back from that date.”

“It forced us just to line things up to make sure they got done in time,” adds Kumove. “The project management team was completed, and the high level planning was done prior to the Christmas holiday.”

Communications played a critical part, both with the vendors and the key agencies in addition to ministry staff and management.

Kumove says in a project of this scope, it was important to have good working relationships with vendors. These included Datacom (datacentre), IBM (back-up and storage systems), HP (servers), Cisco (network switches), Spark Digital (network and telephony systems), and VMware (virtualisation and site recovery.)

A key component of the deployment was highlighting the migration as a business – not technology – project.

“We sold it as a risk mitigation and capability enhancement exercise,” says Kumove.

A key component of the deployment was highlighting the migration as a business – not technology – project.

“Secondly, there was an aspect of compliance,” he says. “We had no choice, because the GCIO directive has essentially told every agency to get out of the datacentre business.”

“It is about having a fit for purpose infrastructure for the long-term,” he states. “The new datacentre exceeds the capabilities of the datacentre that we had and sets us up well for the future.

“It also prepares us now to look at infrastructure as a service,” he declares. “One of the things we did as well to mitigate risk is we didn't go directly to infrastructure as a service. We did a ‘lift and shift’ as the first stage.

“We're actually in a very strong position now,” concludes Major. “We’re almost entirely virtualised; we're on Windows 7. We've got a very, very strong internal team...It's a very good position to be in.”

Send news tips and comments to divina_paredes@idg.co.nz

Follow Divina Paredes on Twitter: @divinap

Follow CIO New Zealand on Twitter:@cio_nz

Sign up for CIO newsletters for regular updates on CIO news, views and events.

Join us on Facebook.