Upgrading core information technology systems is the bane of many a chief information officer's existence. It is a necessary and, for the majority, completely inescapable headache that is an integral part of a CIO's strategic focus. But while many organisations have to live with embedded systems and outdated software, an upgrade is a chance for a CIO to learn from past mistakes and put in place a new system that is at least a little better or more stable than the last one.
Many CIOs don't see legacy infrastructure upgrades as an exciting part of their job. It is time consuming, costly and potentially risky - and importantly, often invisible to executive peers who find it much easier to put their own stamp on the organisation. Not even project managers, whose lifeblood it is to smooth out the wrinkles in an infrastructure upgrade program, look forward to dealing with the host of potential problems of software and hardware upgrades, or even upgrades of the infrastructure on which they all run.
Software upgrades
In terms of IT infrastructure, core software upgrades are an inevitable and critical part of IT strategy and clearly part of the day-to-day operations of a CIO - a part of the job many attack with relish, while some delegate and others reluctantly do what has to be done.
The University of Sydney has been stuck with the same student administration system for a decade. The problem is, the system works and has no serious problem fast-tracking course enrolments of between 8000 and 10,000 students a year. Innovation, like using business intelligence tools to slice and dice student data on course enrolments, units of study and timetables, has had to take a back seat because the system, CIO ¿Bruce Meikle says, is fragile and costly to maintain.
Meikle says the university is considering its options on how best to upgrade its home-made core IT software over the next three years. The university is only one of a handful in Australia, he says, that is not already using off-the-shelf software.
"I inherited this system," he says. "It's on ageing technology and developed and grown over a long period of time. It's costly to develop and maintain, can't easily adapt to new government requirements, the performance is far from satisfactory during enrolment time and we have to wrap it in cotton wool at peak times of the year.
We need to move off it over the next two to three years."
The university's system is built on a Sybase database and is mostly written in the aged PowerBuilder language. It is a cocktail of disparate servers mixed together to deliver a fragile arrangement that is ill-equipped to take the organisation in the direction its strategy dictates.
Meikle says his aim is to drag the platforms up to the level of an industry standard as the practice of rewriting code in order to achieve more flexibility and a more institution-specific database is far too costly. He says data migration is a major headache and one that must be addressed by newer systems.
"We'll certainly keep an eye on the market and we hope to make a decision early next year," Meikle says. "Given what most universities have done, the process is roughly a two-and-a-half year program to purchase, configure and migrate existing information and test the whole thing. So it's big and lengthy. At the moment we're watching other universities and looking very broadly at off-the-shelf products, trying to work out if those offerings are the right place to be, or if we ought to look at emerging stuff."
Hardware upgrades
Keeping the business running while ripping the guts out of the IT backbone, and either upgrading them or replacing the systems, is not a job to be taken lightly.
Upgrading IT infrastructure, whether hardware, servers or, in the ¿University of Western Sydney's case, relocating to a completely new site, takes meticulous planning, an intricate knowledge of the business and customers, and maybe even nerves of steel.
The University of Western Sydney completed an infrastructure upgrade this year. It included upgrading major database servers that drive its main campus in Penrith, which is located in Sydney's outer west, as well as the building of a greenfield data centre in an off-campus site in Parramatta, in Sydney's middle western suburbs.
The upgrade included servers housing the student administration system, finance, human resources, payroll and the main teaching system. The multi-campus university upgraded databases to Oracle 10g and replaced two Sun E10K servers with two Sun Fire E6900 servers.
Michael Houlahan, director of information technology at the university, says the whole upgrade took four to five months because of the disruption to everyday business. He says that careful planning and scheduling were critical to the project's success. Upgrades are easier, if only for an application, because they all tend to lie on a single piece of infrastructure.
"The interdependencies between various applications were sneaking up on us and it became more and more complex over time," Houlahan says. "Even if you think they're already documented and understood, there's always an odd one out, like the authentication system that's critical for application authentication functions."
Upgrades to both IT systems and power supplies at busy, multisite institutions can present major challenges, especially when they need to be carried out simultaneously. Houlahan explains that the university had been using the same servers for about four-and-a-half years, and they needed an upgrade. One data centre was located in Penrith, and Houlahan did not want to install the upgrades there and then have to pick up the new system and move it to Parramatta. Instead, he needed the old hardware in Penrith working in tandem with the upgraded servers. That involved power upgrades to the Penrith site, because in the middle of summer the air-conditioning would be running at full power. So the start-up of the new data centre was scheduled to happen at the same time as the other upgrades.
Houlahan says the critical element to the upgrade was timing. The campus used to be very quiet in the Christmas to new year period and so this used to be the ideal time to upgrade core systems, before the new year enrolments started. But that opportunity has now closed.
"We used to schedule upgrades right across the Christmas vacation period but, increasingly, exams and enrolments are happening in those periods, so we had to come in on weekends or at 3am to do the upgrades very carefully," he says.
However, he says that in the future most of the headache will be taken out of core infrastructure upgrades by using an outsourced model. "The whole challenge of technology availability and having the appropriate technology skills in-house is becoming harder and harder, so I see us moving to using more external parties," he says. "I'm not sure if the business models are available to let us outsource with confidence. But over the next two to three years, we'll consider outsourcing rather than upgrading."
Infrastructure upgrades
Preparation is the key to success in the majority of infrastructure upgrade projects - no matter which part of IT needs to be addressed - to avoid conflict with day-to-day activities and to ensure a smooth and straightforward upgrade with minimal disruption to the business.
Although faced with the same time constraints as the University of Western Sydney and the University of Sydney, the IT team at ¿Curtin University in Western Australia had to dig a little deeper to meet the challenge of upgrading a decades-old data centre - as deep as the backhoe would go.
Curtin was faced with the predicament of legacy and decades-old equipment driving critical university systems that needed to be rebuilt, and a redundant back-up power supply as part of its disaster recovery plan. While building a completely new data centre on campus, the university also had to dig trenches to run cable from a diesel generator to the new site and as well as redesign the older data centre site - a project that had to take place during the Easter break to avoid power outages crippling the student and administration network.
Curtin has also spent the past year preparing to upgrade its core infrastructure, in particular the legacy data centre installed in 1975. The builders are expected to finish the new design at the end of September 2007, and the whole site will be integrated into the university network soon after.
Des Thornton, CIO at the university, says the upgrade partially addresses business continuity plans, but this wasn't the main driver for the project. "The upgrade came about because we were looking at horizontal scalability in the future and using clustered servers, both physical and virtual," he says. "It was critical for us to move away from the architecture we had in place to one that was more versatile. Beforehand, we had one single data centre, which was a large scale point of weakness.
"We have two 130-square-metre data centres live all the time running applications used by the university, and one of the data centres we use as a primary was built in 1975. It has undergone significant upgrades and the new data centre will be in the same functional network.
"We've already refurbished the core data centre and will have another new one at the end of the month. In the first part of this year, we made significant upgrades to our business continuity plans by using one storage area network (SAN) in each building that's mirrored, clustered and serves as a redundant data centre. But in a true disaster recovery project, the physical separation would be greater than just on our primary campus."
The university is now using an EMC SAN array with HP 64 bit servers - and rapidly consolidating some 330 individual servers across the entire university. The university research systems only run on the two "quality" data centres.
Strict project management guidelines were the crux of the upgrade, owing to the critical nature of the data centre and the significant work needed to be performed in the centre's operational areas. Thornton says the work to upgrade the data centre began in 2005 and is still under way.
"In 2005 we had a massive alteration to our data centre in terms of physical layout, such as turning racks around for better airflow and redesigning the cooling as well as replacing all switches," he says. "We needed to upgrade power for at least three feeds - we needed two feeds from our primary tower and another feed from our 450 kilovolt-ampere diesel generator.
"To upgrade this, we needed our project management team to rewire the switchboards and install one power supply from the generator as a failover. We also installed an uninterruptible power supply, which gives us 20 minutes to switch over."
Thornton says the most challenging part of the upgrade was replacing the two SAN arrays for the university network, which he says, were like "replacing the wheels on a car while you're driving".
Planning an upgrade
• Consider critical business hours before planning any upgrades.
• Don't reinvent the wheel, consider standardising equipment/hardware.
• Think practical and not necessarily cutting edge.
• *Consider outsourcing if upgrades are too costly/recurrent.
• Appoint a project manager to keep the project on deadline.
© Fairfax Business Media
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.