Evan Jafa, CIO of First American, says getting your virtualised servers set up and running right really is just the start of any IT leader's virtualisation work. And that if you don't think holistically about virtualisation, you're in for a rude surprise.
"I think lots of folks are going to find out that virtualisation poses a whole new set of concerns for security, networks and applications," says Jafa.
In fact, he calls management a "make or break issue" for virtualisation. "Monitoring and management becomes absolutely critical in the virtual environment," says Jafa, who started work about three years ago on a large project to consolidate First American's data centers and standardize its storage and networking technologies.
To that end, his organisation is directing much of its energy now at management processes and tools, wide application of ITIL process to the virtualised environment, and even reshaping the IT staff.
Jafae hopes that in the future virtualisation will play into better management of his networking technologies as well. He's already testing some new technologies from vendors including Cisco to advance that goal. If you're trying to do holistic virtualisation planning, Jafa's advice may prove particularly interesting.
Big Provisioning and DR Wins
First American's business, which posted US$8.2 billion in 2007 revenues, delivers information such as mortgage, real estate and financial data to clients in those industries and others. So while many IT leaders closely monitor virtualisation projects to ensure that application service levels don't slow down for internal users, Jafa's team also keenly focuses on application performance for external customers.
The company today has two data center locations, one in California and another in Texas, with about 4,500 physical servers, soon to be 6,500 (due to newly added business units.) It serves 40,000 to 50,000 concurrent users during peak business hours; its network carries 10 to 21 terabytes of data per hour, averaging about 15 terabytes per hour, Jafa says.
To keep all that data flowing smoothly, Jafa says his tool chest currently includes offerings from VMware, Microsoft, BEA (WebLogic) and "lots of Linux". His team is also using HP's SiteScope tools for hardware performance monitoring.
Jafa says he defines virtualisation as "achieving the highest efficiency of any given system at any give time". In servers, virtualisation is netting him 50 to 60 percent utilisation rates, compared to 10 to 15 percent utilisation rates on standard servers, he says. The team also sees a 30 to 35 percent reduction in energy consumption per virtual machine compared to its physical counterpart, Jafa notes.
To date, IT has virtualised 25 to 26 percent of First American's servers, Jafa says. These include many production servers, plus test and development and staging boxes. In some cases, whole business units now have had all their servers virtualised, he says. "My goal is to reach 50 percent virtualised this year," he adds.
"Any server that IT's using is going virtual," Jafa says, which means some 300 servers will be consolidated onto seven or eight physical machines. "We continue to work on virtualising disaster recovery." Jafa is bringing disaster recovery in house: "That's going to be completely virtualised", he says, noting he hopes to complete the DR project by end of year.
virtualisation has also helped Jafa's team as the company consolidates and centralizes IT services for an array of business units. "The process of bringing them on and migrating them has become faster," Jafa says, noting that the company's IT provisioning has improved by a factor of 65 percent, changing from weeks to days.
On the disaster recovery and high availability front, virtualisation has given Jafa equally dramatic results. For every 100 servers, he now puts aside four servers for failover purposes and six to eight servers for workload help at peak usage time for the company's customers. Previously, the company's high availability arrangement meant that he needed to put aside 200 servers for the same purpose.
Politics and Process Lessons
But server virtualisation is the start of the changes at First American, not the end. "As we started to think about virtualisation, we started to think about standardization," says Jafa, as in standardizing processes and utilising shared resources, across not only servers but also storage and networking technology.
In the course of doing his planning, he's run across two key challenges IT leaders cited in CIO's recent survey, Your virtualised State in 2008: Workload balancing and IT staff management issues.
"Capacity planning has become critical in the virtual world," Jafa says. "That's one we're still maturing and figuring out. We're using some of the HP [SiteScope] tools for capacity planning."
On the staff issue, he says he hasn't had to deal with much squabbling among IT members who are being asked to work together in new ways, noting his team is mostly excited about virtualisation. The bigger political challenge, he says, involved First American's diverse collection of business groups. "We have many business units and to get them off the notion of 'my servers' and 'my storage' took a lot of work," Jafa says. "I credit our (IT) change management organisation."
Still, he says, he does face a wave of organisational questions, ones that he feels most IT leaders will also have to battle.
"virtualisation will start posing a serious challenge organisationally to IT," he says. "The current vertical disciplines that we have will not be appropriate." For example, he says, IT may need a horizontal group with members from various teams to handle provisioning.
How is his group dealing with this situation? "So far we haven't torn down the vertical organisations," Jafa says. "Through the architecture organisation, we have brought those teams together." His IT team now has an overall architecture team that works on design, architecture and process questions, he says.
Process, he believes is key. Jafa, who was already a big fan of using ITIL to standardize IT process before he started the virtualisation efforts, weaves ITIL into his team's efforts in the physical and virtual worlds. Jafa says close to 55 percent of his team is ITIL-certified and he has a full-time ITIL instructor in house.
"In the virtual world, processes become super critical," Jafa says. "We want to make sure to test our processes to see how they will do. For example, provisioning has to be constantly defined," he says.
"We just did a whole assessment of exactly what will the impact of virtualisation be on our processes," he says. "We have to meld some of our processes for the virtual world," he says, noting network design and even software design has to change in the virtualised environment. "Now we can look at the long term impact."
On that networking front, Jafa is now working with Cisco to test out technologies such as Virtual Switching Systems (VSS) and VFrame (a family of provisioning and automation tools), in a quest to make his network more standardized and simpler to manage.
"Network provisioning becomes as critical as server provisioning," Jafa says. "We are very interested in virtualisation of the LAN, so we don't have all these protocols for storage and fiber in the network. The simpler the protocols and the more collapsed the protocols are, the easier the virtualised world will be managed."
For more on what that means, see "The virtualised Network of the Future." The virtualised Network of the Future
"virtualisation of the LAN" is a concept that can mean different things to different people. The basic idea is the same as server virtualisation: Take a single physical resource, like a LAN, and virtualise it in terms of resource sharing and simplified management.
If you have many separate LANs, you can use virtualisation to consolidate them into one switch, notes Deepak Munjal, marketing manager of Cisco Data Center Solutions. You can also consolidate network services, such as firewalls and network load balancing. (For more details on this, see Cisco's description of its Virtual Switching System, or VSS.)
Looking ahead, vendors such as Cisco would like IT leaders to think about virtualising the actual cables that come out of a server. Today, various cables including fiber channel interfaces to storage and Ethernet interfaces to management tools come out of physical servers. "You can start using new tools to collapse those protocols," onto two cables, a large link and one for redundancy, says Cisco's Munjal.
This is the idea behind Fibre Channel over Ethernet, an open industry standard that makes its Cisco debut in the recently released Nexus 5000, a 10 Gigabit Ethernet top-of-rack switch.
In Cisco's vision of the future, the benefits in a network design using switches like this one resemble the benefits of sever virtualisation: less hardware to manage, reduced power and better utilisation of hardware asset, and hopefully, simplified administration.
Keep in mind, Cisco just shipped that Nexus 5000 switch in early April and even forward-thinking IT leaders like Jafa are still in early test mode with such technologies, so it may be a while before you see large enterprise data centers actually revamping their network design.
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.