Getting at the Truth
If the reference process is so troublesome, then how can CIOs make sure they're not getting hoodwinked? You can protect yourself; first, put the software through its paces as much as possible. Then, look for a trusted peer to talk to you about the software before you go to the vendor. That is why it's so important to stay in touch with your peers, either by attending conferences or joining a professional organisation. "It's your personal relationship with the CIO that you have to leverage to find out what really happened. You won't get that out of a formal reference call," says the anonymous CIO.
When he does talk to vendor references, this CIO asks for contacts at specific companies on the customer list rather than accepting the ones the vendors try to give him. When you call the phone numbers, ask the companies what they did to get on the list. "It's entirely fair to ask: what have you received from the vendor in exchange for allowing me to sit here in your conference room?" says Andrew McAfee, an assistant professor in technology and operations management at the Harvard Business School, who is doing research on the CIO decision-making process. "I would imagine it's a question that gets asked less often than it should."
Merely knowing there are references is not enough. Rob Cohen, vice president and CIO at AstraZeneca, the London-based pharmaceutical company, says that two-thirds of the time, when he agrees to talk to a prospect, the company never calls. This suggests that some companies skimp on the homework - and that's dangerous. "I can't tell you the number of times where [vendors have] trotted out some very big name, but the reference is only using part of the product in one department," says Jim Sutter, senior partner at The Peer Consulting Group, and former CIO of Xerox and Rockwell International.
Companies that decide to do a site visit should send in people who will be on the front lines of the implementation, not just the CIO, Sutter recommends. "Particularly to a peer, [the CIO reference] will be reluctant to say: I really messed up when I made this decision," he says. "If you send some technical people in, they will uncover what worked and what hasn't worked."
Then, during conversations, ask lots of questions - carefully, to extract facts, rather than opinions or generalities. For instance, you might ask: how long does it take for them to get back to you? Not just: how's their service? AstraZeneca's Cohen says. And remember Pickett, the reference who wouldn't listen to himself if he were making a purchase decision? That's not to say he finds references useless. He just won't use them to make a decision. Pickett has found that they're really only helpful once he's made up his mind.
"If it's a highly valued customer, I'm going to find out from them what extra mile [the vendor was] willing to go to get [them]," Pickett says. He'll probe to find out, for instance, what the reference was able to get over the standard vanilla contract. Then, he'll ask for tips on how to actually implement a product - not pick it. "It's free advice if you use it correctly," he says.
And if you don't? Well, you probably won't be worse off for following your own mind. After all, that's what Pickett does.
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.