And though the verdict's still out as to whether humanity is devolving toward Idiocracy, it's certain that folks are continually finding innovative ways to screw up IT's operations.
The last time we scrutinised that ultimate technology risk -- the user -- in "Stupid user tricks: Eleven IT horror stories" (Information Age, June/July 2006), David Letterman eventually called off the lawsuit. Then SWAT managed to defuse the letter bomb. And because that represented the most emotional response we received on any story last year, you knew we had to do it again. So here 'tis: more stupider user tricks.
By user we mean any slug with network access or oversight who manages a brain fart loud enough to halt the network, compromise it, or in some other way cause harm to the enterprise, and thus the company's bottom line.
Something that means at the very least a red face for several days, possibly a secret beating and swirlie administered by other employees in a darkened bathroom, even a Trumpesque directive to explore the outside world for the rest of one's life.
So, take heed. Lessons await. And if you've had a run-in with userusstupidus, feel free to share the fallout.
Trick No. 1: Poor purchasing oversight
Incident: "Oliver, do you know anyone with a pallet jack?" This isn't a question you want to hear from a friend over your Saturday late-morning Dewar's and Froot Loops because there's no way this call can lead anywhere good. So the instinctual answer is where you should leave it: "No." But morbid curiosity always prevails, and the inevitable happens: "Why?"
The central ingredient of this recipe for disaster is an offsite data centre, still largely under construction. A chunk of the cabling has been completed, so it's time to build out the data centre's skeleton: racks, UPSes, cable management, basic monitoring. In other words, a $300,000 payday for APC.
The problem is that IT staff can't make the purchases; they can only make requests. Requests are approved and implemented by the purchasing exec, a bean counter who happens to be as technically inclined as Jennifer Love Hewitt and not nearly as curvy.
The items are purchased by part number, and because the purchasing exec left the order until much too late, he requests that it be overnighted -- incidentally adding a few grand to an already strained initial budget.
So, the purchasing exec signs the overnight APC order. And what does APC do? Well, it sends a truck that arrives at the construction site overnight -- which happens to be Saturday. No execs, no IT staff, not even any construction workers. Just a security guy, who at least knew to call someone who got back to my friend.
This friend is two states away and now winds up calling anyone he knows to find a pallet jack to rescue his $300 grand worth of IT equipment that's sitting outside on a loading dock. In the rain.
Fallout: Yet more budget-busting corporate credit card charges as local movers had to be called to the site on a rush order to move all the stuff inside. The equipment got dry, but the budget post-mortem was no fun.
Moral: If you're spending significant dollars on any kind of specialised equipment, make sure the order gets tracked by someone who understands what's being bought -- whether or not that person can actually place the order. If he doesn't, spend a buck and buy him a clue.
Trick No. 2: Yet another cleaning lady story
Incident: Dust bunnies collect everywhere. Even in server rooms with rubber floors, sleek black racks, and loads of fans. Why this happens has eluded even DARPA's finest scientific minds. What's also eluded DARPA's brains trust is why offices routinely allow their cleaning people access to critical server rooms during unsupervised off-hours.
Sometimes they know to leave the humming, blinking boxes alone. Other times, as was the case in this instance submitted by a reader identifying himself as D. Lartner, they see dust bunnies not just around the server racks, but in them as well. Whereupon they open the racks. Whereupon they see servers with their cases partially open and little dust bunnies bouncing around because the sysadmin was more interested in eating donuts than keeping his machines clean.
So the cleaning people did what cleaning people do: they cleaned. Around the racks, inside the racks, and inside the servers. With Windex. Zot.
Fallout: A new cleaning crew (which I thought was unfair); a stern talking-to administered to the sysadmin; and a new cleaning schedule that includes the server room only when other folks are in the office.
Moral: Two, this time. First, that cleaning people always get blamed because everybody needs a scapegoat. Second, don't expect everyone to know what you know about electronics -- even if you think it's obvious.
Trick No. 3: Figureheads can hurt you
Incident: Once again, it's an issue of executive control. According to a reader who wishes to remain nameless, there once was a vice president of information technology -- the MBA kind, not the techie kind. He's a fan of staff meetings but never knows what's being said at these events because geeks aren't helpful that way unless one asks them right -- and brings sugar and Red Bull or a stripper.
So he takes matters into his own hands and enrols in some online Exchange administration classes. Nothing wrong with that, except that he doesn't tell anyone because he wants to "keep his edge". Yeah, that's smart.
That's also where the current Exchange admin messed up. He kept a spreadsheet of access passwords in his desk and allowed his boss to know of its existence -- after all, the boss would never use them, right? Wrong. After he'd passed his way into Exchange Management 201, the VP of IT snagged a password from the admin's desk and took a tour of the server stats, where he found that Exchange message store was running at more than 65 percent. A huge problem, he decides, because his instructor said best practices dictate 50 percent or less at all times.
Not realising that the current Exchange team routinely allows the message store to run as high as 70 percent (whereupon they receive an auto-warning and begin to prune files), Mr VP decides he's going to solve this problem all by himself and stick it to them at the next staff meeting.
First, he prunes by date and nothing else, thus deleting a couple gigs' worth of e-mails people still needed. Then he gains access to some senior executive e-mail stores and can't resist spending some time poking around where he shouldn't because they apparently didn't get into audit trails at this point in his Exchange curriculum. Naturally, this is a big red flag on the audit logs, which the real Exchange team peruses while trying to figure out what happened to those 2Gb of missing e-mail. They were stumped for a bit because he was using someone else's credentials, but some social engineering caught him later that day.
Fallout: The VP got fired -- not for losing the e-mail, but for sticking his snout above his pay grade. The senior Exchange administrator got reprimanded for the password spreadsheet. The next VP of technology had both an MBA and a computer science degree.
Moral: Being a senior IT exec requires a lot of business knowledge but it still requires some technical chops, too -- and maybe a side order of integrity.
Trick No. 4: Locked data centre?! Well, let me get that for you
Incident: This was an audacious bit of hardware thievery, submitted to us by yet another anonymous reader. According to the reader: "We got called in to restore an entire office server farm from tape because the client said his hard disks were ... gone."
It seems that management decided the office manager had to have access to every room in the office -- or rather the office manager had complained so loudly and for so long that management finally got tired of listening to her and gave in.
This operation ran a certain amount of business off-hours on the east coast because it had customers on the west coast and in Asia. The night before our hapless reader got sucked into this restore, a repair tech walked into the office with a tool bag, wearing a golf shirt that carried the logo of the organisation's usual computer support company.
He said he had to upgrade one of the servers and even had a "work order" with one of the boss's signatures on it. The office manager glanced at the work order and then, in a hurry to get back to her phone, opened up the server room to this guy.
She wasn't worried because off-hour operations revolved mostly around the phone, not the PCs. Ow.
Geek Bond strolls out about 45 minutes later, smiling and assuring the office manager that everything's fine now. Then he leaves. A few minutes later, someone happens to try to access e-mail only to find that the server is "down".
The office manager angrily calls the computer support company demanding they send the technician back, only to find out they never sent a technician in the first place. Operations are shut down because the support company says it can't send anyone until the following morning. Enter storm clouds, stage left.
Before that happens, though, a computer-savvy employee who doubles as the in-house desktop support guru decides to check out the server himself the following morning. No wonder it was down. Hard disks, CPUs and, in some cases, the system RAM has been neatly removed from every server -- and apparently placed in the tech thief's tool bag.
That's a big problem when the actual tech from the support company finally shows up because (1) there isn't much he can do without hardware replacements, and (2) the office manager and the boss start blaming the tech for the problem. This, even though the "work order" was an obvious fake with a signature not even close to that of the actual boss -- something the office manager would have seen if she'd looked closely enough.
That conversation escalates to a phone call with the tech's boss, which leads to a sudden dissolution of the support contract. They wound up calling on our reader's consulting company because it was one of the few in the area that had spare parts and spare servers.
Fallout: Two days of downtime while the servers were rebuilt or replaced entirely and then restored from tape. Nothing happened to the office manager, other than a stern talking-to. But the company wrote up a strong policy detailing who was allowed access to the server room and why. Maybe an APB on an Aston Martin would have been a good idea.
Moral: If you're protecting something of value, you need more than just a lock. You need to manage the keys.
Trick No. 5: Green is great unless it's due to nausea
Incident: This came from a sysadmin who worked for one of our consulting clients. A senior exec went on a green kick. Everything had to be recycled, including old PCs. Unfortunately, instead of talking to reps from each department on how to handle this, the exec simply designated a team of buddies to dole out tasks, which meant that a non-IT staffer was in charge of PC hardware recycling.
To be fair, the guy did his job. He calls a local agency that places decently configured, used PCs in local schools. This outfit even takes care of picking them up; all they want are machines with fully wiped hard disks so that their volunteer techs can install student versions of Windows.
The plan looks good. As the hardware lifecycle on a batch of old machines comes up, new ones are purchased, and the old ones are designated for the recycling bin. Unfortunately, with all the work of a hardware migration in front of them, the company's IT guys are more interested in the new ones than in wiping the old ones.
So they just stack the old ones in the downstairs storage room next to the loading dock with a sign on them saying, "To be recycled". What that sign should have said was: "Leave alone until we say otherwise, or die."
The non-IT exec sees the PC heap and calls for a weekend pickup. On Monday all the PCs are gone, but the IT guys are so busy with the new stuff that they don't even notice until Tuesday afternoon.
Fallout: These poor guys had to chase the PCs all the way to the processing centre, find them amid a few hundred others, and perform the hard disk wipes there. That, or get fired.
Moral: If you're worried about local data safety, then pound the priority into your IT staffers' heads. Maybe make a few of them roll to make your point.
Trick No. 6: Don't bail on e-mail
Incident: A case for covering your rear end when it comes to server support, submitted once again by the highly popular "Name Withheld". According to Mr Withheld: "We normally have an admin rotation for server problems on the weekends. But this time the staff was smaller because we'd just lost two techs in a single week. Both left to go to other jobs. I could have called an outside outfit to cover us on the weekend, but I just didn't make the time, and by Friday it was too late."
Seems Mr Withheld had plans for the weekend, as did his last remaining tech staffer. So he figured: "What the hell, we haven't had a problem in several months. It'll be okay."
Well it wasn't. One of his road-warrior execs left on Friday for a vacation. He knew enough to set an away message on his e-mail and to forward those e-mails to his home e-mail address. According to Withheld, this was back in the days of 10MB e-mail stores for most ISPs, and the exec forgot that his was almost full. The e-mails he gets Friday afternoon quickly fill up his home ISP account almost to capacity.
Saturday morning the exec's dad sends him an e-mail message with a dirty joke in it -- including explicit language. Withheld's e-mail server kicks off a delivery failure e-mail complete with a copy of the original e-mail, which goes to the exec's home ISP account. That's full now, so it in turn kicks off a delivery failure message of its own. Theirs has a copy of the e-mail in it, too. Boom, you've got an e-mail loop. Back and forth, "I can't deliver your message."
Withheld's e-mail server's disks fill up pretty quick after that, but neither he nor the other tech figure it out very quickly because both are frolicking away from their beepers. By the time they do figure it out, it's in the wee hours on Sunday morning, and Withheld winds up cutting his weekend short to drive all the way back to the office and take care of things.
Fallout: The company set up a better remote management package for all its servers and eliminated the ability for users to forward their own e-mail. "Instead we give them Web access to their office accounts."
Moral: Murphy is a mean mother. If he's got a chance to get you, he will. So take the time to cover yourself with the support staff you need. It's not like there aren't competent techs out there looking for work -- even part-time.
Trick No. 7: A truck load of Murphy
Incident: An apparent relative of Mr Name Withheld above, Mr Name Withheld Jr, sent this in: "I work for a shipping company. We manage a fleet of several thousand trucks and delivery vans with separate divisions for contract moving and parcel delivery. Our boss in the 80s was pretty good at looking ahead and paid to design our own asset management application.
"As I remember it, the whole thing was written in pieces of REXX and C++ that sent ASCII to one another. This worked great while the programmer was on staff with us. But when he left to do something more interesting, the boss made a deal to have him support the thing remotely. That's when the problems started."
Unfortunately for Withheld Jr, they weren't very big problems, just a glitch for a half hour here, a drop for a full hour there. Enough to get Witheld Jr's boss to complain around the water cooler, but never during an actual budget meeting with his own boss.
Withheld Jr got frustrated: "As our technical expertise grew, we realised we had more options -- especially by the late 90s when there were so many other, safer software platforms we could have used." But the boss of bosses had become accustomed to bragging about his super-app that did all the stuff his golf club friends' applications could do, but for a super-low investment that he finished paying 10 years earlier. Every request for a move or an upgrade was denied.
When Murphy finally got to these poor clowns, he got them good. It was the literal week-before-Christmas 1997. Shipping subcontract jobs were at a year-long high, as was the parcel delivery business.
"Suddenly the app just up and died -- hard. So hard, we thought it was a hardware thing at first. But that checked out, and the server-side of the app simply wouldn't restart." Calls to the programmer got his answering machine -- turns out he went on vacation and didn't tell anyone. The company was down for three days before they got a hold of him and then for another day before he could make everything work again. Meantime, the whole organisation reverted to phones, faxes, and hurriedly updated forms.
"We lost a big chunk of revenue in nonrecurring business after that," Witheld Jr said.
Fallout: "We finally got permission to move to a new system. Even that turned into a pain because the original application's data was in a proprietary format for which we had to hand-build a conversion process so we could even import it into the new application."
Moral: A smart technology investment is worth some bragging, but don't push Murphy too much because he always pushes back hard.
Trick No. 8: Cruel and unusual
Incident: I still have trouble believing that this happened only last year. Our consulting guys got called in to rebuild a server farm "from scratch", as the caller put it. That surprised us because we'd done some business with this client before and the client's on-site guy was decent. Calls to him for an explanation, however, went unanswered.
So we show up on-site and find out why: they'd fired him. This didn't come as a complete surprise, as he did have some personality issues that might have made him unpopular. But as it turns out, according to office gossip, they didn't just fire him. The CEO -- whom we didn't like either -- actually did an Ari Gold number on him. Fired him during a staff meeting, embarrassed him in front of everyone, screamed at him, told him he wasn't getting his agreed-on severance for cause.
I can't get into specifics, but let's say that the gossip showed that "cause" was highly arguable. Looked to us like he was getting blamed for a sales engineer screwing up at a client site.
So, obviously, the sysadmin is hurt and angry, but they just let this guy walk back to his office to spend the rest of the afternoon "packing". This is a guy who runs the company's entire IT operation, soup to nuts. Then they watch him walk out with a couple of boxes of stuff. Turns out it really was personal stuff; he didn't steal a thing. But he'd apparently seen the fact of the dismissal coming even if he didn't foresee the manner, because he had prepared.
Five hours after he left -- two hours after everyone had left for the day -- every server in the server farm decided to rebuild itself, as did the CEO's, CFO's and office manager's desktops. The sysadmin was nice enough not to take the backup tapes, however, so we could restore them to about a week before the incident, but it took almost three days.
And this was a small development shop, so he not only took out e-mails and doc files, but all the recent code, too. We had to restore a lot of that from users' local drives.
We couldn't prove anything against this guy because all the scripts -- if there were any -- ate themselves during the operation. And honestly, if what scuttlebutt says happened is true, that CEO deserved it. His employees didn't, though.
Fallout: Loads and loads of painstaking work.
Moral: If you're going to torture and execute your IT guy, make sure he doesn't have unsupervised access to the system after that. Don't you people watch 24?
Trick No. 9: One plan to rule them all
Incident: G. Myers was overseeing a full office migration -- at least from an IT perspective. The company was building a new office site in a nearby town over from the old one. As usual in these situations, the IT staff is tasked with making sure all the data, voice, fax and other ports are wired to the appropriate drops in the appropriate numbers.
According to Myers, everything gets drawn out to painful detail on the blueprints: dual data; single or dual voice depending on where you are; each drop located and numbered. Everyone checks off, the contractors are happy, and the site is built out. Finally, the electricians and cable guys come in and do the drops.
This is where the fun starts. The operations manager, who is definitely the boss in this scenario, says he does a walkthrough, checking everything against the blueprints. The IT guys are not given permission to take a day and go check the drops. The ops manager gives the OK; the walls are closed, and painters are called in. Organ music plays ominously in the background.
Worried, Myers goes to see the CEO a few days later under the cover of good-naturedly asking for a better office at the new site. The blueprints are in the CEO's office, spread out on his mini-conference table. Myers gets to look them over and starts asking questions about the state of the wiring. Things seem to check out, but then he notices a big red circle around a research lab room with a note saying, "No data".
Myers shows it to the CEO, the CEO calls in the ops guy, and all three of them stand around scratching their heads. They call someone on-site at the new office; sure enough, this room received no data drops of any kind -- and originally it was supposed to receive just less than 40.
They circle back with the contractor and the architect. It turns out a senior marketing guy thought that room said "Research Library" and decided that it didn't need data jacks -- which still doesn't make sense, but who's complaining? He decided to mark it that way and nobody checked twice.
Fallout: A busted budget. The walls had to be reopened, new cables run -- the works.
Moral: Always, always double- and triple-check important plans being implemented by contractors. That, or don't let them have access outside of project manager supervision.
Trick No. 10: Porn filters: Use 'em
Incident: A friend gave me this one, and yet another surprising story because it happened only four years ago. A senior management executive calls a status meeting in his office with two mid-level managers, both of whom are female, one of whom has a law degree. The senior guy has a large office, with a small, four-person meeting table at one end near the window and his large desk and credenza at the other. He's set up his desk so that his work area faces the interior of the office. His PC is located directly behind him on the credenza so that he has to turn around to use it. The screen faces the interior of the office. Stage set.
He sets up the meeting so that the two women are at the far end of the meeting table, facing his desk; he's on the other side of the table, desk behind him. The meeting goes along fine until his screensaver comes on. Apparently, he'd had it set to "slideshow", with the picture library coming from a rather extensive collection of XXX-rated, all-action-all-the-time porn he'd been collecting off the Internet.
The women don't know what to do, so they just finish the meeting trying to stare at the tabletop as much as possible. Then they trek out of his office as quickly as they can before he notices what's playing behind him.
Three days later, the second female manager files suit. They settle. The senior guy gets a talking-to -- but doesn't get fired. (Wow!) So, my friend gets called in to discuss the purchase of technology that might prevent this kind of problem in the future. My friend explains that the company's SonicWall appliance already has that capability built-in.
Management calls the company's IT guy, and he explains that he was told not to turn that function on by, surprise, the very senior exec who had his porn slideshow rolling. At this point, the CEO is grimacing, massaging the bridge of his nose with his thumb and forefinger. He needs aspirin forte.
Fallout: The SonicWall anti-porn filters got turned on. The senior exec left "to pursue other interests" three months later.
Moral: Strongly worded memos and HR policies only get you so far -- especially if you don't even enforce them when they're broken. Meantime, turn on those porn filters and kill that stuff where it lives.
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.