It was 2007, give or take a year. I was working a shop that had a WAN connecting four data centers around the world. A couple of hard drives in a SAN at one of the remote data centers had either failed or were in the process of failing.
No problem, we said. We’ll send some drives, and we’ll send along some extras so the next time it happens, you can just grab a spare off the shelf, slam it in, and not miss a beat.
Simple, right? Well, you should never underestimate a human being’s ability to make the simple difficult.
The drives arrived quickly. Unfortunately, the guy who got the package either didn’t understand our instructions, or he ignored them. We figured something was wrong when we noticed our remote Oracle servers started screaming, but I’ve gotten a bit ahead of myself.
The dipstick opened the box to find 10 drives inside. So he walked over to the SAN and yanked out 10 drives. Whether the two failed drives were among the 10 pulls was soon not to matter.
It turned out that the new drives didn’t fit. I can’t remember anymore if we messed up and sent him Fibre Channel drives when we should have sent SCSI drives, or if we just sent drives for the wrong type of SAN. It was probably the latter. The dipstick, unfortunately, knew just a little too much for his own good, so he went and grabbed a screwdriver and started improvising.
He removed the new drives from the new enclosures, then went to his pile of old drives, removed those drives from the enclosures, and started bolting the new drives into the old enclosures so they would fit.
Then he started placing the modified drives into the SAN.
By the time someone noticed that the Oracle servers were screaming that they’d lost all their database drives and we were able to get someone to go out into the remote datacenter and see what was going on, it was too late. The SAN had started replicating nothingness. The old data was sitting on the floor, all mixed up.
The dipstick with the screwdriver heard an earful, but it was too late.
Management assembled a crackerjack team of specialists to go on a road trip to rebuild some Oracle servers. Thankfully, I was in the middle of a patch cycle when this happened, so I was too busy to participate.
That was probably the only time I was glad to be installing patches. If there’s one thing I like less than installing Microsoft patches, it’s installing Oracle.
Our crackerjack team managed to get the pieces put back together and get the Oracle servers going again, though of course it took more time and cost more money than we originally anticipated. But I never heard what happened to the dipstick with the screwdriver.