My first really bad day in IT

Next weekend is Labor Day weekend. I can’t remember if it was one Thursday or two Thursdays before Labor Day weekend in 1997, but one of those two days happened to be the beginning of the first crisis of my career.

Whichever Thursday it was, it was getting close to midnight when my phone rang. It was Max. The print server wasn’t working. That happened a lot. That server had IBM’s Services for Macintosh on it, which never worked all that well, and, worse, tended to make the rest of the server act up a lot. That in and of itself shouldn’t have been a crisis. But I’m getting ahead of myself.

It was 1997. I was working for the University of Missouri. My job title was Network Support Specialist, but really, I did whatever needed to be done. We had one of the oldest Token-Ring networks outside of IBM, so there wasn’t all that much to administering that. My big project at the time was building up a nascent Windows NT 4.0 network to replace what happened to be the oldest OS/2 network in the world outside of IBM. There wasn’t anything wrong with OS/2, except that by 1997, even IBM had pretty much given up on it. If we wanted to run current software, we needed to migrate our network to something current.

The University of Missouri owns a daily newspaper, which they staff with journalism students. It gives them real-world experience. The newspaper hired Max to do their computer support for them. When Max ran into a problem he couldn’t solve, he called us.

Max wasn’t happy with the speed at which we were migrating to Windows NT. Max wasn’t considering there was a lot more to the journalism school’s network than just the newspaper, and building those servers and creating all those user accounts and moving all that data over took time.

Max and I also had some more history. Only a couple of weeks before, Max’s fiancee had come up to me and told me that Max beat her. I told her that situation wasn’t going to improve. She said everyone else was saying the same thing. I said everyone else was right, and she needed to get out. She broke up with him that night. Early the next morning, the first person to arrive at work found a note glued–glued!–to the front door of the building. It read: “Every man, woman and child that works in this building is a homebreaker and an unscrupulous [rude word starting with ‘b’] or [the other rude word that starts with ‘b’].”

Max never admitted to writing the note, but who else could it have been? And up to that point, I’d never been called a homebreaker before. I’d been called all those other things of course.

But none of that was running through my head at the moment. We had a server to fix. So I started down the recovery process. I asked if he’d tried rebooting, of course. He said yes, and it hadn’t helped.

“Oh, this will be easy,” I said. “Open a command prompt and type NET START SERVER.”

“OK.”

“What happened?”

“Nothing.”

Wrong answer. Wrong, wrong answer. The command NET START SERVER never just did nothing. Best-case scenario, it told you the services were starting, and then the service started. Worst-case scenario, it gave you an error message.

“What error message did you get?” I asked.

“Nothing.”

“You typed NET START SERVER and it did nothing?”

“Yes.”

“OK. I want you to type NET STOP SERVER and tell me what it does.”

“OK.”

“What did you get?”

“Nothing.”

So I had him cycle through NET STOP SERVER and NET START SERVER a few more times before I became convinced he was either lying to me, or something was really, truly, seriously wrong with the server. Finally I suggested he direct-connect the printer to one of the computers they used for layout and sneakernet whatever files they needed to move around via floppies. That’s not the ideal way to do things, but I’d done worse to get newspapers to press at deadline. I said I’d take a look at the server first thing in the morning, when I’d have some other experienced help available.

A few hours later, I arrived at work, earlier than usual, and went straight to the newsroom. The server was there, still showing the command prompts from the night before. And sure enough, he output from the commands I’d had Max type were completely normal. The server’s vital signs were all fine. Max had been lying.

So I ran to my office and grabbed someone else to get another opinion. Word of the previous night’s issues had already reached the office. The paper was late, late, late to press, and the newspaper management was livid. Another coworker, whose name was Brian, came with me to look the server over. Brian agreed there was nothing wrong with it, and that Max had been lying.

Was Max trying to force our hand, to make something go wrong that would cost a lot of money that he could blame on OS/2, so he could ride in on a white horse with some Windows NT servers and save the day?

I was thinking it. Brian said it. We went and talked to the newspaper’s business manager and told her what we’d found.

The first words out of her mouth were, “I’m not going to fire Max.”

OK, well, that wasn’t necessarily the first solution we were going to suggest anyway, but we certainly have a problem, and if the first thing you’re going to do is take that option off the table, it might not be the easiest problem for us to solve.

So Brian and I went back to the office to talk with the rest of our colleagues about what we were going to do next. We knew Max was a loose cannon, we’d caught him in a lie, we’d caught him trying to make the rest of us look bad, and it was pretty clear we didn’t want the guy to have administrative rights on our network anymore.

So we did what any sane and rational group of people would do. As soon as they opened, we retreated to Shakespeare’s, the legendary pizza place. We could talk it over without being interrupted by telephones or angry newspaper staff.

I don’t remember anymore exactly what contingency plan we came up with. Today we could just create an OU in Active Directory, drop Max and his servers in it, give him zero permissions anywhere else, and let him fail or succeed without any impact to the rest of our operations. That option didn’t exist in 1997. Interestingly, that’s a reasonable description of what I do for a living in 2011. I wonder what a psychologist would have to say about that?

Whatever our contingency plan was, it went out the door just as soon as we walked back to work. Max was sitting on the front steps. His boss had done the opposite of what she’d said she was going to do, and fired him.

So now we were dealing with a different problem: A disgruntled former employee. And in these days, we had no firewalls. Every server on our network was direct-connected to the Internet. In 1997, that was standard practice in academia, and none of us thought anything of it. But it also meant he could, potentially, get back on our network remotely.

So one of my coworkers disabled all of his accounts and changed the passwords on every account that had administrative access while I went to Max’s office to see what he had going on. I found a lot of servers in various states of disarray, but there was one, sitting in a corner, powered on and doing something. I’d seen that server before, and asked him about it. He got evasive. “What’s that server?” I asked.

“Oh, it’s for… stuff.”

“What kind of stuff?”

“Just stuff.”

A web server, serving up data he shouldn’t be handling at work, most likely. Whether it was illegal stuff or just inappropriate stuff was anybody’s guess. At any rate, it was on our network, and he was a former employee, and there was every reason to believe he still had a way to log into it. I pulled the network cable, but then I wanted to know what it was. None of our standard administrator accounts or passwords worked, of course. Eventually I busted the password on the administrator account. I didn’t find anything running on the server besides the usual services. Whether that meant Max never put the server to his intended use, or just covered his tracks that time, I don’t know.

And at that point, I had other work to do.

I spent the rest of the day racing around servers, running full backups. As far as we knew there was no way for Max to get in, but if he did manage to get in, we’d at least be able to use our backups to get right back to where we’d been on Friday afternoon.

While all of this was going on, one of the other coworkers dealt with Max himself. He thought to look through Max’s office for personal effects, found some things everyone else had missed, and he negotiated a deal. I’ve got some of your stuff, so now tell me what you know, and hand over anything you have of ours. Max caved, told him what he knew, and once he was satisfied with what Max had told him, he handed over the belongings.

The weekend passed without incident, as did the rest of the following week. And I never heard from Max again. It’s been about 14 years now, and I’ve dealt with bigger problems than Max in years since. But you never forget your first really bad day at work, and that was mine.

%d bloggers like this:
WordPress Appliance - Powered by TurnKey Linux