The decline and fall of system administration

Infoworld’s Paul Venizia stirred up a controversy, asking what happened to sysadmins who can fix things, as opposed to just rebuilding machines any time something went wrong.

The definition changed, mostly. At least that’s what I think.

When I started working in IT part-time in the mid 1990s to pay for college, an operator was someone who knew a fair bit about computers, enough to help other users with some things–especially things like creating accounts and resetting passwords–but may or may not have administrator rights.

A systems administrator, or sysadmin, knew the system inside and out. They could install the operating system, they could fix it when things went wrong, and most of them were pretty good with the hardware too. I remember two sysadmins vying for one-upmanship by comparing experience with soldering motherboards in the field. By the mid 1990s that skill wasn’t as necessary, but there were people who still expected a real sysadmin to be able to at least solder a jumper wire if the need arose.

An engineer was a guy who knew everything. You had software engineers, who wrote operating systems–that would be guys like Ken Thompson and Dennis Ritchie, who wrote Unix; Linus Torvalds, who wrote Linux; or Dave Cutler, who wrote VMS and Windows NT–and you had hardware engineers, who designed chips and systems and stuff. Those were guys like Amiga legends Jay Miner and Dave Haynie, but also the guys at Intel who designed CPUs.

I distinctly remember Dave Haynie once saying that you didn’t have to be an engineer to design a circuit board, that a good technician can do it.

Yeah, there was a time when they made that distinction.

But I guess what mattered most is that in days of yore, IT organizations didn’t have engineers. Computer companies had engineers.

Today, everything is down a tier or two. By 1990s standards, I’m a Windows sysadmin, and, in some organizations, my atrophied Unix skills would have only qualified me as an operator. I administered the Unix systems at my previous job, but what that really meant was applying patches, helping the auditors run through their checklists, running backups, shutting them down cleanly when there were power failures, and updating virus definitions. If anything went truly wrong with those systems, I didn’t know enough about them to be able to fix them.

But by 2008 standards, I was an engineer. My job title was “Senior Software Engineer.” There’s some irony there, as I was also the guy who replaced failed components. One day, when one of my five bosses was giving their boss a tour of the facility, I was in the process of disassembling a server and replacing some memory. The only time he saw me outside of the job interview and orientation, I was doing hardware work. Technician work, by 1990s standards. A real hardware engineer would roll his or her eyes if I tried to call swapping memory “engineering.” As would my oldest son’s godfather, who is a real, live mechanical engineer.

Some organizations contradict themselves by dividing engineers into two classes: operations/maintenance engineers, and engineering engineers. Engineering engineers are expected to fix things. Operations engineers… Well, they’re the guys who reinstall a server rather than fixing it. The choice of the word “operations” is curious. In 1997, they might not have even had the administrator password.

People look  at me curiously when I call myself a sysadmin today. By today’s standards, a sysadmin is, well, remember the dipstick with the screwdriver? It’s that guy.

I think the reason for the standards is twofold. Technology lowered the bar, and, frankly, most organizations probably can’t afford enough real, old-school sysadmins to take care of the vast herds of servers they’ve deployed. At my job in 1997, we had about 25 servers, and everyone thought that was a lot. We had five people to take care of them. In more recent years, I’ve worked for organizations that had 250 servers, but only 10-15 people to take care of them.

If a problem comes up and it can be fixed in an hour or two, that’s great, but management isn’t willing to pay extra for 15 of those types. Some aren’t willing to pay extra for even one of them. Not when a server can be built and deployed in two or three hours.

They’re more likely to be willing to pay extra for the ability to deploy and re-deploy faster.

The other reason, I suspect, is HR. Give the fancier job title instead of more money. It’s cheaper that way.

If you found this post informative or helpful, please share it!

2 thoughts on “The decline and fall of system administration

  • March 4, 2011 at 5:14 pm
    Permalink

    I think I may have mentioned this story before, but oh well. In the late 90s, I worked my way up from a support analyst to a help desk supervisor (a progression I don’t really support anymore). One of my job duties was to help interview potential new hires. In each interview I let HR and the other supervisor handle most of the heavy lifting. When they were done, I asked a single question: “a user calls who is having trouble printing to a network printer. How would you troubleshoot this?”

    I was always less interested in their answer than their thought process. The good analysts could break the problem down into different issues (network connectivity, printer issue, software/configuration error, etc.) and talk about isolating those areas. Other people would just throw their hands up in the air. “How would I know? I don’t even know your software or what kind of printers you own!” Those were the people, by and large, that proved to be worthless in the long run. And sometimes HR insisted that we hired those people, too. I never looked forward to training or working with them.

    Even though I’m in my late 30s, I’m already dealing with a new generation of techs underneath me. For grins, in general conversation I’ll present my little printer question to them (sometimes I’ll present it like a real life scenario) and I don’t think any of them give the old answer I used to look for. Many of them just say “I’d just get the error and Google it,” which I guess is a solution. Others shrug their shoulders or just ask me to e-mail them the details so they can ask other people. Many of them suggest reloading the workstation. “I had that problem once; reloading fixed it.” Again, often that’s true — firebombing a hotel will get rid of bedbugs …

    I think my generation, by way of our parents, grew up repairing things. It’s in our nature to find issues, and repair them. The generation below us replaces things. That’s all they know. (When’s the last time you took a television in for repair?) When that’s your mindset, troubleshooting and repairing are foreign concepts. It’s all about formatting and starting over, even for the simplest of problems.

  • March 4, 2011 at 8:27 pm
    Permalink

    Wow, Rob. I can’t say I’ve seen that before, because at this job I’m the youngest guy in my office. But that makes sense. I’ve frequently bought broken computers and game systems from people younger then me, then fixed them. And I don’t know about you, but when I was a kid, I would take the pieces from other kids’ broken action figures and make new ones out of them. That’s probably unheard of today.

    As for reloading the workstation because you can’t print, I guess that works if it’s a network problem. By the time you’re done re-imaging the workstation, someone else probably will have fixed the network issue.

    I guess I have a bright future ahead of me when the backlights in LCD TVs start burning out and I’m willing to open them up and replace them…

Comments are closed.