War Stories

The ghost in the network

Dave Farquhar War Stories , , ,

My logging system died rather abruptly one week. It started with the Active Directory account some of our servers use locking. I got the account unlocked–someone else has those rights–and the system came back to life for a while, but then we had to repeat, and each time we repeated, “a while” grew shorter and shorter, bottoming out at about 2 minutes, 40 seconds.

The way you troubleshoot problems like this is by looking at logs. The problem is, you can’t collect very many logs in 2 minutes and 40 seconds.

Read more

DoSing your cubicle neighbor

Dave Farquhar security, War Stories

My baby at work is a centralized logging tool. That means my system has to touch every other system in this large company’s large network, which is kind of cool. Not many projects deal with that many different things, and I’m seeing some things I haven’t seen since college–and never expected to see in the real world, actually.

A week or two ago, we had some trouble pulling the logs in from a highly specialized system. That happens. Unix is easy, Windows is almost as easy–yes, the world of logging is a little bit upside down–but the one-off systems that don’t fit into neat categories take a lot longer to bring into the fold.

The problem was that the user account my tool uses kept getting locked out. Read more

How many Fortune 25 companies does it take to change a light bulb?

Dave Farquhar Humor/Satire, War Stories , , , , , ,

I’m working right now for a Fortune 25 company. This story is going to sound like bragging, so I’ll ask forgiveness in advance. Maybe if I mention I’m a contractor, then it’s not bragging quite so bad. Read more

Snickering at the Emachine

Dave Farquhar Hardware, War Stories , , , ,

For several years, I administered a command and control system for the U.S. Air Force. I sat in a datacenter, surrounded by racks jam-packed full of servers, and they kept the building at 64 degrees year round. I quickly learned to keep a jacket handy. I did several things, but mostly patch management.

Our system consisted of a diverse collection of Dell 1U and 5U servers, HP blades, and a couple of Sun SPARC boxes. It was a professional-looking setup, and except for the times we were doing massive system upgrades, the system generally worked as well as it looked.

Then we got a neighbor.

Read more

“Why do we have a server named ‘Vicious?'”

Dave Farquhar Retro Computing, War Stories , , , ,

My first non-food service, non-retail job was working desktop support for my college, the University of Missouri-Columbia. They were doing a massive computer upgrade and needed some part-time help. When they realized they’d found a journalism student who knew PC hardware and already knew OS/2, they cut the interview short and showed me around. I started work the next day.

My job was, initially, to unbox a few hundred IBM PC 330s and 350s, install network cards and memory, then install OS/2 on them. We had room for me to set up about 10 of them at a time, on long folding tables on opposite sides of a long room. It was lonely work at times, but I got to work with computers, and they were paying me $8 an hour. I liked it better than retail.

After a few days I had enough time to watch the boot process. OS/2 had a facility called Configuruation, Installation and Distribution (CID), similar to Microsoft’s unattended installation that appeared in later versions of Windows NT, that automated much of the process. An administrator configured machines in advance, and then when build time came, I booted off a floppy, entered a computer name, and the process pulled down what it needed from the network. After 30 minutes or so, we had a functional machine. CID probably saved a couple of hours of repetitive work. On this particular day, after I got nine machines going, I watched the 10th go through its the CID process. I noticed the machine kept addressing a server named \\VICIOUS.

Read more

Java. My bestest buddy.

Dave Farquhar War Stories

I patched Java on a bunch of systems this week, by hand. It was the first time I’ve done such a thing since probably sometime in 2007. So no one would blame me if it didn’t go 100% as planned, and, predictably, it didn’t. I did eight systems, and it worked on all but the first and last. Of course I didn’t discover that it failed on the first one until later in the day. Java itself seemed to work OK, but the log collector that requires Java didn’t.

The log collector is the only reason we have Java installed, so that’s not OK, of course. Read more

Troubleshooting at all layers of the OSI model

Dave Farquhar security, War Stories , , , , , , , ,

I saw this phrase in a job description last week: Troubleshooting at all layers of the OSI model. That sounds a bit intimidating, right?

Maybe at first. But let’s not overcomplicate it. Once you get past the terminology, it’s a logical way to locate and fix problems. Chances are you already do most of this whether you realize it or not. I was already troubleshooting at at least four of the seven layers when I was working as a part-time desktop support technician in college in 1995.

Read more

How I once took down a network, including a radio station

Dave Farquhar Hardware, Retro Computing, War Stories , , , , , ,

I met up Monday night with some other security professionals for some emergency networking of the professional kind. One of the attendees, a penetration tester, had a little incident where he took down a production system when he conducted his penetration test. The system owners were a bit arrogant, and, well, they paid for it.

I’ve taken down a network too, but in my case it wasn’t something security-related. No, in my case, I was a 20-year-old desktop support technician working in a college computer lab, making an honest mistake.

Read more

The phone in the server rack

Dave Farquhar War Stories , , ,

I spent about four years of my life working in a datacenter, administering a system comprised of about 200 computers supporting 20,000 users. I have some stories.

The facility had a lot of rules, some of them extremely petty. One of them involved telephones.

Read more

Dave vs. Mmm-Bop

Dave Farquhar Music, Personal, St. Louis, War Stories , , , ,

NPR recently released its Songs of the Summer, which invokes memories of summers past by conjuring up (or dredging up, in some cases) songs you couldn’t go anywhere without hearing. Songs like “Crazy” by Gnarls Barkley (2006), or “Hips Don’t Lie” by Shakira (also 2006). Or the bane of 1991, the unforgettable “Summertime” by the equally unforgettable DJ Jazzy Jeff & The Fresh Prince.

In 1997, one of the songs of the summer was “MMMBop” by boy-band Hanson. And mercifully, I avoided hearing it. I remember the summer of 1997. While everyone else was listening to that, I was listening to aging bands like The Cure and Echo and the Bunnymen, and that habit saved me. I managed to make it until 2004 without hearing that boy-band staple. It’s an achievement I’m proud of. Read more