Patch management strategy

Last Updated on November 19, 2018 by Dave Farquhar

Vulnerability management and patch management are close relatives. In most companies, think of them as siblings who hate each other. That’s usually how it plays out. It doesn’t always have to be that way, but it takes some thought and strategy from both sides. Here are some ideas for patch management strategy.

My story

I’ve worked in both patch management and vulnerability management. I got stuck with patch management in 2002 because I was the junior guy, so I got the work nobody wanted. Patch management was stressful, but I was OK at it.

In 2005, I took a job working for a defense contractor. I got stuck with patch management because I was the junior guy and nobody else wanted it. See a pattern? I was still OK at it, but on this job, passing score was 100%, and there were these guys called security analysts who checked to make sure the patches actually applied.

I came in to work one day and my boss told me if I wanted to remain gainfully employed, I’d better get really good at applying patches. So I did. It wasn’t easy.

Come 2009, I got a chance to become a security analyst, which eventually led to me working in vulnerability management. That led to working for Qualys, a provider of vulnerability management software, for a time. That means I’ve talked patch and vulnerability management with a lot of people.

But we’ll talk patch management here. These are my own views, not necessarily those of Qualys.

How hacking works

Let me explain hacking briefly, as this can help. I’m sure you know the difference between programs and data. Most hacking involves injecting program code into data. What happens is the program code in that data runs into a bug in a regular program, and tricks the program code in the data into running. The host program may misbehave or even crash, but the hacker got what he wanted, so it doesn’t matter to him.

Some of these bugs work a lot better than others. The bug that Microsoft patch MS08-067 fixed is a classic. It’s not perfectly reliable, but it’s very reliable as bugs go.

Many exploits don’t give an attacker very much. But an attacker can run one exploit followed by another until they get what they want, even jumping from system to system if that’s what it takes.

Patching your vulnerabilities is how you stop this.

Patch management strategy flaws

The biggest flaw I see in patch management, especially early on, is creating an edict to patch everything. I have to be careful about the stories I tell, but let me just say I’ve heard people say they’re going to patch everything. I’m still waiting for the first to succeed. The work is too overwhelming.

Security analysts hate me for saying this, but you have to prioritize and work on part of the problem at a time. If you hand a sysadmin a 700-page PDF and say, “fix all this stuff,” they’re going to make a rude gesture at you behind your back and throw it away.

I once inherited a vulnerability management program that had been doing that for three years. I turned it around by asking the people who push patches to push a small number of patches and tell me what happens. I asked for 10. The 10 I asked for probably didn’t have the best security impact, but they had a good psychological impact, which was what I wanted.

If you’ve never pushed patches day in and day out for a living, I’ll give you an illustration. Have you ever attempted to read the Christian Bible, cover to cover? Few people have. Far fewer have succeeded. The only way to get through that book is to go in with a plan.

Patch management is the same way. You have to have a plan.

Where to start

For psychological impact, push the 10 most common missing patches in your environment every month. Over the course of six months, this measurably reduces the number of vulnerabilities in the environment and gives everyone some much-needed wins.

Now, here’s the problem with that approach. You’re fixing stuff but only prioritizing by commonality, not by risk. Most good security analysts will take issue with that approach.

So I don’t recommend staying with that approach forever. But it’s better to patch badly than not at all, so I have no problem whatsoever with telling you to jump start your patch management program by pushing the most common missing patches for a few months. And if you aren’t already, start implementing my patch management best practices.

Fixing what breaks

Here’s the other thing many security analysts don’t know. There’s more to pushing patches than just hitting a button that says “deploy.” Yes, that’s some of it, but patches can fail. Some fail more often than others. For me, the hardest things to patch were SQL Server and .NET but in other environments, it could be different.

Where you get good at patching is by finding the patches that failed and fixing them. A simple reboot works more often than it should, so that’s always the first thing to try. Sometimes installing by hand does the job. Sometimes it takes another trick. Sometimes you have to pull the log file and look for the error. Sometimes you have to uninstall, reboot, and reinstall. Sometimes systems just fall out of SCCM and/or WSUS and you have to wake them up.

Over time, you learn your environment and what it takes to get the tough patches down. It may take a couple of years. Some people won’t like that, but once you show them what you go through to fix one missing patch, they get an idea what you’re up against.

Partial failures

I want to talk for a minute about false positives. When you scan your network with a tool like Qualys or Nessus, it will inevitably find missing patches that your system says are installed. A sysadmin’s natural reaction is to say the tool is wrong.

Please don’t. No vulnerability scanner is infallible, but a good vulnerability scanner is right between 99.99 and 99.999% of the time.

When a patch fails, it doesn’t always fail in an error state. It may very well succeed to the point where the operating system thinks it’s there, but one or two registry keys got missed, or one file failed to change or got reverted after the fact. It’s pretty rare with run of the mill Microsoft patches, probably less than one percent of the time. With Adobe patches, it happens a lot more. But in a large company, one percent of the time means once a month, some patch is going to fail somewhere.

Always look at that last column of the report. It will tell you what file or registry key is wrong. If I couldn’t clean it up the conventional way, sometimes I replaced the file or the key myself, by hand. That’s not something you want an inexperienced administrator doing, but a good sysadmin can work that way when needed.

Patches falling off

The other thing that can happen sometimes is patches falling off the system. For any number of reasons, sometimes a system reverts a file in occasion. If that file happens to be a vulnerable version, the operating system and your patch deployment tool won’t know about it. It’s your vulnerability scanner’s job to find that and alert you to it.

What to do when the data is imperfect

One trap teams fall into is refusing to do any work when they find something wrong with the vulnerability data. When that happens, work together to find a few things you can agree are missing and fix those, in addition to pushing the new patches from that month.

As you lower the total number of missing patches, analysis gets easier. Much easier.

Gaining maturity

Eventually you need to go from a shotgun approach to a risk-based approach to patch management strategy.

I suggest you start by perusing the Verizon DBIR. Verizon offers a service where they come into hacked networks, kick the bad guys out, and hold the network together until they fix it to the point where you can take over again. The annual DBIR is their observations from doing this every year. They won’t tell you whose networks they fixed and won’t give every detail, but they’ll tell you what missing patches the bad guys used to get in.

What I want you to do is read the DBIR, note any patch they mention, and make sure you aren’t missing it. The bad guys exploit those same missing patches over and over because they tend to be widely available and reliable.

How to talk to a security analyst

When a security analyst hands you a list of missing patches, ask if it’s a scan report or a patch report. Some tools, like Qualys, give a patch report, which omits superseded patches. You can’t download MS08-067 anymore. You have to download and deploy MS12-054.

Always ask for a patch report, if they can produce one. It’s likely those thousands or millions of missing patches can be fixed by deploying a few hundred patches. That’s still a lot, but you can push a few hundred patches. I used to do that in a single weekend. You’ll have a greater degree of success if you roll out hundreds of patches in waves rather than all at once, but you have to do what the boss asks.

Using that patch report really helps once you get the stuff Verizon mentions in the DBIR fixed.

Do your best to collaborate. Ideally the security analyst ought to initiate the collaboration, but one way or another, you both need to try to understand the others’ goals, and figure out how to get your goals in alignment so you can both reach greater success.

Vulnerabilities vs patches

Vulnerability scanners measure the number of vulnerabilities. Most Microsoft patches fix more than one vulnerability, though. A Java patch may fix 60 or more vulnerabilities.  So if your vulnerability scanner says your system has 100 vulnerabilities in it, it doesn’t mean you failed 100 times. If it happens to be a month when Microsoft, Adobe, Google, Mozilla, and Oracle all released patches, 100 vulnerabilities might be normal.

What good looks like

You’ll never get to zero missing patches, if only because of systems occasionally reverting vulnerable files. And in a typical month, you’ll have between one and two dozen new patches to deploy.

So, in any given month, you’ll be pushing a dozen or two new patches to everything, and strategically deploying another dozen or two older patches to catch up.

What I’ve found working for Qualys is that most of the people who think they’re good at patching aren’t as good as they think. Most who think they are bad at patching are better at it than they think. And most who are starting out are too ambitious for their own good.

Patching isn’t a sprint. It’s a series of marathons, and there’s no way to change that.

You have to take it a day at a time, like one of the heroes of my childhood did, Terry Fox. If you’re unfamiliar with him, after losing a leg to cancer, Fox decided to run across Canada to raise money for cancer research. He made it as far as Thunder Bay, far short of his goal. Yet it was more than the equivalent of 127 marathons, much further most healthy people could do.

It starts with a single day, and making incremental progress every day. Even if it’s not as much as everyone would like, any progress is better than none.

Pushing patches won’t make you as big of a hero as Terry Fox, but Terry Fox’s attitude and approach is what it takes to get good at patch management.