The recent Log4J vulnerability brought increased attention on 0 day attacks, but it’s a question that never really goes away. How common are zero day attacks?
Zero day attacks are less common than they seem because attackers don’t understand them as well as less exotic attacks. Managing those less exotic vulnerabilities is a real challenge for many organizations, but it’s something they must get a handle on to defend themselves effectively.
First, a disclaimer
This is my opinion. It represents a quiet consensus among security professionals, but you won’t find a lot of people willing to talk about this. It will come across as a hot take, and ranty. While my opinion, it is an informed opinion, based on a decade of patching in enterprise environments and about a decade of scanning enterprise environments for vulnerabilities. I am a security professional by day and I specialize in threat and vulnerability management. Studying this problem and solving it at scale is part of what I do for about 240 days out of the year.
It is also my opinion. My current and previous employers may agree with some of it. Or they may not. I do not speak for any of them.
Fear of zero day attacks is more common than the attacks themselves
When I first started in vulnerability management, once management figured out I knew my way around Windows, the CISO developed a Monday morning ritual. If I drove into work and saw his car, he would be waiting for me at my cubicle. If I came into work and didn’t see his car, I could expect he would swing by my cubicle after he arrived at work.
And then we would have a conversation. It was as predictable and formulaic as boy band music. Sometime between when I left on Friday and arrived on Monday, he’d heard about some new zero day and wanted to know how big of a deal it was.
Guess what? Except when the vulnerability was named Heartbleed, we always had a bigger problem in our environment that we could fix.
We have been conditioned to fear zero days, and that undeserved attention distracts us from bigger problems that we can do a whole lot more about. The result looks like a case of attention deficit disorder at mass scale.
And yet, when you read about the notorious high publicity breaches, and when you read the annual Verizon DBIR, it’s almost never a zero day attack that brought someone down. It was a misconfiguration or an unpatched vulnerability that did have an available update.
It’s like an irrational fear of getting hit by lightning that keeps you from washing your hands. You probably don’t get hit by lightning, but you get sick because you didn’t wash your hands.
What is a zero day?
I like to use this as a job interview question. I probably should use it a lot more frequently. Zero day, or 0day, is a term that comes from the old software piracy scene. Zero day originally meant a brand new release, less than 24 hours old.
Security professionals adopted that same terminology for vulnerabilities that do not have an update available from the publisher to fix them.
0 days get a lot of hype because they sell security products, and they help security researchers make a name for themselves.
And this is a problem. The fear is paralyzing, and it’s self-defeating. Security researchers, who I will give the benefit of the doubt that they mean well, think that the problem is lack of motivation. So if they drop enough zero days, they think, like a toxic boss, that eventually the world will get motivated enough to update their vulnerabilities.
But vulnerability backlogs aren’t because of laziness. So shortening deadlines and dropping more vulnerabilities faster makes the problem worse instead of better. That’s a bumper-sticker non-solution like, “well, we wouldn’t have problems if we’d use commercial software instead of open source software.” Someone’s already had that idea and tried it. You don’t solve complex problems with solutions that fit on a bumper sticker.
Why organizations have backlogs of vulnerabilities
If more security professionals had ever spent 6 months working as a system administrator in an enterprise environment, they would understand the problem isn’t lack of motivation. The process of deploying updates at enterprise scale is complex. Among other problems, the tooling is only 70-90 percent effective. IT departments are minimally staffed. The political process of making a change is difficult, lots of people have veto power, and it’s very easy for one thing to go wrong and the change to get denied.
Calling it a problem of laziness is itself intellectually lazy. But instead of studying the problem, they double down, create a lot of hype and instead of solving the problem that bothers them, they make it worse. Because the way you make a busy person get more things done isn’t by throwing more distractions their way.
I fixed 800,000 vulnerabilities during the most productive five years of my sysadmin career. But all it took to derail the process was one simple mistake. One mistake is all it takes for someone to pounce and say no. The deployment “solution” I used, Microsoft WSUS, works about 70 percent of the time, so I had to do a lot of work to close that 30 percent gap. But arguably my biggest problem was a pedantic middle manager named Ed. Every organization has an Ed. Some have lots of Eds.
Oh, and how did I fix 800,000 vulnerabilities when there are only 170,000 in existence? Multiple instances of the same vulnerability. Hello, Edward. It’s been a while.
Lack of staffing, tools that only work 70 percent of the time when passing score is 100 percent, and guys like Ed are all bigger problems than zero day attacks. And they are why you probably haven’t fixed the last brand-name zero day by the time the next one comes out.
How common zero day attacks really are
Zero day attacks are relatively uncommon, and for several good reasons. Many of them are practical.
Exploits are computer software. And their quality varies widely. Some work well and are easy to set up, but that is a small percentage of exploits. Less than 2%.
It’s rude to talk about this, but I’ll say it anyway. Most corporate computer networks are poorly updated but reasonably well defended. It is much easier to find money for security appliances than to hire another system administrator to deploy patches. And let’s not talk about how hard it is to get tools to deploy updates.
So if you are an attacker who has just managed to gain a foothold in a corporate network, you are facing a semi-hostile environment with a lot of surveillance, but a lot of holes you can exploit that are well understood.
In that situation, you don’t want to tip your hand. You are trying to gain more access. So you are going to turn to something well understood. It’s faster and easier, has a better chance of working because it’s proven, and if the security products in place do their job and stop it, you haven’t given up much. You lay low for a bit and try again. It’s a long game.
If you charge in with a zero day attack, it’s more work, may be less effective, and if something goes wrong, some product sends the details off to a vendor who can analyze and learn from it. Then you risk losing your zero day attack.
Breaches happen much more frequently because of forever days than because of zero days.
The one percenters
I had a classmate in high school who wanted to be an astronaut. One day last year he told a story. He said he looked into it. The way you become an astronaut, besides being really smart, is getting a degree in something NASA finds interesting, then enlisting in the military and becoming a pilot. The problem, he said, is that 1% of military recruits get to become pilots. And then 1% of pilots get to become astronauts.
Today, at the end of 2021 as I write this, there are approximately 170,000 known vulnerabilities in existence. Currently we are discovering about 20,000 new ones every year.
Something comparable in severity to Log4J happens approximately once every 18 months. So we’re talking one in 30,000. When one of those vulnerabilities appears, it may or may not be a zero day. Flip a coin. 2020’s big problem was called Zero Logon. My blog post about that set a record for traffic at my then-employer. It was a big deal a couple of years ago, but nobody talks about it now. Even though it’s more dangerous now than it was when it had hype.
The vulnerabilities that can hurt you are like military pilots. There aren’t a lot of them. The ones that can really hurt you are like astronauts. There are unknown unknowns, but as rare as the known ones are, those unknown ones that can do comparable damage are extremely rare. Think of how many astronauts you know, versus almost any other skilled job.
Dave’s way to cut through the hype
If you are interested in solving this problem, I have a two-step approach to handling it. It’s easier to describe than it is to execute on. But it’s possible to execute on it. I patched 800,000 vulnerabilities myself as a system administrator. My mean time to remediate was well under 30 days and my success rate was 100 percent. That high success rate is why I know about ETL files. You have to know about stuff like that to have that kind of success rate.
That wasn’t my team, I was a team of one. If you can find someone who patched a million vulnerabilities faster than me and with the same success rate and they say something different, listen to that person. I want to listen to that person too. But I haven’t found that person yet, so I think I’m worth listening to.
Deploying recent updates
The first thing you need to do is get religious about deploying the monthly updates that come out every patch Tuesday. It’s the second Tuesday of every month. Microsoft has been doing it that way for 20 years so people can plan. Not every vendor follows Microsoft’s schedule but just deploy the closest equivalent for every vendor. Imperfect timing is way better than never deploying at all.
I never, ever broke anything with my updates. The reason for that was because I had a test environment, and I stayed up on the news. If Microsoft released a problematic update, I pulled it from testing. Now, not everyone has a well designed environment like I had. If that’s the case for you, deploy a month behind. If Microsoft releases a bad update in December, it takes about 2 weeks for the word to get out. So if you deploy November updates in December, you’re dealing with known quantities. Just skip any update that has a bad reputation. A better one will be out the following month.
With this approach, you’ll be 30 days behind all the time, but it’s better to be 30 days behind than 10 years behind. If management deems being 30 days behind is an unacceptable risk, they can get you a test environment. My question for them is why being 10 years behind was acceptable but being 30 days behind isn’t. Don’t let perfection get in the way of getting better.
Deploying the new updates stops your backlog from growing. It will also cut into your backlog at some unpredictable rate. But it provides you protection against one of this month’s vulnerabilities becoming a really big deal 6 months from now. You already fixed it, so you’re not vulnerable.
And yes, old things sneaking up on you is a bigger problem than zero-day attacks.
Prioritizing the backlog
Next, you need to deal with that backlog. My employer, Nucleus Security, has a product that helps with analyzing your backlog. It pulls in your vulnerability scans and then rates each finding against threat intelligence from Mandiant, the leading incident response firm. If you get hacked, Mandiant is the company you call to regain control of your network, hold things together while you rebuild, and then tell you what went wrong. It’s better and cheaper to get their opinion of your network before you get breached then after.
Of all known vulnerabilities, Mandiant rates about 3% of them as high severity or critical severity. There’s another 3 percent or so that rate a medium risk. The remaining 94% are low risk. The theory goes if you know the five or six percent that stand a reasonable chance of being able to hurt you, you only need to fix five or six percent of your backlog.
And if this sales pitch sounds familiar, it’s about the same sales pitch that a competitor, Kenna Security, has been using with their threat intelligence product. I have used their product successfully in large enterprises, but I like the Nucleus and Mandiant solution better because of the analysis. Kenna gives me a score and some attributes, but Mandiant gives me detailed analysis written by a human being. They also don’t just tell me what vulnerabilities are happening in active attacks, they give names of groups who are using them.
Don’t get me wrong. Fixing 5-6% of your backlog still takes time. Expect it to take months, if not a couple of years. But at the end of a couple of years, you’ll be in a better place. If you maintain focus.
You want to only be susceptible to zero day attacks
It sounds counterintuitive, but you want to be susceptible only to unproven and exotic attacks. Because a would-be attacker may go ahead and try one of those unproven or exotic attacks, but when faced with an above-average network, they are more likely to move on to someone else. You’ve raised the ante. Your job is a security professional isn’t to make yourself invincible, it’s to make yourself too expensive to attack. Because invincibility isn’t attainable, but impracticality is. That is a core tenet of any security certification. Yes, I’m telling you that will probably be on your test. It was on mine.