Why Johnny Can’t Patch

I’m going to break my normal cadence to write a little bit about computer security. It’s been a rough month, and this has been a rough week. In late October we had a high publicity vulnerability in PHP. In November we had one in Nginx, Ruby, Java, and PHP again. This rapid fire strained the industry, to say the least.

Why listen to me?

Why Johnny can't patch
In 1966, Rudolf Flesch published a book about illiteracy called Why Johnny Can’t Read that exposed a problem nobody wanted to talk about. Today, corporate America has a hidden problem that no one wants to talk about either.

My name is Dave. I’ve been finding and/or fixing vulnerabilities in corporate networks for almost 25 years.

I will say there are a lot of people who have opinions about patching vulnerabilities who have never done it before. I spent 4 years patching vulnerabilities for a living, exclusively. My client was the United States Air Force. The system I maintained supported two wars. Keeping that system secure was a matter of life and death.

In those 4 years, I resolved approximately 900,000 vulnerabilities, with a success rate of 100%, and exactly one missed deadline. (I missed that deadline by less than 24 hours.) My mean time to remediate was approximately 21 days. So that’s why I think I’m worth listening to. Also, there was no one to teach me or coach me. I had to figure it out.

In the following years, I worked in corporate security, largely in vulnerability management. I went from being the guy who patches vulnerabilities to being the guy who scans for them and reports on them. After a couple of years of that, one of the companies who makes vulnerability scanners hired me. Today I work for another company that makes vulnerability scanners.

In case you’re wondering, I am writing in my own capacity here. My views do not necessarily represent those of my current or any former employer. I am writing this on my own time, hosting it on my own server, and it was neither sanctioned nor endorsed by anyone else.

A few definitions

I expect not everyone reading this is an IT professional or a security professional. But you’re curious, because you’ve been hearing about these vulnerabilities in the news. My neighbor asked me about them. He’s a preacher. A literal preacher, as in church. Security vulnerabilities affect all of us. When companies don’t fix vulnerabilities, hackers can exploit them and steal company data. About once a year, I get a notice in the mail that some company I do business with was breached, a hacker got data about me, and the company subscribed me to a credit monitoring service. Every once in a while there’s a class action lawsuit and I get a settlement check. The last check I received was for $12. Sometimes it’s less than that. I’m sure you get these notices and small checks too.

A vulnerability is a flaw in computer software that allows someone with bad intentions to make the computer malfunction, and do something it wasn’t supposed to do. All software has flaws.

As software vendors find flaws, they release updates to fix them. These are called patches, a term that goes back to the 1940s. The original meaning no longer applies but people generally understand the concept of covering up holes in software. So we still call them patches.

There’s a widespread notion that large, profitable companies struggle to protect their data. One theory about why is that companies aren’t good at applying those updates, or patches. That’s why I called this blog post Why Johnny Can’t Patch.

The good news

November 2026 was a terrible month for technology. It started with a hangover from a bad security vulnerability that had been released in late October. It served up four more rapid-fire security disclosures the week before Thanksgiving in the United States. In between, we experienced disruptive outages from Microsoft 365, the Federal Aviation Administration’s air traffic control system, Amazon Web Services, JP Morgan Chase, United Airlines, Apple, and the National Security Agency.

Not all of these things were necessarily related. Theoretically some of them may have been. At least conceptually, some of them were.

There is a bit of silver lining to the security disclosures. Those were predictable. At least some of the wave was. Microsoft releases updates on the second Tuesday of every month. That’s been their cadence for more than 20 years, and when they started doing that, they did a pretty good job of publicizing it. Adobe follows the same schedule, and I think that was a good choice. People are already watching for updates on that day, it makes things a bit easier.

Other vendors do their own thing. For decades, Oracle released updates quarterly. I did not realize until this week that Oracle started releasing monthly updates in 2026. But there are two problems. First of all, you don’t get the monthly updates for free. You have to have a subscription to get them. Larry Ellison needs to buy more stuff, y’all. And rather than release on the second Tuesday like everyone else, they release theirs on the 3rd Tuesday. And while Oracle is being different, the monthly updates are not cumulative. The quarterly updates still are. But if you missed the October update, you have to apply both October and November. November doesn’t catch you up.

I also learned that PHP releases on a predictable release schedule. PHP, if you’re unaware, is a free, open-source programming language commonly used for web pages. About 70 percent of active web sites are written in PHP. They don’t follow the open source anarchy release-when-it’s-ready approach, and they haven’t since 2010. They update every 4 weeks on Thursdays.

Maybe it’s common knowledge that Oracle and PHP do monthly updates on a fixed schedule now, but at least in the circles I run in, that was not common knowledge.

In late October, a very serious security flaw in PHP was disclosed that sent corporate security and IT teams scrambling to fix it. About four weeks later, serious flaws in Oracle’s Java programming language, the open source web server Nginx, the open source programming language Ruby, and another security flaw in PHP were disclosed, along with patches to fix them.

Out of the November vulnpocalypse, only Nginx and Ruby were complete surprises. But we had four emergencies in November versus one emergency in October. And these types of very high publicity emergencies normally are fairly rare.

What’s different this time?

A quarter century ago, a Windows update generally fixed one bug. You got one patch per bug, and a month’s worth of updates might only fix 5 or 6 of them. Today, you get one patch that often fixes hundreds of bugs.

The world changed about a decade ago when company bug bounties became popular. Bug bounties are a way for companies to pay individuals for finding bugs in their software. They’ve existed since at least 1983, but became very widespread in the mid to late 2010s. Once several large companies started offering bug bounties, hobbyists found they could make more money finding and reporting vulnerabilities than they did at their day jobs, so they quit their jobs and found vulnerabilities full time. Every year since 2016 set a record for the number of CVEs discovered. It didn’t all happen at once in 2017. It’s just that 2017 was the year the dam broke.

The year 2026 was another change. In 2026, we’ve more than doubled last year’s count even though the year isn’t over yet. This year it’s AI. AI is finding vulnerabilities faster than we ever did before. It’s finding more serious vulnerabilities. And AI is sometimes helping to write the fixes. AI has its own problems, but it’s causing plenty of chaos on its way out.

Designer vulnerabilities

Another phenomenon started in the mid 2010s. We called them designer vulnerabilities. The first of these was a vulnerability in OpenSSL, a software component that encrypts traffic on the Internet to keep undesirables from reading our chats and seeing our credit card numbers. The people who discovered it wanted to make sure people took it seriously and fixed it so they gave it a name, a logo, and a marketing campaign. Its name was Heartbleed. At the time, I was working for a Fortune 20 company as a mid-level security analyst. It shot to the top of our priority list, and we fixed it quickly.

Before long, but seemed like every newly discovered vulnerability had a designer name, a logo, and at least a website. Sometimes they had a slick marketing campaign. Sometimes they had a really flat marketing campaign. And some took a very punk rock attitude toward it, scribbling a logo in Microsoft Paint just to meet the criteria of having a name and a logo in the most unprofessional way possible, thumbing their nose at the trend.

Security researchers know that large corporations are not keeping up with their patching. But they have not studied the underlying problem. They think the problem is IT departments aren’t motivated. And the corporate security department also doesn’t understand the problem, hasn’t studied it, and chalks it up to a motivation problem. The designer vulnerability trend was their attempt to increase motivation.

The designer vulnerability fad seems to be dying down a bit, probably because there are so many AI-discovered vulnerabilities that they can’t really stand out. I don’t miss it.

The staffing problem

Here’s another dirty secret. IT was a booming business in the 1990s. Everyone was hiring for IT jobs. My friends were asking me how to get into it, including a dental technician and a lifeguard. I used to know a truck driver who stopped driving trucks for a few years and got into IT. You can include me among those converts. I was a journalist before I got into IT.

I’m not saying this to be judgmental. The point was, IT was desperate for workers and very little formal training existed. Almost anyone who was willing to learn and could demonstrate some aptitude could get an IT job.

Then Y2K and the dotcom bust happened. Companies started cutting jobs. They outsourced a lot. They offshored a lot. But even counting the offshored and outsourced workers, it’s not uncommon for companies to have fewer IT workers than they had 25 years ago. They have many more computers than they used to have. They are supposed to make up the difference by doing more with less, working smarter not harder, and other management catchphrases that are worth less than the oxygen consumed to say them.

Patching saved my IT career. It was a job no one wanted because it was stressful and thankless. But it suited me. I can solve problems, break them down into a process, follow it, and look for ways to improve it over time. Knowing my job was safe as long as I met deadlines made up for the stress.

The contracting company I was working for even asked me to standardize my methods across the rest of the Air Force contracts they had. I found when doing so that we had all more or less figured out the same methodology independently. I learned a couple of tricks and taught my counterparts a couple of tricks.

A few years later, I found myself running vulnerability scans at a Fortune 20 company. What I found was disturbing. I found one problem in their methodology that I inferred from the scan data. Fixing that problem gave an immediate 55% improvement in their success rate. So I asked the IT staff if they had a documented methodology they followed. They did, so I asked if I could see it. I hoped I could find something else to change to get a big improvement. I found nothing. Their methodology was outstanding.

Several people in different environments had come up with equivalent methodologies. All on their own. There was no one to teach any of us.

So why did I have a 100% success rate that my Fortune 20 counterparts couldn’t replicate?

The major problem was they had one person trying manage 20 times as many computers as I was. And the tools that person had to do the work weren’t as good as the tools I had. Imagine a carpenter with expensive, professional-grade tools versus a carpenter with tools bought from the scratch-and-dent table at an off-brand tool store (or scavenged from the dumpster behind the store) and then expecting the one with scavenged tools to do 20 times the work.

Every other industry knows how many workers they need to get their day-to-day work done. The garage where I take our cars knows, given the number of mechanics they have and the number of bays they have, how many cars they can fix every week. Even the badly run fast food chain I worked for when I was 16 knew how many workers they needed to get through their lunch rush.

Yet corporate IT departments have no idea how many IT staff they need to maintain their computer fleet. They just keep buy more computers every year, lay off more employees every year, and tell the survivors to figure it out.

A quarter century of this ritual, plus a lack of understanding of how to size the staff means there is no budget to right-size. Even if there was a budget to right-size, there wouldn’t be enough workers. Decades of low demand pushed workers into other fields, or involuntary retirement. And when a company does decide to try to throw some money at the problem, they find they don’t know where to start because the problem has been piling up since 2002.

The tools problem

And the perfect storm has another element. A tools problem. Vulnerability scanners have improved a lot in the last 10 years, and much of that improvement has come in the last 5. Training for them is readily available, and at least when it comes to the two most popular scanners, they’re very accurate when configured correctly.

But patching tools, the tools you use to deploy the updates at scale across large companies, have not improved much since 2009. Some of them have gotten worse. I do know of a small number of companies who are trying to innovate in this space, but their tools haven’t gotten a lot of adoption.

And part of the problem is, a company will buy good tools, fix the problem, then switch to a cheaper tool because they think they don’t have a problem anymore. Then the solved problem becomes a problem again. That was one factor in me leaving that patching job in 2009. They switched tools to save a couple thousand bucks and turned a process that had been working well into a problem.

So we have a trifecta. We’re having to push updates to more systems every year, and IT staffs are shrinking rather than getting bigger, and the tools to fix the problem aren’t necessarily keeping pace with the tools to find the problem.

Automation

To some extent the problem can be solved with automation, and part of what I do now is helping people to build the strategies to automate away their patch debt. It’s harder work than it sounds. Imagine someone drowning, and they’re in so much distress that they don’t even know what they want. I’ve got the life preserver, there’s a rope on it, and I can throw them the life preserver. But I also have to convince them that they want the life preserver and they want me to pull them onto the boat.

Drowning in data that you don’t completely believe does weird things to you.

Bad KPIs

But another problem is that a week doesn’t go by that I don’t have to tell a security team what they want. The main thing they want is for critical vulnerabilities to be fixed within a week of their discovery. It’s news to most of them that that is what they should measure and aim for. I think sometimes it’s news to them that that’s what they want. When it’s not news, I had to remind them. And there are times they don’t believe me.

I do find that if I get people to focus on the average age of their not-yet-fixed vulnerabilities, and how long it took them to fix the ones they fixed, and make sure those metrics are in line with their stated security policy, they are much more successful.

What most companies do is just count the number of vulnerabilities and report that big scary number every month. The number never reaches zero, so that leads to a scary phenomenon called learned helplessness. And sadly, it’s possible to reach a point where you’re actually successful, but since you’re measuring a big scary number that will never reach zero because there’s always more work coming in, you don’t actually realize you’re successful.

Why Johnny won’t patch

And finally, sometimes it’s not a matter of Johnny not being able to patch. Sometimes Johnny won’t. Maybe it’s Johnny’s decision, or someone else’s decision.

Microsoft’s notorious outage in November 2026 was traced to a July security update. Maybe someday we’ll know if the bad code was written by an AI or a human. But that’s a dirty secret in the industry. Sometimes patches that fix security vulnerabilities break other things. It happens less often than people think, but people have long memories, and the perception that patches break stuff stops some companies from applying necessary updates. Officially, they have a policy. But off the books, someone is pocket vetoing the updates.

I was able to apply 100% of updates 100% of the time because we had a test facility. It wasn’t perfect, and I was a loud critic of its shortcomings. But it worked well enough that we never traced a production issue to any updates that I pushed out. But that test environment was better than any test environment I’ve encountered in the private sector since.

And it definitely helps for the system that you are updating to be designed with that kind of regular maintenance in mind. Many corporate systems are not designed with maintenance in mind. At least one of the November outages was caused at least partially by rushed maintenance that didn’t follow normal procedure. It would not surprise me if it was a factor in most of them.

Why the problem gets harder to solve every year

2026 is another record-breaking year. This year was different though. We broke the record early. I remember my boss bringing it up. We knew already that AI was causing us to find a lot more vulnerabilities than we ever had before. He speculated that maybe there would be a point in the future where there would be fewer bugs to find and we would stop having record-breaking years. It’s possible. I think I’m a bit more cynical.

I think there is one other thing at play. Hackers don’t take time off for the holidays. One of the most notorious vulnerabilities ever, log4shell, was disclosed around Christmas time. I think November and December tend to be heavy months for vulnerabilities. Heavier than usual. So what we have is a heavier than usual month in an already record breaking year.

I think some researchers try to get their vulnerabilities out before companies freeze all changes for their busy time of year. Retailers are busy in November and December, and insurance companies are busy in January. So they try to cram every discovery they can into November, knowing that anything they disclose later in the year is likely to get delayed and ignored.

We have a weird dynamic in this industry. The way you make money as a security researcher is by pouring a whole lot more water into that body of water where people are already drowning. There isn’t money for ordinary system administrators who fix the problem quietly. But there is big money for the people who discover the vulnerabilities. The monetary incentives are in the wrong place to solve the problem. Instead they usually make it worse.

I wish I had an easy answer. Some problems do have easy answers that fit on a bumper sticker, but this one isn’t one of them. It’s a nuanced problem that spans multiple disciplines and multiple companies, requiring a lot of collaboration to fix. Most people don’t stay in one place long enough to see it to the end.

Sadly, I think the problem is money. As of 2026, the average cost of a data breach is $4.99 million. But if a breach doesn’t happen every year, the incentive is to go cheap in terms of staffing and tooling, pocket the savings, and hope for the best. Until the average cost of a data breach reaches a point that scares companies, I think this problem will continue.

If you found this post informative or helpful, please share it!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.