What peer benchmarking is in vulnerability management

Successful vulnerability management is deceptively simple. It comes down to being able to answer yes to two questions: Are you fixing the right things? And are you fixing them fast enough? But how fast is fast enough? In this blog post, I’ll explain how I use peer benchmarking to help companies figure out how fast is fast enough. I’ll also explain how to know if your security policies are less popular than speed limit laws, and why that will make them fail.

My vulnerability management horror stories

Before I started working for security vendors, I worked two different places that had rather good vulnerability management programs. I also worked someplace whose vulnerability management program had more room for improvement than those two.

I know companies talk to each other about vulnerability management all the time. They will ask pointed questions. The questions include asking how quickly they fix critical vulnerabilities.

It’s unusual for a company to give a straight answer to that question. I won’t say everyone dodges the question, because I haven’t talked to everyone. And one could argue I haven’t talked to a statistically significant sample size. More on that in a little bit.

But I don’t need a statistically significant sample size to tell you the kind of responses I got used to seeing and that one former employer gave when asked.

Talking around the uncomfortable question

The answer was saying something like this: “Our policy is to fix critical vulnerabilities in x days.”

Well, my local government’s policy is that people drive no faster than 35 mph on the main road through my neighborhood. That doesn’t mean that’s the speed everyone drives. But there’s a reason that policy is set at 35 mph, and peer benchmarking helps us to apply the lessons learned from speed limits to complex problems like vulnerability management.

Because while I said vulnerability management is all about two simple questions, being in a position to honestly answer both of those questions in the affirmative is anything but easy.

Part of the problem is that companies make policies without having data to tell them if the policy is achievable. They know what everyone else’s policy is. But they don’t often have any hard data to indicate they are actually complying with it at scale.

Yet, when I come along and dare to suggest that people talk about policy when they aren’t comfortable disclosing the numbers they are actually reaching, the reaction I get suggests to me this is the first time they’ve heard someone make that assertion.

That’s where peer benchmarking comes into play.

What peer benchmarking is

Peer benchmarking is providing one or more metrics from your peers so you can compare them to yourself and see how you are doing.

I once heard of a CISO presenting their exposure score plus their industry exposure score at the quarterly board of directors meeting. They would give the two numbers, state that lower is better, and ask if anyone had any further questions. Because their score was noticeably lower, the only question anyone ever asked was how many companies they were being compared against. That’s an important question.

Tenable doesn’t have the only vulnerability management product that offers peer benchmarking. But, as far as I know, Tenable is the only one who will tell you that if they list a particular industry, they have data for at least 15 organizations in that industry.

Why 15? Statistical significance.

The importance of statistical significance

You probably know that no company calls up every single customer and asks their opinion of every product feature they are considering introducing. They would have to hire several full-time employees to dedicate to that task and only to that task. That would raise overhead, and they would have to raise prices. Furthermore, even if they were to do that, not everyone would pick up the phone. Sampling everyone is impractical and probably impossible.

But if a company talks to a large enough number of customers, it can have a reasonable degree of confidence that their answers are representative of the population of the entire target market, without talking to everyone.

Manufacturers use this technique when deciding whether to release a new product. Politicians use it to gauge how they are doing in their next election. Insurers apply statistical methods to publicly available data to figure out how much money to charge for insurance. No matter what business your company is in, there’s a reasonable chance it uses statistical methods somewhere in its decision-making. It’s unfortunate that it hasn’t gotten more widespread use in information security and information technology.

Applying peer benchmarking to vulnerability management

Peer benchmarking works like this: If there are 75 companies in my industry and I have data from 15 of them, I can be 80% confident that the averages that I take are within 15% of what the count would be if I were to measure the entire population. If I want a higher confidence and lower margin of error, I get data from more. But 80 percent confidence and a 15 percent margin of error gives you something to work with.

To put it another way, if I have data from 15 customers in a particular industry, I can be 80% confident that an average of their KPIs is within 15% of the entire population’s average.

When evaluating a vulnerability management product that offers peer benchmarking, make sure they have enough data to be statistically significant. Otherwise, the insights their data gives you may not be typical.

But if you have peer benchmarking with the right statistics and a statistically significant sample size, you can work wonders. You can use it to figure out if you are patching as quickly as your peers, how your success rate compares to theirs, and even whether the SLAs you and your peers are trying to meet are reasonable.

Using and applying peer benchmarking

Tenable’s Lumin product tells you your MTTR against critical, high, medium, and low vulnerabilities, along with your peers in your industry or your choice of other industries. It also tells you your remediation coverage, which is an underrated statistic. Remediation coverage is your success rate. It also tells you your peers’ remediation coverage. Remediation coverage is my new favorite stat, even if I usually slip and call it success rate.

You can go to the settings menu and change what industry you see. Not every industry acts as quickly as others. So you may be interested in how your numbers compare against both your industry and, say, the financial industry, which you might presume has a lower risk tolerance than yours. Personally, I find it interesting to compare other industries against the technology industry. Technology companies tend to have all of the newest deployment tools and strategies, after all. So I think it’s interesting to see if having better tools makes them faster, more successful, or both.

Keep in mind that MTTR is an average, and not everyone can be above average. This is real life, not a Garrison Keillor story. It’s great if you are beating the average, but if the average happens to be 14 and you are at 16, that doesn’t mean you’re doing badly. If the average is 14 and you’re struggling to break 30, then I might be concerned.

Why might I not be concerned? Success rate. I mean remediation coverage.

The importance of remediation coverage in peer benchmarking

No industry’s success rate is 100%. If your success rate is higher than your industry’s success rate, that’s an accomplishment. But if your success rate isn’t hitting a certain level, that SLA isn’t sustainable.

What’s a good target for remediation coverage, or success rate? That takes us back to my speed limit example. For about 20 years, the United States had a national speed limit of 55 miles per hour. It was one of the least popular laws we ever had, along the lines of prohibition. Furthermore, we had popular music, popular movies, and television shows about disobeying that speed limit. If you ever wondered why Burt Reynolds appeared in so many movies about fast cars between 1977 and 1983, that’s why.

The United States government had its civil engineers study the problem. The civil engineers concluded that any speed limit is unenforceable unless 85% of the population was willing to comply with it. Instead, they found compliance rates closer to 15%. During my time as a criminal justice reporter for the Columbia Missourian newspaper in the late 90s, local and state police repeatedly cited this study when I interviewed them, telling me if the public won’t comply with a law 85% of the time, it’s a bad law.

So if you find that your remediation coverage is significantly less than 85%, and your vulnerability management program is about as popular as the 55 mile an hour speed limit or prohibition, that’s a strong indicator that something needs to change. If your peers are sitting at 13 days, maybe try settling for what your peers are doing rather than trying to significantly outperform them. If you are lagging against your peers and have low remediation coverage, that’s an indication you may need additional people, better deployment tooling, or both.

For what it’s worth, the 85% rule is also useful when deciding if things like banning Wireshark is a good idea.

A tip for meeting good benchmarks

A successful VM program has higher remediation coverage and a lower MTTR than its peers.

Over the course of my day job, I find integrating Tenable’s exposure management tools with popular patch management tools like BigFix or Manage Engine allows organizations to achieve higher remediation coverage and lower MTTR. That’s because the integration takes away the human analysis necessary to determine you are deploying the best available update. A computer can perform that analysis several orders of magnitude faster than a human being can.

You can also use peer benchmarking to get important conversations started. Imagine going to your remediation team and asking them if they have any insight into why other companies in your industry struggle to fix updates successfully more than about 65% of the time. This depersonalizes the question. It potentially opens the door to get them thinking about what it would take to patch more successfully, and ultimately faster than your competitors.

It may take some time, but it’s likely that eventually, they will come up with some process improvements or technological improvements that can help. And talking in this way does two things. It plays into human beings’ natural competitive drive. Second, it’s no longer about perfection.

Why you probably can’t fix everything

Fixing everything this month is possible. I know it because I’ve done it. Early in my career, I was a system administrator who specialized in patching. My calling card was clean Nessus scans at the end of the month. I fixed 800,000 vulnerabilities with an MTTR lower than 30 days, even on low severity vulnerabilities. But that employer spent a significant portion of our overall budget to achieve that. My involvement came to an end after another contractor said they could do it cheaper. I don’t have specifics and it would be wrong for me to disclose them if I did. But I heard a few months later that their success rate and speed slipped after I was gone. Management had to make a decision whether to spend more money or settle for lesser results.

And I don’t envy their position. That’s because I have data now that they didn’t have all those years ago.

While a 100 percent success rate is achievable for a sustained period of time, it costs more than most companies are willing to spend. Figuring out what to settle for when 100 percent costs too much is difficult. But you can use peer benchmarking to start to figure it out. First you start with what success rate and MTTR your peers are achieving, because they probably have similar limitations to you. Then you can make adjustments so you reach 85 percent remediation coverage. If at that point your MTTR isn’t fast enough, you can make further adjustments to figure out how fast you can get while staying within budget and maintaining that 85 percent success rate. It takes the right people, good processes, and good technology to get the balance right. You need all three.

A sad story with a happy ending

Here’s a sad story with a happy ending. I’ve had a couple of people ask me in my day job how to use peer benchmarking. I walked them through it, and I saw they were doing really well. I showed them how to use the numbers so they could prove it. At first it made me sad, because they were good and didn’t know it. But the conversation ended with them knowing where to find the data they needed so they could prove what was working and where they still had a bit more room for improvement. They could also prove whether those improvements were diminishing returns, or worth pursuing. So the story ended really well. It’s always a good day when that happens.

I hope this blog post has given you some ideas of how you can use statistical methods and peer benchmarking to level up your vulnerability management game. That would make today a good day.

Legal stuff

Tenable, Nessus, and Lumin are trademarks of Tenable Network Security, Inc. Opinions expressed in this blog post are not necessarily the opinion of Tenable Network Security, or anyone else I may have worked for in the past. This blog post was neither sanctioned nor approved by Tenable.