I’m not sure any three words strike more fear into the hearts and minds of security analysts than the words “Qualys false positives.” Some number of false positives is unavoidable. But the perceived number of false positives is usually an order of magnitude larger than the real number of false positives. Here’s how to estimate how many you should have, how to investigate them, and break the gridlock.
While I hear sysadmins say all the time that Qualys isn’t accurate, that doesn’t mean they’re right. Equifax was breached partly because its vulnerability scanner wasn’t finding everything. What did Equifax do? Initially management blamed the system administrator. That should be a cautionary tale. In the end, Equifax switched to Qualys, because they needed accuracy.
Home Depot is another example of a high-profile business that suffered a breach, then brought in Qualys.
If Qualys were inaccurate, businesses that desperately need to save face and prevent a recurrence wouldn’t be buying it.
Estimating the number of false positives in your environment
When people tell me they have false positives in their Qualys scans, I tell them I believe them. This surprises them. Then I tell them how many false positives I think they have. It’s usually a much lower number than they want to hear. No solution in this space is 100% accurate. Qualys claims 99.99966% accuracy.
That means we can estimate how many errors will be in your scan. Take the number of live hosts you have, multiply it by 50,000 (the approximate number of checks Qualys can do), then multiply that by .9999966 (that’s five nines and two sixes).
In an environment with 50,000 hosts, that means you can expect 8,500 errors. Now, that’s errors. That’s both false positives and false negatives. That sounds like a lot, but Qualys is conducting two and a half billion checks.
I typically find more false negatives, where Qualys misses a vulnerability, than false positives. Those are tougher to find, because you have to actually look for them.
Realistically, the last time I investigated the false positives in a network with 50,000 hosts, they had about 400 false positives. Most of those were on Cisco devices.
False positive gridlock
It was 2006. The sysadmin in charge of patching got promoted. I had experience patching, so the boss decided I’d replace him. So I shadowed him for a week or so before he moved on.
I learned a couple of things from him. First, the words “false positive” made problems go away. But I could tell from the results column of the CSV files the security analysts were sending us that not all of them were false positives. I also noticed he was spending a couple of hours a week defending supposed false positives that I could fix in about half an hour.
I knew I knew more about how Windows works than those security analysts did. But I didn’t like wasting time. And improving security while being honest seemed like a good idea. So I decided to look at that results column and quietly fix what I could. When I couldn’t fix it, I asked questions, and more often than not we were able to figure out what we needed to fix.
Month over month, I installed the new updates as they came out, and worked with the security team to fix the small percentage of updates that failed. And when my sysadmin career came to an end and I moved into security myself, I’d fixed about 800,000 vulnerabilities.
Improving Qualys accuracy
The main cause of scan inaccuracies is running scans without authentication. Without authentication, Qualys has to probe behavior, rather than just checking file versions. Checking file versions is much more accurate, and also easier on the system. If you’re getting lots of false positives, ensure Qualys has accounts with administrator-level access across your enterprise. This means in Active Directory, on your Linux and Unix hosts, your network equipment, and your databases. Many of the false positives I see are due to network scans of systems that host Oracle products. Oracle doesn’t always report the full version number via a port scan, so Qualys will flag systems as vulnerable based on the partial match. An authenticated scan gives Qualys the correct version number, and therefore, correct results.
Backported patches also can cause these kinds of issues. Qualys knows about backporting, but if, say, you’re running SSH 7.7, the backported fix will still report version 7.7, which Qualys will flag as vulnerable. It may flag it as potential rather than confirmed, but still, not what you want to see in the report. With an authenticated scan, Qualys can check the file, compare it against the backport, and report correctly.
Investigating and reporting Qualys false positives
Investigating and reporting Qualys false positives is pretty easy. Look at the results section of your scan. This tells you what Qualys found that it objected to. If it says a particular update is missing, then it found something at the operating system level that suggests the patch either was never deployed, or it failed badly enough that it never told the operating system it finished. This could mean it failed to update the registry on Windows systems, or failed to update the package repository on Linux systems.
Re-installing that update, or an update that supersedes that update, should correct that issue.
More frequently, you find a file or a registry key in the results section. That tells you Qualys found a file that’s not the version that came with the update in question. For whatever reason that file failed to update. It could be the file was locked, or that the operating system reverted the file after the fact, or a disk error caused the journaling filesystem to revert the file. For whatever reason, that file is out of date now, and it rendered the system vulnerable again. Rebooting sometimes clears the fault, if the file is locked and pending a restart to update. If that doesn’t work, uninstalling and reinstalling the update is usually the fastest way to clear the fault.
If you check the file in question and find it doesn’t match what’s in the Qualys results, rescan the machine. It could be the file updated between the time you scanned and the time you investigated. If it persists, then open a support ticket with Qualys. Your security analyst will know how to do that. Qualys will need the evidence plus scan results for just the particular host in question in PDF format.
If the file matches what’s in the Qualys result, Qualys is right. Period. It doesn’t matter what SCCM says. SCCM is right about 85 percent of the time, while Qualys is right 99.99966% of the time.