Over the weekend of Nov 13, 2021, the FBI acknowledged unauthorized emails coming from a legitimate FBI email address to about 100,000 organizations warning them about ransomware. It appears to be the work of a self-styled white hat hacker, or security researcher.
I am a security professional. I am going to remind everyone that these are not the opinion of my current or any former employer. I have strong opinions on the, and those opinions are incredibly unpopular among security professionals. They may or may not agree with me privately, but agreeing with me publicly is not a great idea.
The motive
I have seen some discussion on social media from people who work outside of the computer industry wondering what on earth is going on here. Someone hacked into the FBI and sent out email messages. They were not malicious email messages, which suggests it wasn’t a bad guy who did it. So why would a good guy hack into the FBI and do this?
Did they seek to embarrass the FBI? Are they trying to tell the FBI they are lousy at their job? Trying to threaten the FBI? Are they showing off?
My professional opinion is it is a combination of all of those factors, along with some degree of immaturity and lack of professional experience.
The field of computer security is a much bigger mess than most people let on. It’s also possible that some people are not aware of what a mess it is. There is no easy solution to the problem, but that doesn’t stop some people from looking for one.
But what do I know?
Let me tell you about my security journey. I started my IT career in the mid-90s. The looming crisis of that time was Y2K. As the new guy, I got stuck doing a lot of Y2K work, including deploying Y2K updates and inventorying systems to ensure we didn’t miss any systems, and all the updates worked.
Soon after the Y2K crisis was over, Microsoft invented Patch Tuesday. I got stuck with those responsibilities, because I was still the least senior member of the team. So it was my job to deploy those updates.
It became my specialty. And from late 2005 to late 2009, my primary responsibility was deploying updates every month, and having a clean scan at the end of every month. I estimate that I fixed 800,000 vulnerabilities over the course of my sysadmin career. And the average age of those vulnerabilities was less than 30 days.
Today, I am a security professional, specializing in finding the vulnerabilities I used to fix. There is a big difference between 2005 Dave and 2021 Dave.
2021 Dave doesn’t fix security issues. 2021 Dave finds them and reports them. It is a completely different skill set from fixing them, and it uses a different set of tools. 2021 Dave finds the issues, and then is supposed to work with someone like 2005 Dave, and that person is the one who actually secures the network.
Most security professionals do not take the career path that I took. Most security professionals go straight to being 2021 Dave and they skip being 2005 Dave altogether.
Let me tell you what that looks like.
Teaching what you don’t know yourself
Imagine putting me in charge of fixing lawn mower engines. I have never rebuilt a lawn mower engine.
In the rest of the world, if I’m going to teach people how to rebuild lawn mower engines, or supervise them, I’m going to go get some experience. Maybe I’ll take a class at a community college, or at least go to the library and get a book about it or watch some Youtube videos.
Then I’m going to acquire a couple of broken lawn mowers to practice on. Then, ideally, I’ll spend some time fixing lawn mowers in my garage to get some experience. But from there, I probably would have to work for someone else for a while before I would be ready to supervise lawn mower mechanics.
The world of computer security is akin to me supervising lawn mower mechanics while having no experience fixing lawn mowers myself. And when that doesn’t go well, I conclude that the problem is that the lawn mower mechanics don’t know what they’re doing, or they’re lazy. The last thing that occurs to me is that my lack of experience is contributing to the problem.
So we have all of these security researchers running around finding vulnerable networks and vulnerable software. That is a skill. But it is a different skill from keeping that network running, and deploying updates to that network at scale while minimizing the disruption to the people who are trying to use that network.
But many of them do not understand the difference. So when other people can’t fix the problem as quickly as they would like, they resort to spectacle.
Security as spectacle
In 2010 at a security conference in Las Vegas, a security researcher from New Zealand named Barnaby Jack demonstrated the insecurity of ATMs by hacking an ATM on stage live and making it shoot out play money like a slot machine after winning the jackpot. It was more than a decade ago and people still talk about it.
Making an FBI email server spew out spam comes from that same mindset. It calls attention to a problem. It also motivates a quick solution, but at tremendous expense, and without any insight into what other things you are diverting money and attention from. And with no insights into what dependencies might need to be fixed first, before it’s possible to fix the problem you care about without disrupting other things.
The real problem: Staffing and funding
Why is the FBI’s network insecure? It’s the FBI, after all.
The same reason every damn corporate network in the United States is insecure. Under funding.
Yes, I said it. Every. Single. Corporate. Network. In. America. Is. Insecure.
And that is because IT departments are understaffed. I was successful because I had 500 computers that I needed to update every month. I had the best available tool for the job, and a manageable amount of computers that I was responsible for. I’m not quite sure what my upper limit was.
But then we decided to cheap out. Rather than using the best of breed Shavlik tool, management decided to use Microsoft WSUS because it’s free. It’s also a good 20% less effective.
The FBI budget received cuts in fiscal year is 2017, 2018, 2019, and 2020. No doubt its own IT budget suffered as a result of these cuts. IT budgets always get cut during tough times.
In corporate IT, it is rather common for staff to be tasked with 10 to 100 times the workload I had, with the crappy free Microsoft tool, rather than the professional grade Shavlik tool I had when I was successful.
Yet somehow, with tools that don’t work, and 10 to 100 times the workload, they are only 75% less effective than I was, on average.
But that’s a lot less interesting than making an FBI mail server spew spam to prove your point.
How to address the root cause of security problems
My controversial stance is that security departments would be best served by using their budget to buy patching tools based on proven Shavlik technology, and to look for involuntarily retired system administrators with Windows NT4 experience, then hire these people and hand them to their IT departments as patch specialists. Why NT4? That’s how you find people with 20+ years experience.
These are people who were senior level with a ton of experience who found themselves priced themselves out of the market due to consolidation and/or budget cuts. Find those highly experienced people, pay them what they used to make, and dedicate them solely to patching.
There are a fair number of those people but probably not enough to go around, so whoever does this first wins.
It will take a Y2K like effort, and Y2K like spending. And it will be semi-permanent. One would like to think that software will get more secure over time, but we have no way of anticipating when that will be. The number of vulnerabilities discovered every year is increasing, not decreasing. So for now we have to assume it’s going to be a long-term problem.
This is how you solve the problem. But it’s a lot less interesting than making an FBI mail server spew spam. That’s why I expect maybe 12 people in total to read this far. Of those 12, maybe 6 will agree with me. But they won’t have the decision making authority required to make the necessary changes.
The FBI may be able to get more money and higher up so they won’t be the next embarrassment. But there will be another one. Until we stop operating IT departments on shoestring budgets, computer security will remain a gigantic game of whack a mole.
David Farquhar is a computer security professional, entrepreneur, and author. He started his career as a part-time computer technician in 1994, worked his way up to system administrator by 1997, and has specialized in vulnerability management since 2013. He invests in real estate on the side and his hobbies include O gauge trains, baseball cards, and retro computers and video games. A University of Missouri graduate, he holds CISSP and Security+ certifications. He lives in St. Louis with his family.