Buffer overflows are a common topic on a Security+ exam. The textbook explanation of them is confusing, perhaps even wrong. I’ve never seen buffer overflows explained well.
So I’m going to give a simplified example and explanation of a buffer overflow, similar to the one I gave to the instructor, and then to the class.
What a buffer overflow looks like in memory
I drew a diagram on the board of a very simple program. It occupied a single continuous area of memory, divided into three blocks. The top and bottom blocks contained code, while the block somewhere in the middle contained data. It looked something like this:
code code data data code code
This is an especially bad design, because if you pass the program more data than the data block can hold, you overwrite the code in the bottom block.
How a buffer overflow works
That’s precisely what a buffer overflow (buffer overrun) does. Someone wanting to exploit this writes a simple program in native machine language that does something bad–perhaps changing the administrator or root password, installing some other software, scrambling data, or formatting a hard drive–and then determines the size of that data buffer. The person then passes that exact amount of data to the program, plus the bad-guy code he wants the system to run tacked on the end.
The bad-guy code overwrites some of the original program’s code, so when the program gets to those instructions, instead of doing what it’s supposed to do, it runs the bad-guy code instead.
Modern compilers and programming techniques make buffer overflow vulnerabilities a lot rarer than they used to be, but they still happen sometimes. I know that at least once in the last two years, Microsoft had to recall and reissue one of its monthly hotfixes because the hotfix had a buffer overflow vulnerability in it. So at least in theory, a system was more secure without that patch than it was with it. (I had fun trying to explain that to the higher-ups.) So they aren’t unheard of.
The best thing a programmer can do to avoid buffer overflows is keep code and data as separate as possible.
Remedies for buffer overflows
The textbook gave several remedies to prevent buffer overflows: Use firewalls, disable all unnecessary services and software, and keep all the latest security patches applied.
Another old trick, which is getting less practical over time, was to try not to put Intel/AMD x86 boxes directly on the Internet. They’d set up a Linux box running on something weird like an old PowerPC Macintosh or a Sony Playstation 3 and set it up as a proxy in front of it for extra protection. Today about the only practical architecture to use would be ARM.
The idea is when a script kiddie tries to pass an Intel buffer overflow to a different CPU architecture, the results will be unpredictable, but I can guarantee the overflow won’t do what the script kiddie intends, since the Intel instructions are gibberish to a PowerPC or MIPS or ARM or Alpha CPU. If the attacker figures out the architecture of the proxy, then they have to attack the other server. It’s not invincible but it increases the complexity of the attack.
So that’s what a buffer overflow is, and a few textbook answers about how to deal with them, plus a real-world solution that isn’t likely to find its way into any books any time soon.