perl script

A minor (but vital) Apache performance tweak

Dave Farquhar Apache, Linux , , , , ,

My problems seem to have become more rare since I started blocking spambots and tuned PHP and Apache but last night my server ran out of memory again and started timing out.

It turns out I still had a critical problem, but one that’s easy to fix with a relatively simple Perl script.

Read more

Encryption on the cheap

Dave Farquhar General , , , , ,

Disspam cruises along. It’s not often that I gush about a program, let alone a 4.5K Perl script, but Disspam continues to make my life easier. Granted, it simply takes advantage of existing network resources, but they’re resources that were previously (to my knowledge) limited to the mail administrator. Literally half my e-mail at home today was spam. Disspam caught every last piece.
A little scripting of my own. I’ve got a client at work who wants absolute privacy guaranteed. He and his assistant have some files they don’t want anyone else to be able to read, period. Well, there’s no way to guarantee that under NT, Unix, or VMS. Under NT, we can take away anyone else’s rights to read the file, but an administrator can give himself rights to read the file once again. We can make it set off all kinds of sirens if he does it, but that security isn’t good enough.

Well, the only way we can guarantee what they want is with encryption. But we’re nervous about making files that one and only one person can read, because last year, one of our executives went on vacation in Florida, fell ill, and died. We don’t want to be in a situation where critical information that a successor would need can’t be unlocked under any circumstance. So we need to encrypt in such a fashion that two people can unlock it, but only two. So the client’s backup is his assistant, and the assistant’s backup is the client. That way, if something ever happens to one of them, the other can unlock the files.

Password-protected Zip files are inadequate, because any computer manufactured within the past couple of years is more than fast enough to break the password through brute force in minutes, if not seconds. The same goes for password-protected Word and Excel documents. Windows 2000’s encryption makes it painfully easy to lock yourself out of your own files.

So I spent some time this afternoon trying to perfect a batch file that’ll take a directory, Zip it up with Info-Zip, then encrypt it with GnuPG. I chose those two programs because they’re platform-independent and open source, so there’s likely to always be some kind of support available for them, and this way we’re not subject to the whims of companies like NAI and PKWare. We’d be willing to pay for this capability, but this combination plus a little skullwork on my part is a better solution. For one, the results are compressed and encrypted, which commercial solutions usually aren’t. Since they may sometimes transfer the encrypted package over a dialup connection, the compression is important.

Plus, it’s really nice to not have to bother with procurement and license tracking. If 40 people decide they want this, we can just give it to them.

The biggest problem I ran into was that not all of the tools I had to use interpreted long filenames properly. Life would have been much easier if Windows 2000 had move and deltree commands as well. Essentially, here’s the algorithm I came up with:

Encrypt:
Zip up Private Documents subdirectory on user’s desktop
Encrypt resulting Zip file, dump file into My Documents
Back up My Documents to a network share

Decrypt and Restore:
Decrypt Zip file
Unzip file to C:Temp (I couldn’t get Unzip to go to %temp% properly)
Move files into Restored subdirectory on user’s desktop

I don’t present the batch files here yet because I’m not completely certain they work the right way every time yet.

They don’t quite have absolute security with this setup, but that’s where NTFS encryption comes in. If these guys are going to run this script every night to back the documents up, it’s no problem if they accidentally lock themselves out of those files. If their laptops get stolen, all local copies of the documents are encrypted so the thief won’t be able to read them. And the other user will be able to decrypt the copy stored on the server or on a backup tape. Or, I can be really slick and copy their GPG keys up onto the same network drive.

This job would be much easier with Linux and shell scripts–the language is far less clunky, and file naming is far less kludgy–but I have to make do. I guess in a pinch I could install the NT version of bash and the GNU utilities to give myself a Unixish environment to run the job, but that’s a lot more junk to install for a single purpose. That goes against my anti-bloat philosophy. I don’t believe in planning obsolescence. Besides, doing that would severely limit who could support this, and I don’t have to try to plant job security. I always get suspicious when people do things like that.

Stopping spam.

Dave Farquhar General , , , , , ,

Forget what I wrote yesterday. I was going to post the stuff I wrote in Ohio when I realized it isn’t all that good, it’s definitely not useful, and the people who annoy me the most are the people who can’t get over themselves. No one cares what I ate for breakfast, and the only people who care what went on in Ohio already know.
So here’s something useful instead. It’s the coolest thing I’ve found all year. Maybe all decade, for that matter.

Spam begone. I hate spam. It wastes my time and my bandwidth and, ultimately, my money. I’ve seen some estimates that spam costs ISPs as much as $5 per month per account. You’d better believe they’re passing those losses on to you.

There are tons and tons of anti-spam solutions out there, but most of them run on the mailserver side, so for an end-user to use them, they have to set up a mail server and either use it for mail or run fetchmail to pull the mail in from ISP’s mail servers. I’ve done that, but it’s convoluted. But that’s trivial compared to setting up the anti-spam kits.

I was crusing along, vaguely happy, when my local mailserver developed bad sectors on the hard drive, so one day when I went to read my mail, I heard clunking noises. I turned around, flipped on the power switch to the server’s attached monitor, and saw read errors. Hmm. I hope that mail wasn’t important…

Eventually I shut down my mail server and put up with the spam, hoping I’d come up with a better idea.

I found it in a Perl script called disspam.pl, written by Mina Naguib.

It took a little doing to get it running in Debian. Theoretically it’ll run on any OS that has Perl installed. Here’s what I did in Debian:

su (to become root)
apt-get install libnet-perl (Perl couldn’t see the network without this, so the next command in this sequence was failing. This hopefully isn’t necessary on other distros, as I have no idea what the equivalent would be.)
perl -MCPAN -e shell (as per readme–I accepted the defaults, then when it asked for CPAN servers, I told it my continent and country. Then it gave me 48 choices. I picked a handful at random, since none were any more obviously close to me than others.)
install Net::POP3 (as per readme)
quit
cp sample.conf disspam.conf
chmod 755 disspam.pl

Next, I loaded up disspam.conf into a text editor. It looks just like a Windows-ish INI file.

The second line gives me an exclude list. It’ll take names and e-mail addresses. So I put in a few important names that could possibly be blocked (friends with AOL and Hotmail addresses). That way if their ISPs ever misbehave and get blacklisted, their mail will still get to me. Then I popped down to the end of the file and configured my POP3 mailbox. I had an account I hadn’t read in a week, so I figured I’d get a good test. Just drop in your username, password, and POP3 server like you would for your e-mail client. If you have more than one account, copy and paste the section.

Bada bing, bada boom. You’re set. Run disspam.pl and watch. In my case, it flagged and deleted about a dozen messages, typical of what I usually get, like mail offering me Viagra or access to horny cheerleaders or how to find out anything about anyone (which I already know–I have a journalism degree). The only questionable thing it flagged was mail from MLB.com. I can’t get off their mailing list ever since I voted online for the All-Star game. No importa, I never read that mail anyway. I could have always added MLB.com to my exclude list if what they had to say mattered to me.

But if you’re like me and get lots of mail–that was my less-busy account–and about half of it is spam, that stuff’s going to scroll by really fast. So here’s what I recommend doing: when you execute disspam.pl, use the following command line:

~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log

Then you can examine disspam.log. If disspam ever deletes something it shouldn’t have, you can add the person to your exclude list and e-mail them to ask what they wanted. It looks to be less work than deleting all that spam. Probably less embarrassing too. Have you ever accidentally opened one of those horny cheerleader e-mail messages when there were people around? Yikes!

I fired up Ximian Evolution, pulled down my mail, and had 15 new messages. No spam. None. Sweet bliss.

It’s just version 0.05 and the author considers it beta, but I love it already.

Unix’s power allows you to string simple tools together to make powerful ones. Here are some suggestions.

You can e-mail the log to yourself with these commands:

mail -s disspam [your_address] rm ~/disspam/disspam.log

If you want the computer to do all the work for you, here’s the command sequence:

cronttab

Then add these entries:

0 0 * * * mail -s disspam [your_address] * 0 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log

If you read your mail on the same machine that runs disspam, you can substitute your user account name for your e-mail address and save your ISP a little traffic.

You’ll have to provide explicit paths for disspam.pl and disspam.conf.

The first entry causes it to mail the log at midnight, then delete the original. The second entry filters your inbox(es) on the hour, every hour. To filter more frequently you can add more lines:


* 10 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log
* 20 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log
* 30 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log
* 40 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log
* 50 * * * ~/disspam/disspam.pl ~/disspam/disspam.conf >> ~/disspam/disspam.log

This program shouldn’t be necessary for very long. It’s short and simple (4.5K worth of Perl) so there’s no reason why mail clients shouldn’t start incorporating similar code. Until they do, you run the risk of disspam and your mail client getting out of sync and some spam coming through. If you read your mail on a Linux box with an mbox-compliant client like Sylpheed or Balsa or Kmail, you can bring fetchmail into the equation. Then create a .fetchmailrc file in your home directory (name it ~/.fetchmailrc to ensure it goes to the right place). Here’s the format of .fetchmailrc:

poll SERVERNAME protocol PROTOCOL username NAME password PASSWORD

So here’s an example that would work for me:

poll mail.swbell.net protocol pop3 username dfarq password censored

Next, set your mail client to no longer check for mail automatically, then type crontab and edit your disspam lines so they read like this:

* 20 * * * disspam.pl disspam.conf >> ~/disspam.log ; fetchmail (your server name)

In case you’re interested, the semicolon tells Unix not to execute the second command until the first one is complete. If you have more than one mail account, add another fetchmail line.

As an aside, Evolution seems to use the mbox file format but it doesn’t store its file where fetchmail will find it. I think you could symlink /var/spool/mail/yourusername to ~/evolution/local/Inbox/mbox and it would work. I haven’t tried that little trick yet.

But even if you’re not ambitious enough to make it run automatically and integrate with all that other stuff, it’s still a killer utility you can run manually. And for that matter, if you can get Perl running on NT or even on a Mac, this ought to run on them as well.

Check it out. It’ll save you time and aggravation. And since it only reads the headers to decide what’s spam and what’s not, it’ll save bandwidth and, ultimately, it’ll save your ISP a little cash. Not tons, but every little bit can help. You can’t expect them to pass their savings on to you, but they’ll certainly pass their increased expenses on to you. So you might as well do a little something to lower those expenses if you can. Sometimes goodwill comes back around.