Home » Linux » Page 5


Optimizing dynamic Linux webservers

Linux + Apache + MySQL + PHP (LAMP) provides an outstanding foundation for building a web server, for, essentially, the value of your time. And the advantages over static pages are fairly obvious: Just look at this web site. Users can log in and post comments without me doing anything, and content on any page can change programmatically. In my site’s case, links to my most popular pages appear on the front page, and as their popularity changes, the links change.

The downside? Remember the days when people bragged about how their 66 MHz 486 was a perfectly good web server? Kiss those goodbye. For that matter, your old Pentium-120 or even your Pentium II-450 may not be good enough either. Unless you know these secrets…

First, the simple stuff. I talked about a year and a half ago about programs that optimize HTML by removing some extraneous tags and even give you a leg up on translating to cascading style sheets (CSS). That’s a starting point.

Graphics are another problem. People want lots of them, and digital cameras tend to add some extraneous bloat to them. Edit them in Photoshop or another popular image editor–which you undoubtedly will–and you’ll likely add another layer of bloat to them. I talked about Optimizing web graphics back in May 2002.

But what can you do on the server itself?

First, regardless of what you’re using, you should be running mod_gzip in order to compress your web server’s output. It works with virtually all modern web browsers, and those browsers that don’t work with it negotiate with the server to get non-compressed output. My 45K front page becomes 6K when compressed, which is better than a seven-fold increase. Suddenly my 128-meg uplink becomes more than half of a T1.

I’ve read several places that it takes less CPU time to compress content and send it than it does to send uncompressed content. On my P2-450, that seems to definitely be the case.

Unfortunately, mod_gzip is one of the most poorly documented Unix programs I’ve ever seen. I complained about this nearly three years ago, and the situation seems little improved.

A simple apt-get install libapache-mod-gzip in Debian doesn’t do the trick. You have to search /etc/apache/httpd.conf for the line that begins LoadModule gzip_module and uncomment it, then you have to add a few more lines. The lines to enable mod_gzip on TurboLinux didn’t save me this time–for one thing, it didn’t handle PHP output. For another, it didn’t seem to do anything at all on my Debian box.

Charlie Sebold to the rescue. He provided the following lines that worked for him on his Debian box, and they also worked for me:

# mod_gzip settings

mod_gzip_on Yes
mod_gzip_can_negotiate Yes
mod_gzip_add_header_count Yes
mod_gzip_minimum_file_size 400
mod_gzip_maximum_file_size 0
mod_gzip_temp_dir /tmp
mod_gzip_keep_workfiles No
mod_gzip_maximum_inmem_size 100000
mod_gzip_dechunk Yes

mod_gzip_item_include handler proxy-server
mod_gzip_item_include handler cgi-script

mod_gzip_item_include mime ^text/.*
mod_gzip_item_include mime ^application/postscript$
mod_gzip_item_include mime ^application/ms.*$
mod_gzip_item_include mime ^application/vnd.*$
mod_gzip_item_exclude mime ^application/x-javascript$
mod_gzip_item_exclude mime ^image/.*$
mod_gzip_item_include mime httpd/unix-directory
mod_gzip_item_include file .htm$
mod_gzip_item_include file .html$
mod_gzip_item_include file .php$
mod_gzip_item_include file .phtml$
mod_gzip_item_exclude file .css$

Gzipping anything below 400 bytes is pointless because of overhead, and Gzipping CSS and Javascript files breaks Netscape 4 part of the time.

Most of the examples I found online didn’t work for me. Charlie said he had to fiddle a long time to come up with those. They may or may not work for you. I hope they do. Of course, there may be room for tweaking, depending on the nature of your site, but if they work, they’re a good starting point.

Second, you can use a PHP accelerator. PHP is an interpreted language, which means that every time you run a PHP script, your server first has to translate the source code into machine language and run it. This can take longer than the output itself takes. PHP accelerators serve as a just-in-time compiler, which compiles the script and holds a copy in memory, so the next time someone accesses the page, the pre-compiled script runs. The result can sometimes be a tenfold increase in speed.

There are lots of them out there, but I settled on Ion Cube PHP Accelerator (phpa) because installation is a matter of downloading the appropriate pre-compiled binary, dumping it somewhere (I chose /usr/local/lib but you can put it anywhere you want), and adding a line to php.ini (in /etc/php4/apache on my Debian box):


Restart Apache, and suddenly PHP scripts execute up to 10 times faster.

PHPA isn’t open source and it isn’t Free Software. Turck MMCache is, so if you prefer GPL, you can use it.

With mod_gzip and phpa in place and working, my web server’s CPU usage rarely goes above 25 percent. Without them, three simultaneous requests from the outside world could saturate my CPU.

With them, my site still isn’t quite as fast as it was in 2000 when it was just serving up static HTML, but it’s awfully close. And it’s doing a lot more work.


Linux kernel 2.6.0 is in Debian Unstable

Fresh on the heels of the Linux Kernel 2.6.1 release, Kernel 2.6.0 made it into Debian Sid (unstable). I haven’t tried it yet, as I messed things up on my main Linux box when I tried to upgrade to a pre-release version of 2.6.
I really ought to put a Linux partition on my Athlon system and upgrade it to Debian Unstable with a 2.6 kernel.

For those of you who currently have a working Debian box, it might be apt-get upgrade time. See why I like Debian? You can upgrade to all the latest stuff with, at worst, two commands: apt-get update and apt-get upgrade, or apt-get update and apt-get distupgrade. No chasing down RPMs and dependencies, no waiting around for stuff to compile and wondering if it’ll work on your system. It lets you be cutting edge, yet conservative.

Slimming down the bird: A 2.6 kernel for PCs with 4-8 megs

In case you haven’t read about it elsewhere, a new branch of the Linux kernel called -tiny was quietly released last month. Its main intent is for embedded systems, but I can see all sorts of uses for it. The smaller the kernel, the faster it loads off a floppy, so it’d be great for boot disks. Likewise for diskless machines that boot off the network.
Some people claim that compiling the kernel with the -Os parameter, as this branch permits, causes faster operation than compilations with the normal -O2. The theory is that the system spends more time in user space than in kernel space, and therefore, the kernel will almost never be in system cache, and therefore, the smaller it is, the faster it will operate. I’m sure someone who knows some specifics of how CPUs and caches work could validate or refute this. Even Steve DeLassus, who has both an electrical engineering and a computer science degree from one of the best universities in the world for that kind of thing, admits that sometimes these specifics get over his head. So if Steve doesn’t always know, then how am I supposed to know?

But the argument sounds good on paper.

With this kernel slimmed down as much as possible (no networking), it can boot in 2 megs. With networking, it can boot in 2.5 megs. With a comfortable set of features, it runs in 4-8 megs, which a mainline kernel doesn’t always do anymore.

Obviously, this is a perfect companion for projects like asmutils and uClibc/Busybox/Tinylogin.

Easy and secure remote Linux/Unix file transfers with SCP

Sometimes you need to transfer files between Linux boxes, or between a Linux box and some other box, and setting up Samba or some other form of network file system may not be practical (maybe you only need to transfer a couple of files, or maybe it’s just a one-time thing) or possible (maybe there’s a firewall involved).
Well, you should already have SSH installed on your Linux boxes so you can remotely log in and administer them. On Debian, apt-get install ssh sshd. If you’re running distro based on Red Hat or UnitedLinux, you may have a little investigative work to do. (I’d help you, but I haven’t run anything but Debian for 2 or 3 years.)

The cool thing about SSH is that it not only does remote login, but it will also do remote file transfer. And unlike FTP, you don’t have to stumble around with a clumsy interface.

If you want to transfer files from a Windows box, just install PuTTY. I just downloaded the 240K PSCP.EXE file and copied it into my Windows directory. That way I don’t have to mess with paths, and it’s always available. Make sure you’re downloading the right version for your CPU. The Windows NT Alpha version won’t run on your Intel/AMD/VIA CPU. Incidentally, Putty.exe is a very good Telnet/SSH client and a must-have if you’re ever connecting remotely to Unix/Linux machines from Windows.

SSH includes a command called SCP. SCP works almost like the standard Unix CP command. All you to do access a remote file is append a username, followed by the @ sign, and the IP address of the remote server. SCP will then prompt you for a password.

Let’s say I want to move a file from my Linux workstation to my webserver:

scp logo.jpg root@

SCP will prompt me for my password. After I enter it, it’ll copy the file, including a nice progress bar and an ETA.

On a Windows machine with PuTTY installed, simply substitute the command pscp for scp.

I can copy the other way too:

scp root@ .

This command will grab a file from my webserver and drop it in the current working directory.

To speed up the transfers, add the -C switch, which turns on compression.

SCP is more secure than any other means of file transfer, it’s probably easier (since you already need SSH anyway), and since it’ll do data compression, it’s probably faster too.

FTE – a DOS-style editor for Linux

I don’t remember what I was looking for, but I found another DOS-style editor for Linux and Unix.

FTE is another editor that harkens back to the look of the typical DOS app of about 10 years ago, similar to SETEDIT. For casual editing, either program will do very nicely, and provide a look and feel comparable to the editor that came with DOS 5 and 6.

I’ve always liked SETEDIT, but it suffers from the same identity crisis as emacs. Is it an editor? An MP3 player? A desk calculator? All of the above? And while it’s workable over a remote terminal connection, it’s not as snappy as I’d like.

FTE is a little sluggish from afar but faster. I like how it gives me the ability to have multiple files open and deal with large blocks of text, and continue to use the same key sequences I’ve known and been using since early high school. Its syntax highlighting is definitely a nice feature. It takes that feature a bit further than SETEDIT. For example, it highlights the corresponding closing bracket when you move over the opening one.

FTE’s main advantage is that it’s already bundled with some distributions. There’s a Debian project page for it. And a Google search turns up anectotal evidence that it comes in recent versions of Suse, Red Hat and Mandrake as well. If you’re a DOS veteran who’s not enamored with vi or emacs, FTE’s probably worth a look.

For what it’s worth, I typically use nano, but FTE is definitely a whole lot more powerful.

Knoppix: A low-commitment way to try Linux

I’m posting this from a computer running Knoppix. Charlie brought a copy in to work the other day and handed it to me. I actually downloaded Knoppix several months ago but never burned myself a copy because I didn’t have any 80-minute CD-Rs–I’ve still got about 20 74-minute jobs I haven’t used yet.
Yep, it’s an excuse. I know a lot of people have several objections to Linux:

1. There’s no software, or at least nothing I’d want.
2. It’s hard to install.
3. If I do manage to install it, I can’t get a good graphical interface working.
4. It takes too long.
5. I really don’t want to go to the trouble of repartitioning my hard drive, and I don’t want to run it on my old computer because it’ll be slow.

Knoppix is a full Linux distribution on a CD. You pop it in a PC (it’s best if it has at least 128 megs of RAM) and boot off it. It detects your hardware and boots you into a full-blown KDE environment.

Software? It has three word processors, three spreadsheets, two web browsers, two graphical mail clients, a desktop publisher, an MP3 player, a bunch of games (including a Civilization clone) and a bunch more stuff. In short, far more software than the typical Windows user has installed.

And if you don’t like it, just go to the “K” menu and hit Logoff. It shuts down. Pop out the CD, and boot back into Windows or whatever else it was that was on that computer.

I really wish the people who talk about Linux on the desktop would take the few minutes it takes to download and boot Knoppix. Their reaction would be telling.

More on building under a small Linux environment

Well, I’ve been playing a little bit with Erik Anderson’s uClibc-based development environment mentioned in the previous two posts.
When I compile, I issue the command export CFLAGS='-Os -s -mcpu=i386 -march=i386' to create small-as-possible binaries. Using the default flags, the Links web browser balloons to nearly 2.6 megs on my dual Celeron, mostly due to the debug symbols. It drops to around 760K with those options. Specifying i386 binaries shrinks them down at the expense of some speed on some CPUs (especially 486s and first-generation Pentiums), so you have to set your priorities. It doesn’t matter nearly as much on newer CPUs. But I’m pretty sure if you’re interested in uClibc you’re not just running it on Pentium 4s.

For the record, Links compiles without warnings without doing anything special to its configuration and seems to run without incident (I immediately used it to locate and download more source code to compile). Samba’s more difficult, giving some warnings in various places. It may or may not require some special configuration in order to actually run (I didn’t have time tonight to test it), and of course that could result in some reduced functionality. The binaries total 9.3 meg, which isn’t bad considering it implements a complete Windows NT-compatible file server as well as some simple client utilities for connecting to NT shares on a network. The files themselves are about 20% smaller than on a stock Debian system.

Erik Anderson says the majority of Unix software will compile under uClibc, which is probably true. I generally see compiler warnings occasionally even when using a completely mainstream system.

So there is a uClibc-based Linux distribution

I think I found just what I needed. Somehow I overlooked it before. Right there on Erik Anderson’s uClibc page, near the bottom, there’s his uClibc development environment. What is it? A Linux distribution based on his uClibc, busybox and tinylogin userspace in addition to enough GNU tools to compile other stuff. If you don’t want all 150 megs’ worth, download his makefile and uncomment just the stuff you want.
It’s not a general-purpose Linux distribution. It’s intended as a development environment. But besides that, it would be perfect for running on a low-end PC, like a 386 or 486 laptop. You get the benefits of a modern kernel and a modern, in-development libc, but with everything designed to lower memory consumption. On an older PC with a slow hard disk, that all translates into better performance.

Now I’m not sure how much of a GUI you get, but frankly, an older laptop, especially a network-capable one, with this stuff and the excellent Links web browser, the machine would be useful. If SVGAlib and the SVGAlib-capable version of Links compile, then you could even have a graphical web browser on a low-octane machine. Wouldn’t that be cool?

A tiny Linux server distribution? Maybe?

OK, so we’ve been talking about NAS boxes at work. NAS (Network Attached Storage) is a simple server appliance. Plug this thing into the network and you’ve got an instant file server.
Problem is, they’re not that much less expensive than a file server, if at all.

Now, file serving isn’t a particularly CPU-intensive task. Put some decent-speed disks in a box with a simple CPU and some memory, running an embedded operating system, and you’ve got a NAS box, right? Sounds like a perfect job for Linux, right? And you can stuff a minimal Linux into 8 megs of disk space and save the overwhelming majority of your disk space for real work, right?

Well, I asked Charlie if I was completely crazy or not. He didn’t seem to think I was completely nuts. He did ask if I checked to see if anyone’s compiled Samba against uClibc, the alternative libc I was talking about using. I know one person has gotten Samba 2.2.8 to compile against a recent uClibc.

And I even found a project that downloads and compiles uClibc, TinyLogin and Busybox, essentially giving you a complete Linux environment in 600K of disk space, not counting the kernel. And it boots very quickly, even off a floppy. The only problem is that its tools are set up for the ancient Minix filesystem.

Charlie didn’t think running the enterprise on the Minix filesystem was one of my brighter ideas. Maybe I should be glad he didn’t tell me exactly what he was thinking.

Well, getting the system up and running with JFS or XFS probably won’t be much of a problem. Those filesystems are enterprise class if anything ever was.

I had difficulty getting Samba to compile though. I forget the exact error message I was getting.

I may have to opt for the uClibc-based Linux from Scratch, since it’s being actively maintained. That’ll be a bit more work.

I suspect it’s possible to get this combination of tools to work together though. I can’t imagine Quantum is running its Snap servers on Red Hat. I’m sure they’re using uClibc and other embedded tools in conjunction with Samba.

The question is how much more time I want to put into it. If indeed I ever get more time to put into it. The surprising thing to me is that nobody else has built and released this.

More on tiny but potentially modern Linux distributions

I found a couple of interesting things on Freshmeat today.
First, there’s a Linux-bootfloppy-from-scratch hint, in the spirit of Linux From Scratch, but using uClibc and Busybox in place of the full-sized standard GNU userspace. This is great for low-memory, low-horsepower machines like 386s and 486s.

I would think it would provide a basis for building small Linux distributions using other tools as well.

What other tools? Well, there’s skarnet.org, which provides bunches of small tools. The memory usage on skarnet’s web server, not counting the kernel, is 2.8 megs.

Skarnet’s work builds on that of Fefe, who provides dietlibc (yet another tiny libc) and a large number of small userspace tools. (These tools provide most of the basis for DietLinux, which I haven’t been able to figure out how to install, sadly. Some weekend I’ll sign up for the mailing list and give it another go.

And then there’s always asmutils, which is a set of tools written in pure x86 assembly language and doesn’t use a libc at all, and the e3 text editor, a 12K beauty that can use the keybindings for almost every popular editor, including two editors that incite people into religious wars.

These toolkits largely duplicate one another but not completely, so they could be complementary.

If you want to get really sick, you can try matching this kind of stuff up with Linux-Lite v1.00, which is a set of patches to the Linux 1.09 kernel dating back to 1998 or so to make it recognize things like ELF binaries. And there was another update in 2002 that lists fixes for the GCC 2.72 compiler in its changelog. I don’t know how these two projects were related, if at all, besides their common ancestry.

Or you could try using a 1.2 kernel. Of course compiling those kernels with a modern compiler could also be an issue. I’m intrigued by the possibility of a kernel that could itself use less than a meg, but I don’t know if I want to experiment that much.

And I’m trying to figure out my fascination with this stuff. Maybe it’s because I don’t like to see old equipment go to waste.