How to increase the capacity of a Log Logic appliance by 45%

My 9-5 gig revolves primarily around Tibco LogLogic (I’ll write it as Log Logic going forward, as I write in English, not C++), which is a centralized logging product. The appliances collect logs from a variety of dissimilar systems and present you with a unified, web-based interface to search them. When something goes wrong, having all of the logs in one place is invaluable for figuring it out.

That value comes at a price. I don’t know exactly what these appliances cost, but generally speaking, $100,000 is a good starting point for an estimate. So what if I told you that you could store 45% more data on these expensive appliances, and increase their performance very modestly (2-5 percent) in the process? Read on.

Read more

We\’ll have to wait longer for PCI RAMdisks

In case nobody noticed, it’s August. July came and went, and there’s no Gigabyte I-RAM on the market yet.

But there are a few benchmarks out there, and Anandtech has an article that, once you get past the usual rambling and over-the-top introduction, has some useful insights.I was going to say the first problem is the somewhat disappointing speed, but actually, there are two bigger problems:

Availability. Now they’re saying it’ll be out sometime in August. And they’re initially only going to make 1,000 of them.

Price. The original $50 MSRP is out the window; now this thing is going to cost $150.

Can anything else be wrong? Unfortunately, yes. The speed is a bit disappointing. The SATA interface is the bottleneck. The very newest hard drives can come close to saturating the SATA interface for short periods of time, so the RAMdisk doesn’t outperform it by much. If this drive were using an interface with more bandwidth, there wouldn’t be as much problem, but squeezing more bandwidth out of the 33 MHz PCI bus is tough. We’re at the point now where the PCI bus is a much bigger bottleneck than the ISA bus was in 1994. The theoretical limit of the PCI bus is 132 megabytes per second, which isn’t much higher than the sustained throughput of 100 megabytes per second that the I-RAM delivers.

The combination of PCI Express and a faster disk protocol has the potential to resolve this issue, but at the expense of limiting the device’s market even further.

I’m disappointed by the review in a couple of regards, though. First, they compare the I-RAM to the fastest SATA drive available at the time of the review. That’s not necessarily what every would-be purchaser would be using. I believe that an I-RAM used to replace (or in conjunction with) a drive that’s a couple of years old would be a mind-blowing upgrade.

Second, they don’t take fragmentation into account. Enthusiasts are more likely to defragment their hard drives twice a day than everyone else, so fragmentation may not be an issue for them. But my wife, mother, and mother-in-law don’t know what fragmentation is. Well, maybe my wife does because she’s probably overheard me talk about it. The thing about the I-RAM is that it makes seek times irrelevant, so it’s never going to slow down due to fragmentation. Translation: For people who have lives, this thing could be phenomenal.

The review complained constantly about the drive’s capacity. So I’m disappointed that they didn’t test the drive with NTFS compression enabled. While data compression is still taboo, and it increases CPU usage, when you’re out of room it’s your only choice. While its effectiveness is unpredictable, it’s fairly safe to bet compression will get you another gigabyte or two of usable space on a 4-gig model. But just as importantly, under some circumstances, compression can actually increase performance. I want to know if increasing the amount of data you’re flowing over the saturated bus makes up for the increased CPU usage.

Floppies, meet your replacement

I must be the next-to-last person in the world to spend significant lengths of time experimenting with these, but for the benefit of the last person in the world, I’d like to talk about USB flash drives, also known as thumb drives (for a brand name), pen drives, or keychain drives, because they’re small enough to fit on a keychain.They are, as that popular brand name suggests, about the size of your thumb. It’s possible to buy one that holds as little as 64 megabytes of data, which is still a lot of Word and Excel files, but currently the sweet spot seems to be 512 megabytes or 1 GB. This is, of course, always a moving target, but as I write, it’s entirely possible to find a 512-meg drive for around $40, although sometimes you have to deal with rebates to get the price that low. It’s harder, but still possible, to get a 1 GB drive for under $90. That will change. Currently a 2 GB drive is more than $200.

I remember when people went ga-ga over a 1 GB hard drive priced at an astounding $399. That price was astoundingly low, and that was only 10 years ago. Progress marches on, and sometimes progress really is an improvement.

The drives are so small because they use flash memory–a type of readable/writable memory chip that doesn’t lose its contents when it loses power. It’s not as fast as RAM, and it’s a lot more expensive, and its lifespan is much more finite, so you won’t see flash memory replacing your computer’s RAM any time soon. But as a replacement for the floppy disk, it’s ideal. It’s fast, it’s compatible, and unlike writable CDs and DVDs, they require no special software or hardware to write.

The drive plugs into a USB port, which is present on nearly every computer made since about 1997. Use with Windows 98 will almost certainly require the installation of a driver (hopefully your drive comes with either a driver or a web site you can use to download a driver–check compatibility before you buy one for Win98), but with Windows 2000, XP, and Mac OS X, these devices should just plug in and work, for the most part. With one Windows 2000 box, I had to reboot after plugging the drive in the first time.

From then on, it just looks like a hard drive. You can edit files from it, or drag files onto it. If the computer has USB 2.0 ports, its speed rivals that of a hard drive. It’s pokier on the older, more common USB 1.1 ports, but still very tolerable.

The only thing you have to remember is to stop the device before you yank it out of the USB port, to avoid data loss. Windows 2000 and XP provide an icon in the system tray for this.

These are great as a personal backup device. They’re small enough to carry with you anywhere–the small flashlight I keep on my keychain is bigger than most of these drives–and it only take a few minutes to copy, so you can copy those files to computers belonging to friends or relatives for safekeeping.

If your only interest in a laptop is carrying work with you–as opposed to being able to cruise the net in trendy coffee shops while you drink a $5 cup of coffee–a pen drive makes a very affordable alternative to a laptop. Plug one into your work computer, copy your files, and take work home with you. Take it on the road and you can plug it into any available computer to do work. It’s not the same as having your computer with you all the time, but for many people, it’s more than good enough, and the drives make a Palm Pilot look portly, let alone a laptop.

So how do you maximize the usable space on these devices? The ubiquitous Zip and Unzip work well, and you can download small command-line versions from info-zip.org. If you want something more transparent, there’s an old PC Magazine utility from 1997, confusingly named UnFrag, that reduces the size of many Word and Excel files. Saving in older file formats can also reduce the size, and it increases the possibility of being able to work elsewhere. Some computers still only have Office 97.

You may be tempted to reformat the drive as NTFS and turn on compression. Don’t. Some drives respond well to NTFS and others stop working. But beyond that, NTFS’s overhead makes it impractical for drives smaller than a couple of gigs (like most flash drives), and you probably want your drive to be readable in as many computers as possible. So FAT is the best option, being the lowest common denominator.

To maximize the lifespan of these drives, reduce the number of times you write to it. It’s better to copy your files to a local hard drive, edit them there, then copy them back to the flash drive. But in practice, their life expectancy is much longer than that of a Zip or floppy drive or a CD-RW. Most people are going to find the device is obsolete before it fails.

The technologically savvy can even install Linux on one of these drives. As long as a computer is capable of booting off a USB device, then these drives can be used either as a data recovery tool, or as a means to run Linux on any available computer. 512 megabytes is enough to hold a very usable Linux distribution and still leave some space for data.

Laptop or desktop?

All this talk today about cheap notebooks like the Sotec 3120x begs another question: Who should buy one?
Nearly six years ago, I published a column in the Columbia Missourian newspaper. My working title was 101 Reasons NOT to Buy a Laptop but a cooler-headed editor toned it down. I pointed out that you can buy twice the computer for the same amount of money, and laptops are hard to upgrade and they break a lot and you shouldn’t buy one without an extended warranty. (I was shocked to read that I’d said that way back then.)

All of that’s still true today. Except for the twice as much computer for the same amount of money bit. Thank goodness that’s changed.

Now you can buy twice as much computer for half the money.

Back then my job was to set up and fix laptops. I didn’t actually use one very much. I’ve been using one nearly every day for the past year and I’ve found some things to like about laptops now.

Portability. Duh. But this means not only can you take it with you, but you can stash it easily when company’s coming over.

Small size. A desktop computer’s going to take up most of the desk. My current computer desk has more usable space on it than my kitchen counter, which is nice because that gives me some room to work. Or put more computers on it. Guess which I do? But anyway, I can set up a laptop on a small desk and still have space to work.

Quiet. A lot of desktop PCs have three, even four fans in them. They make a lot of noise. Laptops have one fan and it doesn’t always even go all the time. Go back to a desktop and you’ll discover you’ve forgotten how much you like quiet. (Apologies to Charlie for stealing one of his lines.)

Gorgeous display. Another coworker came in today to work on my laptop (more on that in a bit) and to complain about another coworker. He was griping about how his laptop display looked when he hooked it up to an external monitor. I asked why anyone would hook up a laptop to a CRT. I guess it makes you look important.

Flat-panel LCD displays are gorgeous. No flicker, great color saturation, perfect focus, really easy on the eyes. They don’t update fast enough to be good for 120-fps 3D gaming, but for everything else, they’re fabulous. Staring at a CRT for 8 hours wears me out. Staring at an LCD for 8 hours has no effect on me. I’ve got a nice 19-inch CRT–an NEC, and it’s one of the professional line, not the consumer line–and it’s great. But I’ll take my laptop’s 13″ LCD.

You can get a similar effect by connecting an LCD to a desktop, but you’ll get digital converted to analog and back on an inexpensive one, which will affect display quality ever so slightly. A laptop is all digital, from video chip to screen.

The downside. After living with one, I’ve changed my tune a little. It used to be when someone said they were getting a laptop, I’d cringe the same way I would if they told me they were getting a sex change. I don’t do that anymore.

But there are still issues. I’ve broken my laptop twice in the past four months. And I treat mine well. The first was a hard drive. The second was the power connector–a piece of plastic snapped off. You’re looking at a motherboard swap to fix that one, in this age of people not knowing how to solder.

Laptop keyboards and mice take getting used to. Every time my girlfriend comes over and needs to use a computer, she sits down at the laptop and asks me for a “real mouse.”

And I miss my IBM clackety keyboards when I’m using a laptop. (I suspect Charlie would get really annoyed if I used one of those at work though, since he’s in the cube next to me, and the way I type, those keyboards can overpower fan noise. Or phone conversations. Or earthquakes.)

Upgrades remain a problem. I’ve got an IBM Thinkpad 600. Great display, great keyboard, and it’s small and light. But it’s slow. The memory tops out at some weird amount–I don’t think I can put 256 megs in it. CPU upgrades are all but out of the question. I can put a faster hard drive in it, but desktops give a lot more options. Even in my old original IBM AT case I can shoehorn a newer motherboard with an 800 MHz VIA C3 processor, and I can put in a 15K SCSI hard drive if I really want to. And that’s a 17-year-old case. I’ve got better upgrade options with a 17-year-old IBM PC/AT than I do with a four-year-old IBM Thinkpad!

So should anyone buy this new generation of cheap laptops? Well, remember, “cheap” is relative. Even when you can finagle into buying one for $800 through creative use of coupons, that’s still a pretty serious chunk of change.

And because they break as much as ever, I have trouble recommending a laptop as an only computer. If you’ve already got a desktop and plan to keep it and can afford a cheap snazzy laptop, then by all means go for it. You’ll love the freedom to move around. If you can’t afford $800 plus the extended warranty, wait a month or six. They’ll come down. I believe you’ll be able to buy a budget laptop for $599 by this time next year. Possibly even $499.

But if you’re buying your first computer, I think you’re better off with a low-end desktop and a nice flat-panel LCD display. The LCD will outlive the desktop PC, and the desktop PC will give you a lot more upgrade options. And as someone who’s been playing with these things for 20 years, trust me: You’ll want upgrade options.

Rare DOS disk utilities

Mailbag:

RAM disk; Your book; Mobos; Monitors; Net folders

I’ve been doing a bunch of work in DOS the past few days, and I’ve found some useful disk tools. A lot of people use the shareware WinImage or GRDUW to create images of floppy disks. That’s with good reason, seeing as floppies are so unreliable–this way, you’ve got a backup on a hard drive or CD-ROM drive, and it’s so much more convenient when you need a particular disk to just grab a blank, make a fresh copy from an image, and go do your thing. But I found some DOS utilities, some recent and others oldies but goodies, that give you the functionality of these shareware utilities but with the advantage of being free, smaller, faster, and in most cases running on a wider variety of operating systems–all good things. So they don’t have a nice clicky mousey interface… I don’t like using a mouse anyway. Maybe you’re like me, or maybe you like powerful utilities and don’t mind giving up the mouse to be able to use them.

So here goes.

Creating disk images. My favorite is  Diskwarez DF — of course I like this utility, seeing as it bears my initials. DF is a short and sweet utility for creating and writing disk images compatible with Rawrite and the Unix dd utility. Runs under DOS and under Windows 9x and NT in a command window. There are dozens of DOS disk imaging utilities out there, but this one has the advantage of being compatible with a very common cross-platform standard. Check out the Diskwarez site, as it’s got tons of info on disk programming, as well as some other utilities like free disk editors. Despite the name, it’s not a pirate site–Diskwarez software is distributed under a free license somewhat similar to the GPL.

If you prefer self-extracting images, you can use the similarly named DOSDF to create them.

Bigger, faster, better floppies. The other feature of GRDUW is to format high-capacity floppy disks and floppies that give faster access than disks formatted with Windows Explorer or the DOS format utility. Enter FDFORMAT . You can do that and plenty of other cool things with this utility. You can gain more usable space on a 1.44-meg floppy without resorting to weird disk formats just by reserving fewer root directory entries. For example, FDFORMAT A: /D:16 gives you the maximum available space on a 1.44-meg floppy by reserving just 16 root directory entries (if you’re storing large files you don’t need more than 16 anyway, probably).

For extra speed, use Sector Sliding: FDFORMAT A: /X:2 /Y:3 speeds up the disk by 50-100 percent by arranging the tracks in a more optimal order. Supposedly you can gain even more speed by playing around with the gap length, but the author says disks are less reliable when you do this. If you’re more interested in speed than in reliability, add the /G:32 switch to the command listed above.

And by default, the boot sector on disks formatted by FDFORMAT automatically try to boot to the hard drive rather than giving you the dreaded “Non-system disk or disk error” message. Why couldn’t Microsoft think of that?

And of course you can also format high-capacity disks. Use the /F168 option to format a 1.68-MB floppy, and the /F172 option to format a 1.72-meg floppy. These switches can be combined with the others as well. Keep in mind that extra-capacity disks aren’t bootable.

FDFORMAT’s downside is it won’t run from inside Windows NT or Windows 9x. The best thing to do with it is to format a disk with it on a PC booted into DOS (DOS mode from Windows 9x’s boot menu is sufficient), then take that disk and use the aforementioned DF or DOSDF utilities to make an image of that disk, then when you need to format a new high-speed disk or a new disk that won’t give you errors when you leave it in the drive, use the image.

Formatting bad disks. And finally, for those dreaded Track 0 Bad errors that render a disk unusable, there’s FR , which uses workarounds to try to make the disk usable again. Typically I get rid of floppies with bad sectors pretty quickly, but if it’s an emergency, this program might bail you out. I used to get around Track 0 errors by formatting the disk in my Amiga–for some reason the disk always worked after that–but seeing as I usually don’t have my Amiga set up, this is an alternative.

And wouldn’t you know it, as soon as I wrote that I found a better way. SmartFormat also does Track 0 workarounds, uses the date and time to create unique disk serial numbers (instead of Microsoft’s license-plate method), provides a fast format that’s up to 60% faster than Microsoft’s method, and can optionally format 1.72-meg disks. SmartFormat runs within Windows, usually.

Mailbag:

RAM disk; Your book; Mobos; Monitors; Net folders