A lot of people seem to be looking for help cleaning up hacked or infected web sites lately, so here’s a trick I used in the past to find PHP malware in Linux and clean up the infected files.
So the other day I got blindsided with a question at work: What are we doing about Winshock. Winshock, I asked? I had to go look it up, and I found that’s what they dubbed what I’ve been calling MS14-066, the vulnerability in Schannel, which is Microsoft’s implementation of SSL/TLS for Windows.
Based on that, I’d argue it has more in common with Heartbleed than Shellshock, but I guess “Winshock” is catchier than “Winbleed.”
Then the lead of another team asked me to brief his team on Winshock. I actually managed to anticipate all but three of the questions they asked, too, which was better than I expected. Some of what I shared with them is probably worth sharing further.
If you have a Windows domain, there’s a fairly good chance you have Backup Exec servers, because you probably want to take backups. Because you need them. (As a security guy, I no longer care how you get backups; just that you’re getting them somehow.) Backup Exec is a popular solution for that. But there’s a problem.
A security problem, that is. The quality of Backup Exec as a product hasn’t been my problem since 2005. The problem I have with it now is that Backup Exec stores its passwords in a database. The passwords are encrypted, but it’s possible to decrypt the backup copy, if you’re determined enough.
A coworker tipped me off the other day to how it’s possible to use a certain major-brand antivirus to infect a computer. “I didn’t have admin rights,” I overheard him explaining, “So I got them with [redacted] antivirus.”
My head spun around violently. “You did what?”
“Google ‘confused deputy persistence,'” he deadbeated. “It’s the first result.” Then he went back to explaining the problem at hand. Read more
This week, ARM said what several people seem to have figured out: The key to mass adoption for smartphones and tablets is the $100 price point.
It may happen this year. It’s not hard to find a decently fast $80 Android tablet, but you’ll have to put up with a sub-optimal screen to get it–800×480.
I take issue with this Register story, which says Veritas has a better name in the storage arena than Microsoft.
Enron has a better name in the storage arena than Veritas. Ditto BALCO and FEMA and Michael Jackson and Martha Stewart.
So Microsoft wants to get into the backup business? Good.I gave three of the best years of my life to the shrink-wrapped stool sample that is Backup Exec. I believed, wrongly, that the Constitution protects sysadmins like me from that piece of software in the clause that mentions cruel and unusual punishment.
After that last job put me out with Thursday night’s garbage, one question I always asked on job interviews was what they used for tape backups. Had anyone said Backup Exec, I would have walked out of the room immediately.
Nobody did. That was good. There are still some smart people in the world. My confidence in humanity was somewhat restored.
Microsoft’s offering will no doubt have problems, but when batch files and Zip drives are more reliable than your competition, who cares? Backup software is one area that desperately needs some competition. Microsoft entering with its usual less-than-mediocre offering will force everyone else with their less-than-mediocre offerings to either improve or die, because Microsoft’s offering will be cheaper, and there will be people who will assume that Microsoft’s offering will work better with Windows because nobody knows Windows better than Microsoft. (In this case, that assumption might actually be true.)
What’s wrong with Backup Exec? Ask your friendly neighborhood Veritas sales rep what they’ve done about these issues:
If a Backup Exec job backing up to disk contains both disk and system state data and it’s the second job to run on a given night, it will fail just as certainly as the sun coming up the next morning. Unless they finally managed to fix that bug, but I doubt it. I sure reported it enough times.
Remote backups happening over second-tier switches (D-Link, Linksys, Netgear, and other brands you find in consumer electronics stores) usually fail. Not every time. But more than half the time.
Those are just the problems I remember clearly. There were others. I remember the Oracle agent liked to die a horrible death for weeks at a time. I’d do everything Veritas support told me to do and it’d make no difference. Eventually it’d right itself and inexplicably run fine for a few months.
Maybe competition will fix what support contracts wouldn’t. And if it doesn’t, maybe Backup Exec will die.
And if Backup Exec must die, I want to be part of that execution squad. Remember that scene in Office Space with the laser printer and the baseball bat?
I never thought I’d say this, but now I’m saying it.
This is not the worst week of my life.
I’ve been evasive about my job, purposely, almost since the beginning. I realized–the day after I turned 24, I think–the dangers of blogging about my career and my employer, so I stopped doing that. I even went back and deleted at least one entry that I knew I probably should have never written.
I can’t say much, but I can safely say I lost my job on Thursday.
I went through all the emotions. Panic. Pure lividity. Betrayal. Lots of betrayal.
I saw it coming. You always see the end of bad relationships coming, don’t you? Actually I saw it coming in January. I made an effort to try to seek out opportunities, but it was hard to do that while working the kind of overtime I was working. I think my peak week was 72 hours. The extra money was really nice–it went straight into the Honda and it’s probably already saved me a couple hundred bucks–but my fiancee could see the years coming off my life. Was it worth it?
I looked in the mirror and saw my Dad. But not Dad at 30. When Dad was 30, he looked younger than I look. I was working the kinds of hours Dad worked–both in quantity and in the irregularity–and I was getting old before my time.
Dad died at 51. Do I want to only have 21 years left? No job is worth that. Certainly not this one.
I applied for a job in Milwaukee. They were great people, but they turned me down. That was OK. I figured I was a long shot anyway. The only better job in the world would have been manager of the Kansas City Royals (they have an opening–maybe I should apply) but in my heart of hearts I didn’t think I was really ready for it. It was journalism, and I know how to write and edit, but I really need another five years’ knowledge under my belt.
Late at night when I was feeling sorry for myself, I’d hit the job sites and see what was out there. Nothing grabbed me.
Something happened last week. It was the last-straw moment, like when Lumbergh took away Milton’s stapler. I could have handled the situation better, but there were about a dozen people who needed to handle that situation better.
But bad weeks have a way of blowing over and usually after a series of bad weeks there’s a good week. You know, those weeks when you can disappear, work on whatever it is you’re supposed to be working on instead of putting out fires all day, go home at 5, and your phone doesn’t ring all night. Going into the weekend, I tried to tell myself this would be one of those. I was going to Kansas City for my wedding shower. Kansas City is always good. I might even be able to sit and watch KCS trains (real ones) from my sister’s back porch to unwind for a while.
Monday came, and it was just meetings. Meetings are bad enough; meetings where you have the sense that the person running it just doesn’t have a good grasp on what’s going on are worse. We had problems, and that meeting wasn’t going to solve any of them. This wasn’t looking like a good week.
Tuesday started out better. I heard a song on the radio as I drove up I-55. It’s been almost two years since I heard a song on the radio and it grabbed me, but this one did. I scrawled down a couple of lines after I pulled off I-55 and punched them in to Google. Google is the best thing that ever happened to music. The song? “What About Everything?” by Carbon Leaf.
Every time I saw something I didn’t like that day, some line from that song popped back into my head.
Lines like these:
In search of some rest, in search of a break
From a life of tests where something’s always at stake
Where something’s always so far
What about my broken car?
What about my life so far?
What about my dream?
What about everything?
What about aeroplanes?
And what about ships that drank the sea?
What about the moon and stars?
What about soldier battle scars
And all the anger that they eat?
But the line that came back more often than any other was the final one:
Well, I am not, I am not, I am not in need.
It made me feel better.
That evening, I saw a job posting. I made a phone call. It sounded promising. The job description was challenging, but it’s a cakewalk compared to the last two years. “Challenging” is good. Doing two or three people’s jobs is bad.
On Wednesday, all of the higher-ups were acting funny. I should have picked up on that. I know the look. I saw it in 1993 when the managers at the fast-food joint I was working told us the restaurant was closing and the company was pulling out of the St. Louis market. I saw it in 1994 when Best Buy decided it needed to terminate all of the holiday help they’d brought in, even the part-timers who were just back from college and would be leaving in two days anyway.
It never occurred to me that look is universal. I guess it’s a good thing that I’m 30 and have only seen that look three times. The first time didn’t hit home–I was leaving for college anyway, and my last day was a good three weeks before they’d be closing. They let me keep my work shirt. I might still have it somewhere. I came back for the last day. One of my coworkers summed it up pretty well. “Well, Dave, I’ll probably never see you again. Good luck!”
But I did see him again. Five years later, at Mizzou.
The second time hurt. I don’t know anymore if I lost a day’s work or a week’s work, but I thought I really needed the money. I brooded for a week and then went back to college. Get a good degree from a good school, I told myself, so that would never happen to me again. It motivated me.
Ah, to be 19 and naive again.
Thursday toward the end of the day, my boss’ boss was waiting for me. She had that look. She motioned me into a meeting room. Inside, the man who recruited me in the first place was waiting for me. I went to high school with his brother. When he wanted to start doing some writing on the side, he came to me for advice. I counted him as a friend. But he had that look on his face too, and this was strictly business. Cutbacks… Job eliminated… I’m sorry….
Maybe you are, maybe you aren’t. That’s between you and God. I didn’t wait long enough to hear their cheerleading rah-rah schpiel about how they’d help me find another job. I didn’t need lines from an HR policy manual.
That song is in heavy rotation right now on my favorite radio station, but it wasn’t playing as I drove home. But I heard it again this morning.
In search of some rest, in search of a break
From a life of tests where something’s always at stake
Well, I finally got that. I was wanting to take some time off this upcoming week so I could tend to things I’ve been ignoring from working too much. I didn’t expect to get it.
What about that midnight phone call…
The one that wakes you from your peace?
Well, that won’t be a weekly occurrence anymore. My ex-boss called me this morning to answer some questions. He mentioned that Backup Exec blew up again Thursday night. “It sympathized with you,” he said. I told him to get used to it.
Then I think about being done, with no resume
With no one left to blame
Well, they can blame me all they want but it won’t make any difference anymore. I may be done, but I’ve got three different versions of my resume.
What about fortune and fame?
I don’t need fame. Fortune? Their money doesn’t spend any differently from anyone else’s. At the rate I was going, I probably could have retired at 55. Maybe this will slow me down by a year, maybe it won’t. But my likelihood of actually living to 55 probably just went way up. The ideal thing, of course, is to find something I like so I won’t want to retire at 56 anyway.
What about your love to obtain?
What about the ring?
Oh yeah. That. The important thing. I’ve still got my bride. She trusts me to see to it that her needs are met. Right now I can’t give her the kind of security I want to give her, but she knows I’ll do the very best I can with what we have to work with. She believes in me during those times when I don’t really believe in myself.
Now at least I get to spend some time with her. She got a good taste of what was to come the day we met. I was supposed to meet her at 5:30. But a tape drive broke and we didn’t have a spare, and HP wasn’t going to be there until 7 with a replacement. So I called her and let her know I was going to be late while I showed him what needed to be done.
Bad first impression. Very, very bad. But she got over it. She got over me going out to the parking lot three times during the date to answer the phone and answer another question.
Van, if you’re reading this, thank you. This is the girl.
I told her that bad jobs are just like bad relationships. You tell yourself over and over again that there are better jobs out there but there are a million and one reasons why you’re not qualified for one. So you put up with your love-hate relationship with your job, maybe you pretend someday it’ll get better and maybe you don’t, but you stay there because you don’t think you deserve anything else.
“And then it breaks up with you anyway,” she finished for me.
I won’t find a better girl. I can find a better job.
Well, I am not, I am not, I am not in need.
Zero times anything is still zero. But I’m not sitting here with zero.
If you run your web servers on Windows under IIS, you’d better install the Hisecweb security template unless you want to find yourself hosting a warez site.
But Hisecweb breaks Backup Exec. So what do you do when upgrading to Apache and Linux isn’t a solution?The problem is that Hisecweb makes the system state (shadow copy components in Windows 2003) and SQL server not show up in the selection list. Not only does it not show up in the selection list, Backup Exec cannot find the resources. So backups fail, and if you have to restore from them, you won’t have the registry or a number of system files, which vastly reduces the value of your backup.
The solution is to tell Backup Exec not to use null sessions on those components, which seem to be one of the many things disabled by Hisecweb. On the server being backed up, go into Services and disable your Backup Exec Remote Agent. Now, fire up Regedit. Navigate to HKLM\Software\Veritas\Backup Exec\Engine\NTFS and locate the key called Restrict Anonymous Support. Set this value to 1. Close the registry editor and restart the Backup Exec Remote Agent service.
SQL Server and the system state or shadow copy components should now show up in the selection list for the server you just changed.
This registry hack can also fix visibility problems when the two machines are on different sides of a firewall.
I saw the following in one of my Backup Exec failure logs (directory names changed slightly to protect the client’s name, and me):
Directory F:\ITWEB\Flash Stuff\Welcome Page Animations was not found, or could not be accessed.
None of the files or subdirectories contained within will be backed up.
Hmm. Flash animations.I’m torn. My duty to the client who is paying me, of course, is to fix the problem so the file is backed up.
But they’re blinky, annoying Flash animations. Flash, of course, is the third worst thing to ever happen to the Internet, behind popups and spam. OK, it’s the fourth worst thing. I’ll put it behind spam. But I’ll even put it ahead of Microsoft Internet Exploiter.
So an opportunity to snuff out some blinky Flash animations that have been foisted on the world is a great temptation.
Or am I the only one who feels this way about Flash?
Incidentally, I turn off animated GIFs too–I find a Web without animated GIFs and Flash is a much more pleasant place. I don’t know if that makes me boring and extremist or what.
Veritas is trying mightily to unseat Microsoft as my least-favorite software company. I do believe Backup Exec to be the worst piece of software of any kind on the market. In fact, babysitting Backup Exec is the reason I haven’t been around much.
I’m looking to version 10 for some relief (and the much-needed 1.0 quality that Microsoft usually delivers around version 3–when Veritas will deliver it probably is an interesting Calculus problem).The downside to version 10: I’m told there’s no more Windows NT 4.0 support. Can’t back ’em up. I haven’t actually tried installing the remote agent on an NT4 box to see if it’s unsupported as in we-won’t-help-when-it-breaks or unsupported as in no-can-do. Smart businesses hocked their NT4 servers a couple of years ago. I won’t say anything else, except that not every business is smart.
More downside: If a tape fills up and you can’t change it because the server is offsite and/or behind locked doors that require approval from 14 middle managers and a note from your mother to get to, under some circumstances Backup Exec 10 will hang indefinitely while cancelling the job. Version 9 had the same problem. Bouncing the services will usually relieve the hang, but sometimes you have to reboot.
It’s tempting to put Backup Exec and your tape drive on your biggest file server to get faster backups. But trust me, if you put it on a server that’s dedicated to backups–its day job can be as a domain controller or some other task that’s shared by multiple, redundant mahcines–you’ll thank yourself. It’s very nice to be able to reboot your Backup Exec server without giving your seven bosses something else besides the cover sheet on your TPS reports to grumble about.
If you must put Backup Exec on your file server, set up DFS and mirror the file shares to another server. It doesn’t have to be anything fancy–just something that can prop things up while the server’s rebooting. And run Windows 2003, because it boots fast.
The upside: I can make Backup Exec 9.1 die every time by creating a direct-to-tape job and running it concurrently with a disk-to-disk-to-tape job. The tape portion of the second job will bomb every time. Veritas technical support tells me that bug was fixed in 9.1SP1. It wasn’t. But it’s fixed in 10.
There are some other features in 10, like synthetic backups, that promise to speed backups along. That would be very nice. It would also be nice if it would be reliable.
I’m not going to put it in production yet–when I first deployed 9, it fixed a lot of problems but it made a whole bunch of new ones–but maybe, just maybe, Backup Exec 10 will do what it’s supposed to do well enough that I can work something close to regular hours again.
Otherwise I’ll look forward to Backup Exec 11 and hope that it features more changes than just a new Symantec black-and-gold color scheme and wizards featuring Peter Norton. We’ll see.