Diary of a Network Geek

Yeah, I know Win2k.

And, today, I know it better than I’d like. This weekend I spent 21 hours moving a Windows 2000 file server/domain controller/IIS server/proxy to a new hard drive. At least I’ll get paid by the hour.
Okay, so you regular readers (you know who you are) know that I’m mainly a Netware/Linux guy. So, what am I doing working on Win2K? Well, times are tough, so Uncle Jim has to work at two jobs. The second job is a realtively small company that I consult for on a strictly after-hours, part-time basis. Most of the time, I don’t put in more than five or six hours a month, which makes it a nice supplement to my primary income. And, frankly, I took the job because I needed the extra experience on Windows 2000 Server and consulting. So, for months they’ve had a 16 Gig drive that has been on the edge of full and last month they finally got a new 70+ Gig drive in. Great, right? Not so much. See, I tried to use Ghost to get it moved over, but Ghost kept failing. It took me a month and about 20 hours to finally figure out that the drive was just way, way, way too fragmented for Ghost to deal with. That meant one thing: the old backup-and-restore method. Ugh.
Okay, so I’m figuring it’s going to be a four or five hour thing, right? Nope. The tape drive isn’t working for some reason and it takes me at least four hours to get that worked out. Then another hour or two for a full backup. Then, an hour or more to get Windows 2000 installed on the new drive and the backup software installed. Another hour to get the tape merged and a restore launched. So, then I reboot and ….. Blam! Error!!
Can you say “Security Account Manager initialization failed”? Does that seem like a problem? You bet it is! The error told me to restart in “Directory Services Restore Mode” to try and fix the problem. Sounds good. But, when I restart, it asks me for the Admin password before repairing and proceeds to tell me that I’m putting in the wrong password. Okay, so I start all over again. Guess what? Same problem.
So, since I still have the old hard drive in original condition, I hook it back up and go searching for an answer. I finally find that I have to do a special backup of the System State with a Microsoft tool, not my backup software (which claimed it was doing a System State backup), boot the new drive into “Directory Services Restore Mode” and do an Authoritative Restore to get the old Active Directory information onto the new Domain Controller. About this time it’s 10:00pm or so and I have hours of work to do. But, since I am, of course, Mr. Indestructable, who’s motto is “I only have so many hours on this planet and a lot to do. I’ll have plenty of time to sleep when I’m dead”, I push on. Better, I figure, to push and get it done than have to come back and waste a second day in this office.
Hours, and several redundant restore jobs, later, I get almost everything back up and running. At this point it’s around 3:00AM and I’ve already changed my watch. What else did I have to do while I was waiting for the restore jobs to finish? Now, as you might imagine, I’m a little fuzzy, so it takes me about another two hours to get all the Internet settings right again and confirm that all the required services are started and will restart at reboot. Finally, I decide I can head for home.
The ride home was, er, interesting. I don’t remember large sections of it and it’s about a 30-40 mintue ride with no traffic. There was only one drunk on the road at 6:00AM, but there was fog and mysterious “things” in the corner of my vision. In other words, I’m too old for this stuff. I was freaking seeing things from the fatigue!
Still, when I got home, I tested the remote connectivity and sent an e-mail off to the developer or their custom database application so that he can verify that everything is running okay. Then, it was off to bed for a couple of hours until the Sun just wouldn’t let me sleep anymore. And, I’ll be back to bed soon.

So, what did I learn from all this? First, always backup your Active Directory with the stupid Windows 2000, built-in backup tool and keep a copy around. Shoot, throw it on tape, too, while you’re at it! Second, if you must have a Domain Controller, have two. They like company. Third, do NOT believe the backup software vendors when they tell you that their product will backup the System State or Active Directory! They’re LYING to you!!
Fourth, I am way too old to play Mr. Indestructable anymore. If these people weren’t so dependant on me to keep them going, I never would have done a 21 hour stint at an office. The kicker is, now, I’m getting resistance to my charges! Yeah, isn’t that something? I bust my ass to save their’s and now they’re not real excited about paying me. Damn this is a thankless business. Of course, I have, as they say in the “family”, made my bones on Windows 2000 and Active Directory. Hell, I have a friend who used to be on the a Lead on the Microsoft Corporate Support team that didn’t think I could pull it off! He was shocked that I managed it at all, much less in 21 hours.

So, once again, we the unwilling, lead by the unknowing, have done the impossible, for the ungrateful.
Crap. I’m going to bed.

What if I did a report and no one noticed?

Well, two days ago, that’s just what happened. I do the regular check of all the backups for our company. I mean everything. I check all our local servers as well as all 35+ remote servers. Then, of course, I do up a report in Excel and e-mail it to everyone. Seems fairly important, right? Hmm, maybe not. On Tuesday, I was up to my ass in alligators and was all the way home before I realized that I hadn’t sent the report out. I created it and dealt with most, if not all, of the problems on it, but I never sent the e-mail.

But no one noticed!

So, why do I send out the stupid report? Why did I get chewed out a couple of months ago for not doing it “right”, even if I was doing it the way I was “trained” to do it? If a network admin falls in the forest and upper management isn’t there to hear him, does his resume make any sound?
Well, at least it’s a job.

Some tools for thought for security auditors…

Some time ago, I speculated on what a network “rescue” kit should contain. Well, the other day, I was cleaning up around the house and found an old copy of Information Security that had an article about security audits. They included a sidebar of suggested tools for performing an audit.
Some of them were fairly obvious: ping, traceroute(tracert), nslookup, and grep. Then they listed several free tools that were, well, a little bit more “robust”:
First, there was Nmap, which is an OS fingerprinting tool that is well-known to the Linux community.
Next, there was Crack, which is a well-known password cracker, as well as John the Ripper. There were two that I was not familiar with, namely which is apparently a BIND version checker. (That can be good to know, considering how many pesky security problems have been found with old versions of BIND/) And, finaly, ghba.c, which is a tool for extracting machine names and IP addresses for a class B or C subnet. (Those last two are actually links to source code that has to be compiled before it can be run.)

The article went on to talk about several commercial and Open Source scanners that check for security vulnerabilities. I won’t bother to mention the commercial ones, since they have big advertising budgets. But, I will list off the high-powered open source tools. I’m familiar with the first one, Nessus which has gotten very good reviews in several Linux magazines. The second one is a relative of SATAN, which got press right about the same time Linux really started to get going, Security Auditor’s Research Assistant (aka SARA). I’ve just started hearing about the next one, Whisker, which scans for CGI script vulnerabilities. That’s a nice one to know if you do as much CGI stuff as I have lately! And, finally, Hping2, which I’ve never heard of but seems to be a generic port scanner.

In any case, my point is that there are lots of tools out there that don’t cost a thing, but time, to use in your pursuit of a more secure environment. And, if you’re just reading this wonderint what a security audit is, or why it should be done, it looks like it’s time to get out there and start reading! (I’d start with a free subscription to Information Security magazine.) Happy hunting!

Guess what I’ve been doing today?

Well, at least for the past couple of hours. Yep, I’ve been trying to beat MyDoom into submission at my second, “part-time” consulting gig. I don’t know where they got it, how many machines have it, yet, or even how long they’ve had it, but they got it. I even did the updates via the Micro$loth Update service, but that just wasn’t good enough. So, while I type this entry, the MyDoom removal tool is downloading, as is Spybot Search and Destroy. Why both? Well, at least one of the people at this site has KaZa loaded on their machine. And, while I’m sure it’s a great program for trading perfectly legal files, it’s about the most pernicious virus infection vector I’ve ever seen. If I were making the policy here, I’d just uninstall it and tell them not to ever even consider asking about the possibility of installing it again. But, no, I’m a “consultant” so I have to be more diplomatic than that. Which, I think, I was. (My wife was there, so I think she’ll back me up on that.) Is it wrong to pray for something so small and trival like someone forbidding another to run a piece of software? Well, if it is, it’s too late for me, because I’ve already done it.
Oh, well, at least I got paid for the four hours of work by the hour, at my “on-the-side” rates…

I think…

Way back when I started this blog, before I even started using MovableType, I was out of work and looking for a job. One of my first interviews, which I don’t think I even mentioned on this site, was with Kaiser Aluminum. I remember crossing a picket-line to get to the interview, which was bad enough, and then finding nothing but envrionmental protest stories when I did a search on the web for more information. Well, they decided that I didn’t have enough Windows experience at the time and decided to take a pass on me. Honestly, at the time, I was a little relieved that I wouldn’t have to decide if was okay working with a company that had as much union trouble as they seemed to have at the time. It turns out, they may have been in worse shape than even I suspected!
I heard on the news yesterday that Kaiser Aluminum had filed Chapter 11 and was possibly going to stick it to their employees by reducing retirement benefits. Yikes! Talk about trouble waiting to happen! Anyway, it was a nice reminder that sometimes things really do work out for the best.

I know why the PERL developer cries…

Sometime ago, I wrote an entry about PERL on Netware. At the time, I wondered why no one used PERL for Novell System administration. Now, I know. Novell has virtually no documentation on how PERL interacts with Netware. There are a few, simple example scripts and a couple of suggestions for things to do with it, but that’s about it. How frustrating! Here I am, having actually gotten a little bit of PERL savy and to what end? My favorite server OS supports it, technically, but they offer virtually no information about how to actually make use of it! Argh!

Ah, well, now that they’re getting into the Linux arena maybe we’ll start to see more support for Open Source and “hackerish” tools, like PERL. I sure hope so…

Wow, I’m becoming a Unix admin!

Now, this might not be a big deal for old-time Unix geeks, or anyone who prefers to work with Micro$oft products, but I think it’s cool. Last week, I was wrestling with a little problem that I solved with a judicious application of XWindows.
We run Oracle 9ias at my office and to get it to print graphics we have to do a kludge work-around. Not that we’re special, everyone who runs Oracle 9ias has to do it. What we’ve been doing is using a pair of Windows workstations running a Windows-base X terminal. Well, over the weekend those workstaions locked up and the DBA couldn’t get them restarted without physically rebooting them. So, Wednesday he came to me and told me he wanted a Linux solution to the problem.
After a bit of Googling, I found what I hoped would be the answer, but I didn’t have time until Thursday to try it out. Here’s what I did, in a nutshell:
1. On my Red Hat 9 workstation, I opened a terminal and ran “xhost +”
2. On the Solaris 8 test server running Oracle 9ias, I telnetted in and ran “display=my RedHat 9 workstation IP address:1.0″ and then “export display”
3. Then, still on the 9ias server, I ran”Xterm” and it popped up on my XWindows session on my RedHat workstation!

Woot! So, I know it might not be a big deal to an old Unix hand, but to this old Novell geek, it was pretty cool. Oh, and it works, too. Now, I just have to figure out how to script all that so it runs automagically……

WOW! Talk about a blast from the past!

So, I’m just sittin’ there, minding my own business, trying to look casual while reading e-mail and the boss walks in and starts firing questions about Netware licensing at me. Naturally, I’m suspicious from the start, but I listen and answer questions as best I can.
Not too long ago, we bought a company on the East Coast that had, well, an “interesting” mix of technology in their IT department. It seems a couple of these sites are have problems with the number of licensed connections available. The office has about 30 people, but they only have a 10 user license. Talk about 10lbs. of crap and a 5lbs. bag. But, wait, it gets better!! They’re running, yep, you guessed it, Netware 3.2. Oh, God, what did I do in a past life to deserve this?
Well, wailing and gnashing of teeth aside, I’m gonna’ have to deal with this. Somehow. Without spending any money. Yeah, right.
So, I dig into Uncle Jim’s Magical Bag O’ Tricks™ and pull out not one, but two copies of unserialized server.exe that I had from way, way back by way of a guy who was a developer at a Novell channel partner or some such. (Don’t worry, he’s since turned rogue and been seduced by the Evil Micro$oft Empire, so I’m not getting him in trouble.) But, you know what? The boss says to me, “That just doesn’t sound ‘right’.”
What’s this? A boss who actually cares about doing things the Right Way™ ? Whoda’ thunk it!
So, the work may not always be challenging and the pay can always be better, but at least I have a pretty damn good boss these days. That’s something. Hell, somedays, it’s the only thing.

but, I lost the energy.

So, last week Friday, I bust my butt to get a damaged server fixed and out into the overnight deliveries so that my customer, the end user, can get working again. Good for me, right? No. They get their server before 3:00pm on Monday, as promised, then plug it in, turn it on, and walk away. Just hit the road and head home without making sure that it came up at all. Now, I’m sure that they had their reasons, but I worked an extra hour on a Friday, doing a less-than-perfect rushed job, just so they could get their files. And they didn’t care. That’ll teach me.

Then, I’m chasing backup issues all week this week. Trying to get people to change tapes should not be this hard. I mean, it’s once a week on Monday and that’s it. Just one, little 10 minute job to make sure they have their data in case of a crash. But, it’s like pulling teeth to get some of them to remember to do it.
And, while I’m doing that, I discover that one remote site has basically moved out of the building they were in a week ago and there’s no one there to change the tape. But no one told me!
Shoot, a majority of the people who are responsible for it don’t even read the report I send out every day with backup status. So, why do I bother?

Oh, yeah, because I’m a professional with 10+ years of experience and that’s what I do. Because working in IT is a damn thankless job that no one appreciates until they’re having problems. Because I’m damn good at what I do, even when the odds are against me. And, because that’s why they pay me. That’s my job.

I got my business cards yesterday.

Okay, so I’ve gotten the Human Resources speech and filled out forms. I actually collected my first paycheck. But, now, I am officially a Loomis Fargo and Company employee.

Cool.