Diary of a Network Geek

What do you run on your monitor server?

Do you think you’re too small to run a monitoring server? Well, I have two local servers, a remote web server and a remote e-mail server that I’m in charge of worrying about and I run a monitoring server. It’s not much of a server, really, just an old workstation to which I added a bunch of spare memory and a large, clean hard drive. Naturally, I run Linux on my monitoring server, which, ironically enough, I named Monitor. Specifically, Monitor runs Red Hat Fedora.

Monitor runs Nagios, which I’ve mentioned before. With Nagios, I monitor both my main file server and my accounting SQL server. I also watch the off-site web server and the SMTP and POP3 e-mail services on the managed e-mail server we have through our ISP, just to make sure they’re up and running. (It’s a long story on why we have that, but, rather than run my own, to reduce hassle, headache and potential disaster, I let someone else worry about it.) Nagios tells me the status of drive space, the memory usage, the CPU usage and uptime on both servers. On the accounting SQL server, it verifies that the SQL service is available and that users can log into it. On the file server, it tells me the status of the Backupexec modules. Unfortunately, I haven’t figured out a way to get Nagios to tell me more than the running status of Backupexec, but, in my spare time, I still try to find a way to have it report the status of the last backup or restore job run. No joy yet, but I keep trying.

I also have a browser window open to the SolarWinds installation at our ISP. They monitor inbound and outbound traffic over the Internet connection we have. Usually, I keep a window open on the standard “interface details” reports which update regularly. Most of the time, I also open a window to the weekly history report on the min/max/average packets in and out. I have to update that manually, but it lets me quickly compare today’s traffic to network traffic for the past week. It’s nice to see those trends!
Lately, I’ve been keeping a browser window open to the national weather forecast, by hour, for our local area. In hurricane country, keeping track of the weather can be vitally important! But, if you live in snow country, the same thing would probably be true, too. I don’t recall heavy snow causing an outage during my time up North, but it’s not out of the realm of possibility.

Finally, I almost always have Wireshark running a packet capture, too. If I see a sudden spike in traffic, having a packet capture already running could make a big difference. I have that capture set to save files locally, too, just in case. I’ve been setting the capture to rotate nine files and to keep the files at seven megabytes each. That should give me a pretty good spread of captured network data if I ever need to go back and diagnose a traffic problem. And, since the machine is actually kind of stinky hardware and crashes on occasion, when I restart the packet capture, I rename the base file using the current date. That way, I can tell at a glance when the capture was started.

One day, I’d like to move this all to another machine that’s more stable, faster and has more drive space, but, until then, this works. It’s only on the private network, so I can’t look at it directly from the Internet, but, it still does enough for me. One of these days, I’ll look into some of Nagios’ data presentation modules and teach this old dog a few new tricks, like automated uptime reports and that kind of thing.

Hopefully, that hasn’t bored too many of my non-geek readers. And, I hope it’s given my geek readers something to think about. So, tell me in the comments, if you have a monitoring server/station/whatever, what does it run? If you don’t have one, why not?

So, you want to build your own Linux server for home?

Okay, you’ve seen me write about it and many of you think it’s too geeky to do on your own, but I swear to you, it’s not. In fact, here is a list of easy howto sites:

Who needs Windows Home Server with Linux around?
Linux Home Server HowTo at LinuxQuestions.org
Build It: Linux Home Server
The Australian Linux HowTo

And maybe, just maybe, when I finally get around to doing this on my home network, I’ll write up some documentation on how to do it. Or, at least, how I did it.

Trying to get back to basics.

So, as many of my long-time readers know, I used to write. Well, I used to write fiction. Oh, okay, I used to write a lot more fiction. But, I get distracted easily. Apparently, so do a lot of writers. It’s very easy for me to get obsessed with formats and customizing displays and so on. More so when I’m having trouble with plots. Or endings. Or, most often, beginnings.

So, when I saw an article on 43 Folders about Scrivener and other simple writing tools for OS X. Sadly, everything he talked about was for Mac only, but he linked to a Slate article on the same subject titled In search of the distraction-free computer desktop, which had a few more options.
Now, I’ve read more books and articles about writing and the writing trade than I care to admit, so I’ve seen a lot of information about different writer’s writing environments and choices. Steven Brust, who writes top-notch fantasy, uses Emacs and macros to export that to Word format to send to publishers. I tend to use OpenOffice Writer, because I can use it on any platform I might find myself and my backup laptop is running Linux. But, the articles talked about several options that I haven’t looked into yet.

One that seemed very popular is called Scrivener, but, sadly, it’s only available for OS X. Another was called WriteRoom and it has a “clone” for Windows called DarkRoom. I liked the look of DarkRoom because it was, well, simple and clean and I’ve just about convinced myself that’s what I need. A simple, clean interface with minimal distraction so that I can get to the business of writing. Finally, there was a Windows-only program called RoughDraft that one of the commenters suggested. It, too, has a clean, simple interface, though it looks more like a old-fashioned Windows file-manager than anything else. And, neither of those options truly addresses the fact that I really want to be able to write on my Linux laptop in a pinch with the same tools and configuration.
So, in the end, I’ll probably just stick with OpenOffice. Perhaps I can find a way to customize and simplify the interface on Writer to my liking. Surely, someone, somewhere, has done this and has a convenient HowTo. If not, maybe I’ll do that myself and write it up.

Then, of course, I’ll have to find another excuse not to write…

Advice from your Uncle Jim:
"My obligation is to do the right thing. The rest is in God's hands."
   --Martin Luther King

I don’t think I’ve mentioned this before.

Some time ago, I was having problems with traffic on my network. Something, somewhere was apparently causing some issues with bandwidth on our Internet connection. Or, at least, that’s what our ISP kept telling us. It was, I think, the excuse they were using to avoid dealing with an e-mail problem.
Regardless, I had to find a tool to monitor our network traffic. I ended up using Wireshark for that, but along the way, I discovered a number of OpenSource monitoring tools for various purposes. The one that impressed me the most was Nagios.

Nagios is, according to the opening paragraph on their website, “an Open Source host, service and network monitoring program.” While I never did configure anything to monitor the network, per se, I did configure this to watch both local servers and third-party web and mail servers.
First of all, it’s important to know that Nagios runs on Linux. So, to install the software, you first have to have an available Linux server on which to install it. I’m using an old workstation that I installed the latest version of Fedora, the free version of RedHat. Getting the initial install done wasn’t very hard at all. In fact, there were RPMs available, so all I had to do was use RedHat’s package manager to get the base install loaded on the machine.

After the initial software load, I mainly followed the Quick Install instructions that they link to on the first page. Then, since I was mainly monitoring Windows servers and workstations, I found the cleverly titled help page, “Monitoring Windows Machines“, and followed that. This page ran me through the basics of installing the NSClient++ on a Windows machine and configuring Nagios to connect to and monitor that client. One thing that I had to find out the hard way was that the entries for the monitored systems have to be duplicated for each host. In other words, there is no way to just list all the Windows systems you want to monitor. You have to created entries describing each host individually. That’s not a big deal, honestly, since you can open the configuration files in a text editor and just copy, paste and edit the required entries.
I did have a few false starts here, until I figured out the correct syntax and the fact that every host has to be part of a previously defined group. But, other than that little glitch, configuration was fairly simple.

It took a little more digging, but I later found instructions for passively monitoring services running on servers without a client. I now use my private installation of Nagios to monitor our company webserver, both POP3 and SMTP on our hosted e-mail server, as well as my two Windows 2003 servers. I can even check on the Microsoft SQL database, thanks to information I got from this post on the OSdir mailarchive. And, did I mention that all this software was free? Yeah, the documentation wasn’t the best and it took me a little while to figure out the install and config, but it was far easier than the other monitoring software I played with and I can let anyone who has the username and password check these stats from their own workstation via a web browser. How cool is that? Oh, and did I mention that this can be used to monitor Linux/Unix systems, Windows systems and even Netware systems? Nagios pretty well covers it all!
(Oh, and as a side note, if you’re messing around with the configuration and want to reset the statistics, just stop the service and delete /usr/local/nagios/var/status.dat, then restart the service. All your counts will zero and all the checks will start fresh.)
In short, if you’re looking for a low-cost but versatile monitoring system and aren’t afraid to read the documentation, I highly recommend investigating Nagios.

So, I figured it was about time for another hardcore geek post.

I’ve been using the Linux-based imaging solution that I detailed here and here for quite a few weeks now. Mostly, it’s been going very well.
Okay, once I figured out it was better to start with a small drive partition as my base system, from which I created the standard image, the process got easier. The thing is, it’s always easier to start with a small partition and then use tools built into the Knoppix live cd to grow the partition larger than to try and shoe-horn a big disk partition onto a small disk. And, by “easier”, I mean it’s the only way to do it. I spent quite a bit of time trying to make it work the other way, but I never did. On the other hand, starting with an image based on a 30 gigabyte or less partition then expanding it to fit a 150 gigabyte hard drive has worked just fine.
Incidentally, I used a bootable gparted cd to make that change.

Also, I had a small problem with a particular Intel motherboard chipset. Specifically, it was the Intel 965 chipset, and the problem may effect other motherboards. In short, the problem was that Knoppix didn’t see the SATA drive to mount it. If Knoppix can’t see the drive, it can’t image it or take an image from it. Luckily, there is a work around. If you’re using my method to image WindowsXP machines running the Intel 965 chipset, ensure the BIOS is set to AHCI. To do that, get into your system’s BIOS and go to Advanced > Drive configuration > Configure SATA as AHCI. Then, when booting into Knoppix, hit F2 and use the following command-line to boot:
knoppix 2 all-generic-ide pci=nommconf
This will start Knoppix in text-only mode, so you can run everything from there instead of opening up a terminal session.
After you put the image on the fresh machine, you need to ensure the BIOS is NO LONGER set to AHCI. To do that, get back into your system’s BIOS and go to Advanced > Drive configuration > Configure SATA as IDE. After reconfiguring the BIOS, you can boot into the new Windows XP clone and proceed as I’ve already described in the other posts.

Oh, one last thing on this…
I kind of cheated on reimaging machines in text mode with that Intel 965 chipset.  Because the tools I used to resize NTFS disk partitions were all GUI based and XWindows was having a problem running on those Intel 965 boards, I installed one machine from scratch and just grabbed the larger partition table and master boot record.  Then, when I made the new machines, I just used the larger partition and MBR images to get everything out of the disk.  I still used the smaller data disk partition images, but I used NTFSResize to expand the NTFS partition to fill the disk.  Worked like a charm.

Anyway, I apologize to my non-geek readers, but, hey, I am a professional network geek and I love this stuff.  I think I strutted around for a full five minutes after figuring out that set of little tricks the same afternoon.  Of course, it wasn’t long before some other stupid thing brought me right back to reality, but that couple of minutes where I was the king of the world, network geek genius extraordinaire, made it all worthwhile.

I got a kind of Christmas bonus Friday.

So, Friday, after long resisting it, I finally cleaned up my office.
I had all kinds of junk there, most of which I threw out. But, there was this older iMac. It was in decent shape, outside of a temperamental wireless card. So, rather than get rid of it, I asked the boss if I could have it. Mac lover that he is, he was thrilled to give it to someone who’d appreciate it. And, I think maybe he thought he’d converted me to the Apple camp. He hadn’t, but now I have two versions of desktop Windows, Linux and OS X in my house. Not to mention Novell and Linux server systems. So, now, when someone asks me to convert files for them, format shouldn’t be an issue.

Now, that is what I call a Christmas bonus!

This time, we’ll restore the image we made last time

mkdir /tmp/server
mount -t smbfs -o username=my_user_name   //server_name/share /tmp/server

Now, this assumes that you don’t have anything installed on the new pc yet and are working with just a blank, unformatted hard drive. First, I restored the partition table and the master boot record:
# sfdisk /dev/sda < /tmp/server/images/cad1r-sfdisk-sda.dump
# dd if=/tmp/server/images/cad1-sda.mbr of=/dev/sda

And then the partitions. Since I had several files produced by split for my primary partition, I needed to take them all, in the right order of course. Now, keep in mind that when we used split in the last post, it added “aa”, “ab”, “ac”, etc. to the end of the file name.

# ls -l /tmp/server

will help you check which files you need for the next step.

Now, again, this all assumes that you had a single partition on the original machine and are restoring a single partition. First, change to the directory where the images are:

cd /tmp/server

Now, we restore the image files with the command below. You’ll have to “cat” all the files in the image directory on the server. Notice the pattern of the file names? That’s what split did for us. Now, rather than do the work manually to restore them, I used a little bit of shell code and a variable reference to save you the need to type the names of all the files. Also, the little “-” at the end of the line does matter, so don’t forget it. Oh, and this command should all be on one line.

cat cad1-sda1.img.gz_a[a-z] | gunzip -c | ntfsclone --restore-image --overwrite /dev/sda1 -

Reboot into your new Windows XP clone.
It’s just like magic, isn’t it?

Hey, I get paid to be a geek, right?

So, I’ve been having some issues with my network and several Windows XP machines. In a nutshell, these machines seem to lose connectivity after approximately nine hours and fifteen minutes from the last restart. In other words, when my crazy-dedicated engineers work past their ninth hour, their machine slows to a crawl and eventually locks tighter than a Catholic school-girl’s knees. In any case, after weeks of troubleshooting this issue, I’ve come up empty. The best that I’ve got for these guys is either a) Don’t work such long hours or b) Reboot the machine at lunch.
In a further attempt to fully understand what is happening and at what level, I’ve gotten one of these machines and I’m going to install Windows 2000 on it. If we have the same issue, I know it’s hardware. If I don’t, I’ll be certain, within a reasonable percentage of sureity, that the issue is some arcane aspect of Windows XP. Either way, I should be closer to a real answer.

But, before I wipe my current experimental machine, I decided I wanted to back it up. Naturally, I turned to my old friend, Linux. A quick Google turned up a blog entry titled “Cloning XP with Linux and ntfsclone“. So, with a few modifications for my own environment, I followed the instructions there. Incidentally, I used the latest version of Knoppix as a boot CD.

First, open up a terminal/shell session and create a mount point with the following command:
# mkdir /tmp/server

Then, because my DHCP server didn’t give the Knoppix virtual machine the right DNS information, add your server to the /etc/hosts file.
Next, mount the network share that you want to dump the images on.
# mount -t smb -o username=administrator //server1/share /tmp/server

Check how your live CD sees the partitions you want to save with the following command:

# cat /proc/partitions
major minor  #blocks  name

8       0   78150744  sda
8       1   76211608  sda1
240     0    1939136  cloop0

I want to save that 80 GB disk sda, which has a primary partition sda1. First I saved the partition table and the Master Boot Record this way:

# sfdisk -d /dev/sda >/tmp/server/images/cad1r-sfdisk-sda.dump
# dd if=/dev/sda bs=512 count=1 of=/tmp/server/images/cad1-sda.mbr

and then the partitions:

ntfsclone -s -o - /dev/sda1   | gzip | split -b 1000m - /tmp/server/images/cad1-sda1.img.gz_

Note that this saved disk image in 1G files, in case the way I mounted the share to the network server didn’t allow for large files. Sometimes that can get tricky going from Linux to a Windows 2003 server and back, so I decided not to take any chances. It makes a mess of files, but at least it took the guess-work out for me.

Coming soon, the restore process! Keep an eye out!

This is old news.

So, lately, there’s been a bit of buzz about Windows Home Server. The press has gone on at length about how this was such a brilliant idea who’s “time has finally come” and patted Microsoft on the back for thinking of it.
Well, this is really old news. As I was going through some old magazines recently, before throwing them out, I saw a short article on the Memora Servio Personal Server. A Linux-based home server that did everything from share files to filter e-mail to act as a firewall for home users. The device was auto-configuring and sat between your home network and your broadband connection and even could be ordered with wireless built in. The magazine was from 2001.

Sadly, the company doesn’t seem to be around any more, though you can see the old Memora About page, thanks to Archive.org. Again, this company was doing this six years ago and, from what I can tell, only seemed to have closed up shop in the past year or two. I wonder how well this product sold? What’s more, I wonder if anyone has the distro around, with the configuration programs on it? I know, I’d love to get my hands on that!

So, some of my readers are geeks, too. Ever heard of this product? Or, maybe, something like it?

(And, if you haven’t looked at my pictures yet, scroll down to untill you see them, then vote on the sidebar!)

No, not the kind you’re thinking of…

Remember all the hype about the Juice Box Personal Media Player for kids a couple of years ago? Well, they didn’t do so well in the market place, but, now you can get them cheap at Overstock, though to get the ones that let you upload your own media, you’ll pay a little more. Why is that cool, you ask? Simple, according to Street Tech, they run a flavor of Linux and are hackable. And, based on my own, personal research, that seems to be true. You can load Linux on them and they sure seem hackable.

Hmm, cheap, hackable, Linux-based… I might have to get “hands-on” with this…
Well, I have to do something while I wait for my eyebrows to grow back in!