Diary of a Network Geek

I don’t care what anyone says; you just can’t have enough data.

With storage being so relatively cheap, I don’t really get rid of any old data any more. It’s true. I have so much cheap storage around my house that I have literally hundreds upon hundreds of digital books, documents, photos and other files. I used to have a huge library in my house. Literally thousands of books. Books in virtually every room. The problem was, a lot of the books were horribly out of date. Or, I’d gotten them with the intention of reading them, but I was never honestly going to get around to reading many of them. Instead, they just took up space. So much space, in fact, that when my wife was getting ready to move in, I think she despaired of having room to fit! She really helped me realize that I didn’t need to keep all those physical books around. Though, I’m not sure she truly understands my personal obsession with data, or the brobdingnagian archive I have quietly lurking upstairs by the wifi router. I assure you, it’s epic. And, now I know that I’m not the only one, thanks to an article on Gizmodo this week.

My problem, though, is that I often remember some obscure bit of information that I read once on a website. Sometimes, I remember the site, but the page is missing. Or, the site is gone. Or, even worse, the site is still there, but it’s been taken over by domain squatters who are squeezing the Google pagerank to shill some internet snake oil of some kind. Then, I’m stuck trying to find that bit of data, that one reference that will take me to the promised land of information, often to no avail. Well, this week, while no doubt doing something totally unrelated, I stumbled across a Chrome plugin for the Wayback Machine. If you’re not familiar, the Wayback Machine is the search engine on The Internet Archive. And, it’s fantastic for guys like me, trying to dig up obscure and forgotten information. The plugin, according to its page, “[d]etects dead pages, 404s, DNS failures & a range of other web breakdowns, offering to show archived versions via the Internet Archive’s Wayback Machine. In addition you can archive web pages, and see their most recent & first archives.” And, I assure you, it’s glorious. It’s also free, so well worth installing. And, if you, like me, use Firefox as much as Google Chrome, there’s a Firefox version as well!

So, go ahead, fellow data hounds, install those plugins and relive the days of data past!

This post originally appeared on Use Your Words.

Guess who’s spending the weekend upgrading the company’s main server?

Finally after dealing with an aging server for too long, we’re upgrading.  And, not a minute too soon, either.  I have the joy of migrating Active Directory from a Windows 2003 server to a Windows 2012 server.  Not to mention, I get to migrate printing services, an iSCSI array connection, DNS and DHCP.  Wee!  What fun!

Well, I suppose that’s why I get the “big bucks”, right?  A system administrator’s work is never done!

Published via Pressgram

DNS has inherent weakness.

In it’s current form the Domain Name System, by it’s open nature, is pretty primed for exploitation.
Some of these attacks are more obvious than others, but there are two that I find particularly troubling.  More so that I can see them being used together to really mess with a website owner.
The first of these two attacks isn’t new.  But, the fact that it isn’t new and has been dealt with before doesn’t mean that it has suddenly stopped being effective.  The attack is called “DNS poisoning” and it works by corrupting the DNS cache on a server, which then forwards those poisoned DNS records as legitimate to other, unsuspecting servers.  The end result is that the attackers can redirect traffic from a legitimate website to their own site.  It’s hard to flat out stop right now, though, once discovered, it can be fixed with relatively little trouble.  This attack was used recently against several websites who were supporting SOPA and PIPA.  Of course, since these folks were trying to make a statement, it was pretty clear what had happened, so techs were working to fix it pretty quickly.
The second attack, which I would think include the first attack at its initial stages, is sub-domain hijacking.  In this attack, the attackers redirect the sub-domain of an existing site to another location.  This is a little more subtle and hard to detect.  In this case, the attackers are looking to profit from a well-established domain by “piggy-backing” on their reputation.  They poison the DNS records to point something like Viagra.google.com to their actual website, selling Viagra, or a site filled with spammy links that redirect a potential victim to their website selling Viagra, or whatever.   This attack takes a proactive system administrator to catch.  Since it doesn’t redirect any of the main, honest, actual site anywhere, but only uses its reputation to improve their own spammy links, it’s not always obvious that it’s going on.  Regular DNS record audits are about the only way to catch this, barring an angry end-user contacting the main site.

The internet is still a wild and wooly place sometimes, folks.  The reasons the professionals get paid what they do is because, theoretically, they have to deal with all that stuff and keep us safe!  Which reminds me, I have to go check my own company’s websites and DNS records, not to mention my own!
(The title, incidentally, was inspired by the movie that helped get me into this business, Sneakers. “Cattle mutilations are up.“)

I’m seeing traffic about this, so I thought I’d write up what I found.

I tweeted about a strange DNS-based network/malware attack that I saw on Friday, but, at the time, I didn’t see any interest, so I didn’t go into any real details.  Besides, I may be a hardcore geek, but I do have a life and was going out.  But, now, I’m seeing search engine traffic hitting my blog apparently looking for details, so I thought I’d describe the attack, as I saw it.

First of all, let me mention that I’ve seen a higher-than-usual occurrence of malware infections the past couple of weeks.  I mean, it’s a hazard of my business that, sooner or later, people are going to get infected, either through bad behavior or by accident, but the past three weeks or so I’ve seen way more problems like that than is even remotely normal.  So, bearing that in mind, I’ve been on a kind of high-alert status looking for any malware problems, but this was something new.

It started with someone from another location, who’s on a totally, physically separate network which uses a different internet service provider to connect to the Internet, calling me with a problem.  It was, apparently, a recurrence of a virus he had previously that we cleaned.  He described being taken to a webpage that featured a maroon graphic background with a white icon of a policeman holding up his hand to indicate “stop”.  The text on the page gave a message that said the user’s browser was not the correct version to access the page and that an upgrade was required.  Helpfully, it provided a button to press to receive the “upgrade”.  Obviously, the “upgrade” was an infection.  (You can see an example of the graphic here.)  Thankfully, I trained my users well enough to be suspicious of these kinds of things and no one who reported this actually clicked on it.

About the same time this happened, I noticed that my iPhone wasn’t connecting to the wifi hotspot I have setup in my office.  I checked the configuration and noticed that the DNS servers listed were wrong.  In fact, they’d all been replaced with a single DNS server; 188.229.88.7  Obviously, that seemed suspicious to me, so I opened a command prompt on my PC and did a tracert to see if I could figure out where this server was and, from that, why it had become the default DNS server on part of my network, despite my having very carefully configured totally different DNS servers that I knew were safe.  It looked like the tracert results showed me a network path that led out of the country somewhere, which was, to me, very suspicious.

Before I could really pursue that, though, I got another call from a user at my location reporting the exact same error message and graphic, but going to a totally different website! I went to his computer and checked the IP configuration and found that his DNS servers had been replaced by the rogue server as well.  I refreshed his network config, several times actually, and the DNS servers reset, but, when I thought to check some other people in the same area of the building, his configuration set itself back to the rogue DNS server!  So, I reset the local network equipment to clear the DNS cache, and whatever other caches may have gotten poisoned by this attack, and the problem seemed to go away.  Unfortunately, whatever had caused the compromise was still active and seemed to poison the DNS cache and the DNS configuration again.  It did seem sporadic, though, as if the ISP was trying to correct the issue at their end.

As far as I can tell, the attack actually seemed to be network-based in some way.
At least, I couldn’t find any computer on my network that was infected with anything that AVG, Norton Anti-virus, or Malware Bytes could find.  It is, I suppose, possible, that this attack was so new that no of those programs had an updated detection pattern for it, but, based on the lack of detection, and the fact that it happened on two physically separate networks almost simultaneously, leads me to believe that this was a network-based attack.  I suspect that an ARP cache or DNS cache or something similar was attacked and compromised on a major network router somewhere.  Possibly one of the edge routers at a trans-continental connection somewhere.  From the tracert results I had, it looked like it was the East Coast somewhere, leading to Europe via London to France, though I could be wrong.  It’s possible that was a blind alley meant to throw researchers off the trail in some way.
Also, as of this writing the rogue DNS server seems to be out of commission, though that might change, too.

The Internet is a wild and wooly place, ladies and gents, and you can’t always count on your friendly, neighborhood Network Geek to watch over you and keep you safe!  So, be careful out there!
(And, if you’re a fellow professional who’s seen this, too, leave me comments and tell me what you found!)
UPDATE: Looks like the server is still active, but my ISP has blocked DNS traffic to it, to fix the problem.
Also?  I hate the bastards that do these things.  I hate every last one of the little rat bastards!

UPDATE/FOLLOW-UP: So, it seems like a lot of people have been effected by this problem!
Check the comments for what other folks did and tools they might suggest to help with the problem.  Frankly, I wish I’d had known about those tools when I started my day!  Yes, I was *totally* wrong when I said it looked like it was coming in from outside the routers.  It was, in fact, *several* PCs that were infected with whatever it was.  I found it, much like at least one commenter, by checking the results of “ipconfig /all” in a command prompt.  I noticed that the DHCP server listed in the config was NOT my actual DHCP server!  So, as I went from machine to machine, I saw several PCs that kept coming up as DHCP servers.  I used Malware Bytes to scan the infected PCs and it seemed to clean them off.  At least, for now.  I’m not sure what I’ll find in the morning.
Apparently, Friday, when it looked like the problem was getting cleaned up, it was really just people shutting their workstations down early for the long weekend.
In any case, as at least one commenter has mentioned, it looks like updates for the various scanners should be coming out this week, so keep updating your antivirus and antispyware programs and scan your networks!  Well, scan them more completely and carefully than you already have.
And, as always, if you have any new information or suggestions for tools to clear up the issue, please, leave them in the comments!

No, not your personal name, network names!

Yeah, since I’ve been thinking about computer security a little in this new year and new decade, I’ve noticed a slightly disturbing trend.  Spammers have been working at redirecting you to compromised domains.  One way they do it is something called DNS cache poisoning.  Another is straight-up DNS hijacking.

Okay, let me back up a second.  For my slightly less-technical readers, DNS stands for Domain Name System.  That’s the system of servers that translates website names, like “www.google.com”, into addresses that your computer understands and can connect you to via a browser.  It’s how you found my blog, though you may not have even realized it.
DNS Hijacking is usually accomplished via a “rouge” server, which is a server setup by spammers to publish bad information.  The more usual method, I think, and more insidious, is DNS cache poisoning.  With that method, spammers trick good, valid DNS servers into updating their records with bad information.  Giving them poisonous information, if you will.

So, now, back to the hard-core server admins.  Last week I was reminding everyone that the start of a new year is a great time to change passwords, but it’s also a great time to check on other security issues, like your DNS.  Luckily, Michael Kassner over at TechRepublic has written a blog post titled Test your DNS servers for spoofability.  It’s worth a read and worth running through.  Maybe even making it a regular practice, to see if your DNS has been compromised.

Oh, and if you all want to read more about DNS, and how to implement it, there’s a great book from O’Reilly titled [amazon_link id=”0596100574″ target=”_blank” ]DNS and BIND[/amazon_link] that’s well worth owning.  Trust me.

Advice from your Uncle Jim:
"Every calling is great when greatly pursued."
   --Oliver Wendell Holmes

My house almost blew up tonight.

No, really.
As I write this, I’m waiting for a serviceman from Centerpoint Energy to come out and turn my gas back on. Apparently, shortly after I left for church this evening a neighbor smelled gas by my house. They called emergency services in Jersey Village, where I live. When the emergency services got out, they could smell gas, so they turned off my power and got Centerpoint out. Centerpoint repaired the leak, or so it seems. Actually, the first thing the serviceman is going to check is to make sure the leak isn’t a leak any more. Then, assuming all is well, he’ll turn the gas on and I can light all my pilots and so on. That’s why the gas was turned off when I got home. Naturally, they couldn’t find me, since I’d gone to church, so they couldn’t turn anything on until they knew I could light all the pilots again. Otherwise, my house would have filled with gas, hit something electric and, well, let’s just say I was irritated with the whole thing until I got the lights back on and figured out what almost happened. Thank God!

I guess my little efforts and building up a little good karma in the short term paid off today.
See, I helped two people with computer things today. This morning, I started reformating and reinstalling my friend J.’s wife, L’s, laptop. She’d been having problems with it and I looked at it several times, but I couldn’t find anything obviously wrong. So, rather than spend any more time on it, I backed up her files and started reinstalling it. Of course, that got interrupted by my power being out, but, still, I’m sure I’ll get it done pretty quick now while I wait for Centerpoint to turn my gas back on.

The other person I did a little mitzvah for was Alison over at Inspired Work of Self-Indulgence.
She was having a little trouble with Comcast and their network. More specifically, it seemed to me she was having a problem with the assigned DNS servers. As any good network geek knows, an Internet connection with out DNS isn’t very useful. She had a connection, sure, but she couldn’t get anywhere. Comcast had been out multiple times trying to fix things. They’d even changed out her cable modem, which seemed to be connected fine when I looked at it. But, her wireless gateway was picking up the normal, default DNS servers from Comcast’s autoconfiguration. I reset the wireless router to the factory default and tweaked the settings a bit. The part that I think mattered the most was adding additional DNS servers from outside Comcast’s network. So now, if the main DNS server goes down or stops working, the backup servers should still find what she’s looking for.
Though, if the problem comes back, it occurred to me on the way home from church that it might be her laptop turning off her wireless connection in a power-save mode, so I might have to check that. (If she’s reading this, that’s a hint!) In any case, it was nice to at long last finally get to meet her. We’ve exchanged e-mail and comments and read each other’s blog posts for quite literally years now. She even sent me cookies when I was in the hospital getting chemotherapy that first time! But today was the first time actually meeting. Hopefully, now that she’s seen I’m completely harmless, it won’t be the last time.
Oh, and I got to meet Piedmont and his new pet human, er, owner. Both were sweet, too. And, of course, I got to meet Cheyenne, Alison’s chocolate lab. She’s as cute in person as in the pictures Alison takes of her and as spoiled, too! She has almost as much tummy as my own Hilda!

So, in spite of almost coming home to a smoking hole in the ground where my house used to be, it turned out to be quite a good day indeed! Stay warm, y’all, and make sure to check your gas!

Update: Apparently, my sins still out-weigh my good deeds, since the guy from Centerpoint Energy never showed up. I stayed up waiting for him until 4:00am, then went to bed, mumbling about liars in the night. I wish it were the first time.
I’m half tempted to turn it on myself and to hell with them. But, better safe than sorry, so I’ll just mumble quiet, Lovecraftian curses on them and try to be on about my day until the next lying serviceman arrives.

Advice from your Uncle Jim:
" Reasonable men adapt themselves to their environment; unreasonable men try to adapt their environment to themselves. Thus all progress is the result of the efforts of unreasonable men."
   --George Berbard Shaw

Yes, Directory Naming Service needs security.
Oh, my, in this modern, Internet-enabled age DNS security is a larger and larger concern every man-hour. DNS Spoofing, DNS Poisoning, DNS Highjacking, just to buzzword out a few scary key words, are all issues that need to be addressed. And, once again, TechRepublic has an article on it. This time, though, it’s a download: Ten Things You Should Know About Securing DNS. Granted, this is a pretty esoteric topic, but, then again, you’re at the Diary of a Network Geek, where Alpha Geeks come to sniff each other’s… Er, I think I’ve gone one metaphor too far with that one! Besides, with a name like that, did you expect something simple and user-friendly? No, gentle readers, not here. Nothing but the most difficult and challenging questions and issues. (Coming soon: Geek Dating!)
Anyway, best to read the article before someone starts calling your website names.
Take some advice from your Uncle Jim, computer security is nothing to ignore!

Advice from your Uncle Jim:
"Not a shred of evidence exists in favor of the idea that life is serious."

Well, I’m home again.
I got in last night around 10:00pm, which, after a stop at the store for milk, got me home around 11:15pm. I was supposed to get in three hours earlier, but, well, getting back to Houston from Panama City was rather like Orpheus returning from Hades. In fact, the entire trip became Hell when we ended up staying up until 3:30am Wednesday night to get the server changed over. That was about 6-8 hours later than planned due to a wierd problem with DNS on the Netware 6.5 server and a couple of gigabytes of new data that had to be copied to the new server. Apparently, someone felt the need to backup a bunch of MPEGs, many of a dubious nature. I believe that many of the movies in question got deleted, but it still messed up our plans.
The rest of the trip was okay, but colored by the lack of sleep that Wednesday night. I feel like my soul is only now catching up to the rest of my body. Someone once described jet lag as the time it takes for your soul to catch up to your body. That, of course, is based on the idea that air-travel moves your body faster than your soul can keep up and the gap while the two resync is why you feel all wonky after air travel. I’ll buy it.
Anyway, much to my relief, nothing was wrong with the house when I returned. Everything was in its little place and all was well with my little world. Though, I would have been happier if my dog had been there to greet me. Ah, well, perhaps there’ll be another dog one day.
Oh, speaking of “one day” I have pictures, but I’ll post them next week sometime. Right now, I’m still readjusting to my soul having arrived sometime after my bags.

Er, well, actually, it’s specific to ZENWorks.

I’ve been yapping about all the ZENWorks desktop imaging stuff that I’ve been doing lately and, apparently, a couple of you have been listening because you’ve asked for the CD. Well, I’m pleased to provide it for you.
A couple of notes, though…
1) You’ll have to edit the ISO to update the “settings.txt” file to reflect your own default server.
2) If you choose “Automatic” from the first menu, you’ll get an error because this shell script has been modified to TFTP a menu to the local server. (This link let’s you grab an example.)
3) The script that pulls the menu tries to hit ZFD1, so you either have to change that or add a DNS entry for it.

I think that’s it, but I’m not going to support this, ya’ hear? Y’all are on your own! I spent the better part of a month working this bit out and I know that it’s not perfect, but it works for us.
Oh, yeah, here’s a link to the ISO. (Be warned, it’s about 10meg. It didn’t shrink when I ZIPed it, so I just uploaded the ISO straight.)

Good luck!

Note: I removed the ISO due to lack of downloads and space issues. And, also, it’s probably so far out of date that it’s not really useful any more.

On Google, at least.

For the longest time, I was the #1 hit on Google, when you searched for “CNE Resume”. But, now, I’m not. If you do a search, you can see that some bottom-feeding scum-sucker has illegally pirated my page to manipulate himself into the top postion. When I looked at the source of the page, I see my entire HTML code that I so carefully crafted to optimize my page for the engines. If you notice, there is now a copyright at the bottom of my resume. So, now it’s explicitly clear that this low-life scum-bag has violated Federal copyright law by illegally reposting my copyrighted material. So, should I send a “cease and desist”? Should I hire a lawyer to do it? (I know a few and could probably get one to draft a letter cheap.) I’ve even thought about being less than morally upright and hitting the nasty, little bugger with some DNS Spoofing. But, of course, that might have legal ramifications, so, I won’t. Still it was nice to contemplate…

Anyway, if you want to help me out, loyal readers (reader?), link to me. Check Google once in awhile and click on my valid link. In the meantime, I’ll be updating my page and working on my keywords!!