Diary of a Network Geek

DNS has inherent weakness.

In it’s current form the Domain Name System, by it’s open nature, is pretty primed for exploitation.
Some of these attacks are more obvious than others, but there are two that I find particularly troubling.  More so that I can see them being used together to really mess with a website owner.
The first of these two attacks isn’t new.  But, the fact that it isn’t new and has been dealt with before doesn’t mean that it has suddenly stopped being effective.  The attack is called “DNS poisoning” and it works by corrupting the DNS cache on a server, which then forwards those poisoned DNS records as legitimate to other, unsuspecting servers.  The end result is that the attackers can redirect traffic from a legitimate website to their own site.  It’s hard to flat out stop right now, though, once discovered, it can be fixed with relatively little trouble.  This attack was used recently against several websites who were supporting SOPA and PIPA.  Of course, since these folks were trying to make a statement, it was pretty clear what had happened, so techs were working to fix it pretty quickly.
The second attack, which I would think include the first attack at its initial stages, is sub-domain hijacking.  In this attack, the attackers redirect the sub-domain of an existing site to another location.  This is a little more subtle and hard to detect.  In this case, the attackers are looking to profit from a well-established domain by “piggy-backing” on their reputation.  They poison the DNS records to point something like Viagra.google.com to their actual website, selling Viagra, or a site filled with spammy links that redirect a potential victim to their website selling Viagra, or whatever.   This attack takes a proactive system administrator to catch.  Since it doesn’t redirect any of the main, honest, actual site anywhere, but only uses its reputation to improve their own spammy links, it’s not always obvious that it’s going on.  Regular DNS record audits are about the only way to catch this, barring an angry end-user contacting the main site.

The internet is still a wild and wooly place sometimes, folks.  The reasons the professionals get paid what they do is because, theoretically, they have to deal with all that stuff and keep us safe!  Which reminds me, I have to go check my own company’s websites and DNS records, not to mention my own!
(The title, incidentally, was inspired by the movie that helped get me into this business, Sneakers. “Cattle mutilations are up.“)

There is one, apparently.

So, it seems someone has figured out the answer to an old question which has often plagued IT professionals: How long are your skills good?  According to Eric Bloom, over at IT World, longer than you think.  He claims that the tech skills you have now will be half as marketable in two years.  If you read Slashdot, you’ve seen this article and the comments that followed.  Here are my thoughts, though.

First, I think it depends on the skills involved.
For example, if you’re working on Windows Server, your skills will probably translate fairly well and that two-year half-life is about right.  For Unix, maybe a bit longer than that.  For Novell, well, sadly, I’m not sure who actually uses that old warhorse any more, as much as it makes me sad to write it.  For other, less vendor oriented skills, I think two-years may be a bit short-sighted.  Take routers, for instance.  Now basic routing hasn’t really changed in quite a long time.  Even Cisco routers, the creme-de-la-creme of enterprise routers, haven’t really changed that much on the inside in the last 15 years.  I was in one the other day and I have to admit I was shocked at how quickly the skills came back to me after quite literally years of disuse.  Far more than two years, I might add.
Also, skills that are a little harder to quantify certainly stay “fresh” longer than those hypothetical two years.  Things like troubleshooting and the so-called soft skills involved with user support are something that I think are deeply engrained in someone.  They’re part of a work ethic.  So the customer service skills I learned more than 20 years ago when I worked for Hyatt Hotels are certainly still more than “good”.

Secondly, Mr. Bloom is talking about marketability, not actual utility.
So, the fact that, for instance, I don’t have a Cisco certification, even though I’m clearly capable of configuring a Cisco router, means that quite probably was never what he would have considered a “marketable skill”.  In fact, based on what many recruiters may have felt about the marketability of my skills, I should be farming beets right now, not working as the Lead Tech/IT Manager of a fairly prosperous design and manufacturing company.  Instead, of course, all through my career, I’ve managed to talk my way through the door and then show the people in charge that versatility and adaptability, not to mention mad Google-query-crafting skills, are far more important than any specific past experience or certification.

So, what about you, gentle readers?  What do you think?  How long are tech skills “good”?  And does working on legacy systems harm your future employability?

I’m seeing traffic about this, so I thought I’d write up what I found.

I tweeted about a strange DNS-based network/malware attack that I saw on Friday, but, at the time, I didn’t see any interest, so I didn’t go into any real details.  Besides, I may be a hardcore geek, but I do have a life and was going out.  But, now, I’m seeing search engine traffic hitting my blog apparently looking for details, so I thought I’d describe the attack, as I saw it.

First of all, let me mention that I’ve seen a higher-than-usual occurrence of malware infections the past couple of weeks.  I mean, it’s a hazard of my business that, sooner or later, people are going to get infected, either through bad behavior or by accident, but the past three weeks or so I’ve seen way more problems like that than is even remotely normal.  So, bearing that in mind, I’ve been on a kind of high-alert status looking for any malware problems, but this was something new.

It started with someone from another location, who’s on a totally, physically separate network which uses a different internet service provider to connect to the Internet, calling me with a problem.  It was, apparently, a recurrence of a virus he had previously that we cleaned.  He described being taken to a webpage that featured a maroon graphic background with a white icon of a policeman holding up his hand to indicate “stop”.  The text on the page gave a message that said the user’s browser was not the correct version to access the page and that an upgrade was required.  Helpfully, it provided a button to press to receive the “upgrade”.  Obviously, the “upgrade” was an infection.  (You can see an example of the graphic here.)  Thankfully, I trained my users well enough to be suspicious of these kinds of things and no one who reported this actually clicked on it.

About the same time this happened, I noticed that my iPhone wasn’t connecting to the wifi hotspot I have setup in my office.  I checked the configuration and noticed that the DNS servers listed were wrong.  In fact, they’d all been replaced with a single DNS server; 188.229.88.7  Obviously, that seemed suspicious to me, so I opened a command prompt on my PC and did a tracert to see if I could figure out where this server was and, from that, why it had become the default DNS server on part of my network, despite my having very carefully configured totally different DNS servers that I knew were safe.  It looked like the tracert results showed me a network path that led out of the country somewhere, which was, to me, very suspicious.

Before I could really pursue that, though, I got another call from a user at my location reporting the exact same error message and graphic, but going to a totally different website! I went to his computer and checked the IP configuration and found that his DNS servers had been replaced by the rogue server as well.  I refreshed his network config, several times actually, and the DNS servers reset, but, when I thought to check some other people in the same area of the building, his configuration set itself back to the rogue DNS server!  So, I reset the local network equipment to clear the DNS cache, and whatever other caches may have gotten poisoned by this attack, and the problem seemed to go away.  Unfortunately, whatever had caused the compromise was still active and seemed to poison the DNS cache and the DNS configuration again.  It did seem sporadic, though, as if the ISP was trying to correct the issue at their end.

As far as I can tell, the attack actually seemed to be network-based in some way.
At least, I couldn’t find any computer on my network that was infected with anything that AVG, Norton Anti-virus, or Malware Bytes could find.  It is, I suppose, possible, that this attack was so new that no of those programs had an updated detection pattern for it, but, based on the lack of detection, and the fact that it happened on two physically separate networks almost simultaneously, leads me to believe that this was a network-based attack.  I suspect that an ARP cache or DNS cache or something similar was attacked and compromised on a major network router somewhere.  Possibly one of the edge routers at a trans-continental connection somewhere.  From the tracert results I had, it looked like it was the East Coast somewhere, leading to Europe via London to France, though I could be wrong.  It’s possible that was a blind alley meant to throw researchers off the trail in some way.
Also, as of this writing the rogue DNS server seems to be out of commission, though that might change, too.

The Internet is a wild and wooly place, ladies and gents, and you can’t always count on your friendly, neighborhood Network Geek to watch over you and keep you safe!  So, be careful out there!
(And, if you’re a fellow professional who’s seen this, too, leave me comments and tell me what you found!)
UPDATE: Looks like the server is still active, but my ISP has blocked DNS traffic to it, to fix the problem.
Also?  I hate the bastards that do these things.  I hate every last one of the little rat bastards!

UPDATE/FOLLOW-UP: So, it seems like a lot of people have been effected by this problem!
Check the comments for what other folks did and tools they might suggest to help with the problem.  Frankly, I wish I’d had known about those tools when I started my day!  Yes, I was *totally* wrong when I said it looked like it was coming in from outside the routers.  It was, in fact, *several* PCs that were infected with whatever it was.  I found it, much like at least one commenter, by checking the results of “ipconfig /all” in a command prompt.  I noticed that the DHCP server listed in the config was NOT my actual DHCP server!  So, as I went from machine to machine, I saw several PCs that kept coming up as DHCP servers.  I used Malware Bytes to scan the infected PCs and it seemed to clean them off.  At least, for now.  I’m not sure what I’ll find in the morning.
Apparently, Friday, when it looked like the problem was getting cleaned up, it was really just people shutting their workstations down early for the long weekend.
In any case, as at least one commenter has mentioned, it looks like updates for the various scanners should be coming out this week, so keep updating your antivirus and antispyware programs and scan your networks!  Well, scan them more completely and carefully than you already have.
And, as always, if you have any new information or suggestions for tools to clear up the issue, please, leave them in the comments!

No, not your personal name, network names!

Yeah, since I’ve been thinking about computer security a little in this new year and new decade, I’ve noticed a slightly disturbing trend.  Spammers have been working at redirecting you to compromised domains.  One way they do it is something called DNS cache poisoning.  Another is straight-up DNS hijacking.

Okay, let me back up a second.  For my slightly less-technical readers, DNS stands for Domain Name System.  That’s the system of servers that translates website names, like “www.google.com”, into addresses that your computer understands and can connect you to via a browser.  It’s how you found my blog, though you may not have even realized it.
DNS Hijacking is usually accomplished via a “rouge” server, which is a server setup by spammers to publish bad information.  The more usual method, I think, and more insidious, is DNS cache poisoning.  With that method, spammers trick good, valid DNS servers into updating their records with bad information.  Giving them poisonous information, if you will.

So, now, back to the hard-core server admins.  Last week I was reminding everyone that the start of a new year is a great time to change passwords, but it’s also a great time to check on other security issues, like your DNS.  Luckily, Michael Kassner over at TechRepublic has written a blog post titled Test your DNS servers for spoofability.  It’s worth a read and worth running through.  Maybe even making it a regular practice, to see if your DNS has been compromised.

Oh, and if you all want to read more about DNS, and how to implement it, there’s a great book from O’Reilly titled [amazon_link id=”0596100574″ target=”_blank” ]DNS and BIND[/amazon_link] that’s well worth owning.  Trust me.

Advice from your Uncle Jim:
"Sticking to good habits is like having a savings account: when hard times come, we can take the 'investment' we've made and overcome our problems."

So, Friday I went to my first geek networking event.

Okay, now, to be clear, I mean the first event I’ve ever been to that the point was to get to know other geeks, not work on computer networking. It was the first of what I think will be many. The event is a monthly “Geek Gathering” put on by Jay Lee and Dwight Silverman of TechBytes and the Houston Chronicle. I was convinced to come out this time by Kristie “Suburban Goddess” MacLaughlin. Though, I have to admit, I think she did it simply to get me active on Facebook, since the event was announced there. We’ve followed each other’s blogs for some time now, and exchanged a few e-mails, but never met. And, before any of my regular readers who are often eager for me to get involved with someone, mainly for the jokes that come out of my so-called love life, let me hasten to emphasize that this was not a date! She’s quite happy with her boyfriend and doesn’t need my uncivilized self mucking things up. Just in case anyone was wondering.

Still, she was quite eager to get me out to meet Jay and Dwight and “the gang”. Now, I did meet those folks, but I got caught in ugly traffic, so I got there a bit late and didn’t get to meet everyone I might have liked to know. For instance, I missed meeting the guys who run the Houston-based Japanese animation and manga convention known as ONI-CON. I did meet a couple of very nice journalists who got laid off from the Chronicle, however, who were there networking as well.
Funny thing about that, the networking thing. I’ve never really done it before, and I think it showed. Living alone, my conversational skills have atrophied so as to be almost non-existent! Thankfully, I was surrounded by people who were good at it and gracious. Donna, aka @Cottonwood2009, was very nice and kept my end of the conversation up as well as her own. And, of course, it seemed that everyone was on either Twitter or Facebook or both. I’m on Twitter for the tools I can add to this blog in case of losing my connection to the Internet again during a hurricane, so my family up North will know that I’m still alive. (They worry.) I was on Facebook, but hadn’t really done anything with the account. Since meeting a few folks, most notably Dwight and Jay, my Twitter followers have more than tripled in the space of two days and my Facebook friends have gone from one to eighteen.
And, I got to talk with several people about photography, too. In fact, it seemed like every third person there had a camera in hand. I didn’t take many pictures this time, but you can see the few I did at my Flickr page, under Geek Gatherings. There will be more eventually.

I remember when the on-line world and the “real” world were mostly separate. I’ve blogged for almost nine years now and never actually expected to meet most, if not all, of my readers. But, as the song goes, the times, they are a-changin’ and now, I fear, I’ll be meeting more and more of them. Perhaps I’ll have to start writing better and more relevant things! Good gravy, I may have to start writing technical posts again! What a strange turn of events that would be.
I have to be honest, and in all seriousness, meeting some of my readers makes me, well, a little uncomfortable. It’s bad enough that my readers often think that they know me based on what I write here, but now… Now, having met me in person, I’m afraid that the lines will become even more blurred. And, anyone who’s read my blog for a long time knows how I like my life neatly segmented and clearly defined. Still, what else is there to do? It’s either that or keep talking to the dog and, frankly, I think she’s getting a bit tired of hearing the same jokes over and over.

But, it was good to be out of my comfort zone Friday. It was good to go to a new place and meet new people. And, it was even good to put some faces to names that I’d seen on-line in any number of venues. I may not be very good at this networking thing, as ironic as that seems considering the title of my blog, but I’m going to keep working at it. It’s the one thing that I should have been working at harder all these years. Who knows, maybe I would have had a better series of jobs than I did? Maybe even a better series of girlfriends? Well, maybe not, but a guy can dream! Certainly going forward, if anything happens to me at my current job, I’ll be better off if I have a good, strong, professional network. If you’re in Houston and a geek like me, it would be worth checking out next month.
And, who knows, maybe one of these times I’ll meet the future ex-Mrs. Hoffman?