Diary of a Network Geek

I’m seeing traffic about this, so I thought I’d write up what I found.

I tweeted about a strange DNS-based network/malware attack that I saw on Friday, but, at the time, I didn’t see any interest, so I didn’t go into any real details.  Besides, I may be a hardcore geek, but I do have a life and was going out.  But, now, I’m seeing search engine traffic hitting my blog apparently looking for details, so I thought I’d describe the attack, as I saw it.

First of all, let me mention that I’ve seen a higher-than-usual occurrence of malware infections the past couple of weeks.  I mean, it’s a hazard of my business that, sooner or later, people are going to get infected, either through bad behavior or by accident, but the past three weeks or so I’ve seen way more problems like that than is even remotely normal.  So, bearing that in mind, I’ve been on a kind of high-alert status looking for any malware problems, but this was something new.

It started with someone from another location, who’s on a totally, physically separate network which uses a different internet service provider to connect to the Internet, calling me with a problem.  It was, apparently, a recurrence of a virus he had previously that we cleaned.  He described being taken to a webpage that featured a maroon graphic background with a white icon of a policeman holding up his hand to indicate “stop”.  The text on the page gave a message that said the user’s browser was not the correct version to access the page and that an upgrade was required.  Helpfully, it provided a button to press to receive the “upgrade”.  Obviously, the “upgrade” was an infection.  (You can see an example of the graphic here.)  Thankfully, I trained my users well enough to be suspicious of these kinds of things and no one who reported this actually clicked on it.

About the same time this happened, I noticed that my iPhone wasn’t connecting to the wifi hotspot I have setup in my office.  I checked the configuration and noticed that the DNS servers listed were wrong.  In fact, they’d all been replaced with a single DNS server; 188.229.88.7  Obviously, that seemed suspicious to me, so I opened a command prompt on my PC and did a tracert to see if I could figure out where this server was and, from that, why it had become the default DNS server on part of my network, despite my having very carefully configured totally different DNS servers that I knew were safe.  It looked like the tracert results showed me a network path that led out of the country somewhere, which was, to me, very suspicious.

Before I could really pursue that, though, I got another call from a user at my location reporting the exact same error message and graphic, but going to a totally different website! I went to his computer and checked the IP configuration and found that his DNS servers had been replaced by the rogue server as well.  I refreshed his network config, several times actually, and the DNS servers reset, but, when I thought to check some other people in the same area of the building, his configuration set itself back to the rogue DNS server!  So, I reset the local network equipment to clear the DNS cache, and whatever other caches may have gotten poisoned by this attack, and the problem seemed to go away.  Unfortunately, whatever had caused the compromise was still active and seemed to poison the DNS cache and the DNS configuration again.  It did seem sporadic, though, as if the ISP was trying to correct the issue at their end.

As far as I can tell, the attack actually seemed to be network-based in some way.
At least, I couldn’t find any computer on my network that was infected with anything that AVG, Norton Anti-virus, or Malware Bytes could find.  It is, I suppose, possible, that this attack was so new that no of those programs had an updated detection pattern for it, but, based on the lack of detection, and the fact that it happened on two physically separate networks almost simultaneously, leads me to believe that this was a network-based attack.  I suspect that an ARP cache or DNS cache or something similar was attacked and compromised on a major network router somewhere.  Possibly one of the edge routers at a trans-continental connection somewhere.  From the tracert results I had, it looked like it was the East Coast somewhere, leading to Europe via London to France, though I could be wrong.  It’s possible that was a blind alley meant to throw researchers off the trail in some way.
Also, as of this writing the rogue DNS server seems to be out of commission, though that might change, too.

The Internet is a wild and wooly place, ladies and gents, and you can’t always count on your friendly, neighborhood Network Geek to watch over you and keep you safe!  So, be careful out there!
(And, if you’re a fellow professional who’s seen this, too, leave me comments and tell me what you found!)
UPDATE: Looks like the server is still active, but my ISP has blocked DNS traffic to it, to fix the problem.
Also?  I hate the bastards that do these things.  I hate every last one of the little rat bastards!

UPDATE/FOLLOW-UP: So, it seems like a lot of people have been effected by this problem!
Check the comments for what other folks did and tools they might suggest to help with the problem.  Frankly, I wish I’d had known about those tools when I started my day!  Yes, I was *totally* wrong when I said it looked like it was coming in from outside the routers.  It was, in fact, *several* PCs that were infected with whatever it was.  I found it, much like at least one commenter, by checking the results of “ipconfig /all” in a command prompt.  I noticed that the DHCP server listed in the config was NOT my actual DHCP server!  So, as I went from machine to machine, I saw several PCs that kept coming up as DHCP servers.  I used Malware Bytes to scan the infected PCs and it seemed to clean them off.  At least, for now.  I’m not sure what I’ll find in the morning.
Apparently, Friday, when it looked like the problem was getting cleaned up, it was really just people shutting their workstations down early for the long weekend.
In any case, as at least one commenter has mentioned, it looks like updates for the various scanners should be coming out this week, so keep updating your antivirus and antispyware programs and scan your networks!  Well, scan them more completely and carefully than you already have.
And, as always, if you have any new information or suggestions for tools to clear up the issue, please, leave them in the comments!

There’s a new version of Wireshark out.

If that means nothing to you, then you’re not one of my geek readers.  And, that’s cool.  Hold on and something more interesting will be coming for you soon.

For those of you who are geeks, check this out, okay?  There are tons of improvements, including a Windows 64-bit installer, improved reports and assorted output, the inclusion of GeoIP lookups, and many, many more.  I’ve used Wireshark, on and off, for several years now and these are really good additions.  I especially dig the ability to lookup where IPs may be coming from more easily.  I often have to deal with international TCP/IP traffic and knowing which is from where can be really helpful.
And, yes, there is still a version for Mac and Linux, besides the new Windows versions.

So, go check this out.  If you didn’t want to fillow the link above to Lifehacker, you can just jump right to the Wireshark site to see their marketing pitch on the latest version.  And, of course, it’s still all free.

It seems like this comes around on a regular basis.

Server names and naming conventions are a constant source of argument and irritation in big IT departments.  Everyone has their own idea of just what naming schema should be used for the servers and workstations and such on the network.  And, since it hasn’t shown up recently on Slashdot, we were about due for an article on it.  There is; Why do we name servers the way we do?  The comments, if you can be bothered to dig down deep into them and wade past some of the worst attempts at humor, are quite telling.  It doesn’t take long before the relative merits of using quirky, easy to remember names is being quite hotly debated.

The original article  over at IT World, titled Would a server by any other name be as functional?, seems to weigh in on the side of the more creative names.
I’ve worked both kinds of places, actually.  In one job, we used a very precise naming convention that had been put in place after some, apparently, very intense debate.  There, we used the LocationFunctionOperatingSystemNumber kind of naming system.  So that the first Accounting server in Houston running Novell Netware would be HOUACTNW01.  Perfectly clear to me, actually, because of that job.  It’s a logical system and works well enough, though it does lack a certain “zing”.
At most other jobs, though, we tended toward the other way.  Once, I worked with a guy who named his servers after dead musicians and actors, but that was only so he could ping his favorite router and see “Hendrix is alive” come back to him.  Another place, we used various things and it was, well, far less themed and much more confusing.  I think it’s best to choose from a very, very large mythology or naming pool so that you don’t have to switch themes mid-stream.  We had some servers named for “gods of the underworld” and others that were named after space shuttles at the same company.  There was no rhyme or reason to it, really, just what the last guy felt like doing.

I’m not sure what naming convention I’ll finally use when I finally get around to redoing my network at home.  It’s hard to get motivated, you know?  When you do it at work all day?  Makes you feel sorry for sex workers and gynaecologists, not to mention urologists, doesn’t it?
(Yeah, this is what happens when I stay up way too late.  Or is it too early?)

Advice from your Uncle Jim:
Knowing that something is the right thing doesn't always make it easy to do, but that's not an excuse.

I love stories and today is the National Day of Listening.

One of the hardest things to explain to geeks on a helpdesk is that the everything they do is about people. It’s not about systems or networks or computers, but, rather, the people who use them. It’s amazing to me how many people don’t seem to get that. And, for me, people are about stories. The story of someone’s life can be a fascinating thing, if we just take the time to listen.

In my family, I’ve become my generation’s historian. I’ve collected the stories of all those relatives marching back into time and memory. I got them from both my parents and my paternal grandmother, who lived with us from the time I was born until she passed away when I was in college. All that time, I collected stories. I can tell you the story of my great-great-great grandfather who fought in the civil war after getting drunk and signed up by a recruiting agent. (But, since I’m in the South, I won’t tell you what side he fought for!) I can tell you about my great-grandfather who rode the rails with Hinky-Dink Kenna and Bathhouse John Kenzie, two of Chicago’s most notorious Aldermen. I can even tell you about how that same great-grandfather took my father to that Bathhouse John’s house of ill-repute and the “nice ladies” who doted on him while he was there on the porch.
But, many families have lost their stories. They don’t know their history. The National Day of Listening is meant to help keep that from happening to another generation. I saw this on LifeHacker first, but I’ve heard about the group running it, StoryCorps, on NPR. The idea is simple. Go find one of your older relatives and ask questions about their life. Interview them, if you will. And then, listen, and pass the story on. Go to the link and download the guide and then, do it. When you start collecting the stories of your family’s life, I think you’ll be glad you did. I hope so.

Advice from your Uncle Jim:
"Sometimes a winner is just a dreamer who never gave up."

In my “other life”, I write poetry.

I do more than network PC’s together with Netware and Windows2000. I have a life. Sure, it’s a sad, geeky life, but still, I do have one. In my other life, I write poetry and fiction. Though, I have to admit, I spend a lot more time being a Network Engineer than I do being a poet.

In any case, I finally got a bit of fiction up on my other website and thought I’d throw a link to it here. Enjoy!

Okay, so I’ve gotten a little deeper into the network and I’m finding more “little” things wrong. For one, the servers aren’t all on the same version of Directory Services. For those of you who aren’t Certified Netware Engineers, that’s a bad thing. NDS, or Netware Directory Services, is the distributed database of everything, including security, on the network. If all the servers don’t have the same revision of the NDS programs, called NLMs in Netware Land, there can be problems with security and access between servers. Enough of those problems and the network stops running. But, so far, that hasn’t been a real issue. So, I’m not too worried. I’m working on correcting the problem, but it can wait until the next “maintenance weekend” to reboot the servers so this will take effect.
The other “issue” that sticks out for me is a bad time synchronization plan. I don’t think that it got this way on purpose, but time synchronization is all wrong. Mainly, it looks like the last Netware Admin had no experience on Netware 4.x or greater. I won’t go into the details, but… Well, let’s just say it needs work. This is something that I can work on, and implement, without taking the servers down. I’ve started working on that, too. It’ll take a couple of days to get straight, but that will improve network performance quite a bit.
And, of course, there’s virtually no network documentation. Not a big surprise, if you have any experience with taking over a network. How many Network Engineers really want to write documentation? But, I know it needs to be done. And, I also know how good that looks to management. (Yeah, yeah, I know. I’m kissing butt already. *sigh*) Anyway, I need to go get my sleep so I can be early to the office and hard at work. Updates soon!