The Day(s) the Earth Stood Still

No.  It isn’t a reference to the 1951 science fiction movie, it is a reference to last Friday when my Internet connection slowed to a crawl and despite the best efforts of Comcast, I continued to experience outages and slowness.  Time to take matters into my own hands.

I started with taking a look at the firewall, in my case Small Business Server 2003 (ISA 2004 I think). Plenty of activity there with Network bandwidth utilization running at about 5% but as I watched I saw a weird pattern emerge.  My pings to an external server began to time out as the network utilization spiked and then I got long latency on returns. Then the network utilization dropped for a minute or so and began to ramp up again. A weird pattern.  Of course, figuring out what was causing the traffic spikes is next to impossible with the tools ISA has but through trial and error I discovered my Asterisk server was the source.  Looks like I was hacked. And even through only SIP traffic was allowed to and from the box, there must have been an exploit that someone used to push files on and off the server.  So I pulled the plug (I have rebuilt with AsteriskNow).

On Monday, late in the afternoon we had a wind storm come through Salt Lake (69 mph nearby) and the lights flickered for only a second.  I really didn’t give it a second thought, my servers have redundant power supplies and UPS’s.  A few moments later I lost Internet connectivity and thought that I had been hacked again (I have another Asterisk server that isn’t exposed to the outside world…but, you never know) so I went down to the server room to take a look.  SBS 2003 was rebooting but the blade center was merrily humming along.  Looks like I need a new UPS battery.

Tuesday morning I discovered that the JS12 blade in the blade center had dropped as well and my IBM i wasn’t running (nor the Linux instance running Asterisk and Nagios) so I am a bit baffled by that.  A almost instantaneous drop in power dropped two servers, both of which had UPS’s attached (one of them with TWO UPS’s attached).

So the day(s) stood still while I diagnosed and fixed dropped and hacked servers.  So much for productivity increases due to technology.

This entry was posted in Uncategorized. Bookmark the permalink.