BennyEast.Com/Blog The official blog of Kenny West

In-ternet, Out-ternet

The internet went out at work today.  I guess one might call it an outernet instead of an internet.  It's rare but it does happen.  Ideally you'd have a secondary or load balanced internet gateway from a secondary provider so that when it goes down on one of the fiber connections, the other one can handle the traffic.

Unfortunately, this is expensive.  So... you could spend an extra 5 or 10 grand a month on a second internet pipe.... Or you could just deal with a 3 hour downtime once in a blue moon.  Actually today was the longest we've ever seen for downtime from our ISP.  Usually it's never like this.  This was a major outlier as far as ISP outages go.

Now, our provider promises something like a 99.95 percent uptime per year.  Assuming a 365 day year... Unless you're in a leap year... You'd have 24 hours a day times 365 days... or 8760 hours.

The last time we lost internet had to have been at least more than a year ago.  Maybe two years ago?  I can't actually recall when we lost it last...

So let's assume this was the first time in a year that it went down.  We had 3 hours down....

99.95 percent of 8760 would be 8755.62.

I'm gonna give them some slack... Ok so a half hour a year is their promise.  I guess they blew it by 2.5 hours.

However today was a pretty serious downtime thing for them.  It actually took their main website offline and as well it took down their phone system, so we couldn't even call them .  At first we thought they had gone out of business.  Just packed up shop and left everyone in the dark.  We tried to call but none of the phone numbers worked.  We've NEVER experienced this before.  We kind of freaked out a tiny bit when this happened.  Mini panic attack.  We also tried to go to their site from an alternate internet connection.  We actually have a secondary Comcast connection that can be assigned to specific ports around campus.  However... it doesn't provide local access to any of the shared network resources.  It acts as an off campus network for things like Skype calls, or it's for testing purposes.  It wasn't practical to pipe that network through to the rest of campus.  There are various other reasons why we can't use Comcast for load balancing/redundancy.  I'm not going to get into that in this blog.

With their phone system and website offline... I'm assuming it was a pretty serious issue.  Eventually we got through to their NOC via phone towards the end of the downtime, once we had already found out what was going on.  So... To me at least, it just doesn't seem worth it to pay like an extra 50 or 60 grand a year for double the bandwidth connection if you might have an extreme outage of 3 hours a year only once every few years.  Most likely the most you'll have is perhaps 20 minutes a year.  That seems acceptable.

I guess it depends on the industry.  In education we cancel work for snow days.  So obviously things are a little bit less pressing and classwork can be rescheduled as homework and such.  If you can't show a YouTube video in class one day... You can always show it the next class.  Or just email the students the link for them to check at their own leisure.

That's the academic atmosphere.

When the internet connection initially went down and we couldn't contact the provider, and also couldn't go on their site.... I mentioned to my boss and coworker that we could try to find alternate forms of communication.  My first idea was... Just search Twitter!  Or their Facebook page.  Or any other one of their social media sites.  They probably would have posted something to social media about the outage.

Nothing.

So then I decided ok, if the company itself isn't going to post system status updates about downtime... Maybe, if OTHER people are also having an outage, they PROBABLY are tweeting about it.  So I started keying in different search strings.  Eventually I found a couple guys @ replying back and forth about how they were also down.  One guy kept @ mentioning the ISP's Twitter account handle name, but was getting no response from them.  He just kept asking ETA for restore.  Then a lady from Chicago eventually replied to one of his tweets saying to him she was down too at her workplace.

Then another guy from Northern NJ chimed in.

Then a guy with a pepperoni pizza cover photo started mentioning things on his and another guy with a Star Wars cover photo also started to Tweet about it.

We found out that one of the other Twitter users finally got through to NOC on phone and  were being told that they had identified the issue and one of the techs was 10 minutes away from the problem site.  They didn't have an exact ETR (estimated time to restore) but it shouldn't be much longer.

Suddenly seeing the pepperoni pizza cover photo again I got hungry and decided to eat lunch.

While eating lunch my boss asked me how I knew all this stuff.  I told him...

"Eh, an ex girlfriend and I used to stalk each other's Twitter accounts.  So, I've gotten pretty good at stalking social media to find out information about things."

Well there ya go!  I can't say I would have ever thought that stalking ex lovers on social media would be something I could list as a useful workplace skill set... But it seems to have come in handy today.

We were then able to convey the information about the status of the network to the rest of the college community.

So, this blog post goes out to the ex that taught me my savvy social media sleuthing skills.  You saved the day today and you don't even know it.