Remember last week me talking about how the bandwidth provider for my company decided to be just plain stupid and put the screws to my company, thus ensuring we would hate them forever? Guess what? They weren’t finished.
This morning, as I was walking out the door, literally, with one foot on my patio, I get a call from my boss. It went something like this:
He: “Did the domain transfer screw up our network name resolution?”
Me: “Not that I can think of. Let me check the transfer status.”
He: “Because we have no communication, again. No email, no web, no vpn.”
Me: “It looks like the domain name is still in the previous registrar approval phase. I am not sure what is causing our outage.”
He: “Could you call our sys admin to see if you can figure out what is happening. Right now we have no email, no web, no vpn.”
Me: “Sure thing, let me call him.”
I called my guy and together we came upon something a little different from the last outage last week. We started with our normal troubleshooting techniques: ping, tracert, dig. Very quickly we came to the conclusion that our DNS was not resolving again for some reason. The only difference between this time and the last time was that this time the DNS servers were not responding at all to our requests. I called my boss back and told him that I had given our sys admin the name and number of someone to talk to, even though my attempt at calling him was an epic fail. The phone went to voice mail, but they have not message center so I could not leave a message. Instead I was transferred to the switchboard where I was told that there is no known number and I was hung up on. Nice.
Knowing that my hands were tied at home I told my boss I would be in shortly and began my drive to work. When I got in about 30 minutes later my boss was on the phone. Wanna know with who? You guesses it. Our bandwidth provider.
Now I know you want to know this. What was the cause of this outage? Was it a catastrophic failure of a hard drive? Perhaps a full blown upstream network outage? Maybe a power outage? No. You’re going to love this…
We didn’t pay our bill.
What? Wait, we just took care of that last week didn’t we? Apparently not. See, the email that contained the invoice we were supposed to pay from never sent. Their paper invoice printing system has been offline for a few months so they could not send us a paper invoice. And apparently it was not noted anywhere in our account that we had spoken to their highest level AP person who had assured us that our services would not be terminated again. With that, someone who saw that our account was past due decided to pull our plug all over again.
Are there any further lessons to learn from this? Sure. Try not to be stupid more than once in a given weeks period of time. I tend to think that if that company were a person my boss would have driven to that person’s house and kicked that person’s butt halfway across their bandwidth pipe. Which might not be as far as I think seeing as they are now pretty much nothing more than a name and a phone number. That doesn’t allow messages or actually route to anyone.
This morning I was faced with an extraordinary issue when I got to work. We had no network services to our domain name. And apparently I was the one that was supposed to know why this was.
After looking closely at what was happening, and realizing that emails had not been received to our mail server for a time, and that none of our web sites were running, and that users of our VPN were calling complaining that they could not connect to the network for some reason it dawned on my that it was time to check our domain name servers.
A quick look as it was pretty apparent that our name servers zones were completely missing. After a little more time trying to figure out what was happening, and at the same time trying to vindicate myself from savage disciplinary action due to a recent domain name transfer request that has gone horribly awry, I finally came to the conclusion that our domain name provider (and our bandwidth provider) must have gone out of business last night. They have been on their last leg for quite some time and their services have tanked considerably. I figured this was the last leg of their journey into the toilet of despair.
I was wrong.
After following their horrible excuse of a menu system on their call-in support system I got to their new domain name providers (they have sold off every one of their divisions to other companies so I was actually talking to another company). At the same time I decided to call the guy that has been not helping transfer our corporate domain. While on hold with the new company I actually made contact with our current guy. And I found out something interesting: our zone files in their DNS server were removed because of failure to pay a bill.
Under normal circumstances I would say that this would make perfect sense from a business standpoint. Except for a few points:
- The amount we owed was $225. Our normal monthly bill is about $2,500.
- The last invoice they sent us was in May. For about $2,500.
- The last invoice was for bandwidth and domain name management services.
- Our DNS services were terminated because of an outstanding bandwidth invoice. Bandwidth remained online.
- They never sent a bill for the outstanding amount.
- They never attempted to contact us for the outstanding amount.
- They have been providers of our for more than 10 years.
- We are their biggest account.
That said I would say their handling of this case was an epic FAIL. Worse yet, the one guy that any answers for us (one of four remaining employees with the company) had no access to their systems, billing or otherwise, and was only able to add our zone files back. He was not able to transfer calls, take payments or offer any other insight into what was happening. He couldn’t even give us the phone number of their account person so we could contact her to straighten this out.
Once my director of IT was able to actually traverse their phone support system and get in touch with the woman that handles their billing and accounting it came to light that they never did send us an invoice and that we were apparently supposed to be mind readers when it came to paying for stuff we may or may not have used.
Ultimately the situation ended with my boss receiving confirmation that he would indeed receive an invoice that he could send to the AP department here so they could pay it and get us out of hock. And after a few minutes of downtime we were back on the air.
Now I know there are lessons to learn from all of this. But I think it would have been more useful and helpful if our service providers had taken to learning those lessons before shutting down a US$100M company for a stupid $225 invoice that was, get this, only 45 days late.
Wanna keep your customers happy? Try communicating with them. It would have made all the difference in this case.