This morning I was faced with an extraordinary issue when I got to work. We had no network services to our domain name. And apparently I was the one that was supposed to know why this was.
After looking closely at what was happening, and realizing that emails had not been received to our mail server for a time, and that none of our web sites were running, and that users of our VPN were calling complaining that they could not connect to the network for some reason it dawned on my that it was time to check our domain name servers.
A quick look as it was pretty apparent that our name servers zones were completely missing. After a little more time trying to figure out what was happening, and at the same time trying to vindicate myself from savage disciplinary action due to a recent domain name transfer request that has gone horribly awry, I finally came to the conclusion that our domain name provider (and our bandwidth provider) must have gone out of business last night. They have been on their last leg for quite some time and their services have tanked considerably. I figured this was the last leg of their journey into the toilet of despair.
I was wrong.
After following their horrible excuse of a menu system on their call-in support system I got to their new domain name providers (they have sold off every one of their divisions to other companies so I was actually talking to another company). At the same time I decided to call the guy that has been not helping transfer our corporate domain. While on hold with the new company I actually made contact with our current guy. And I found out something interesting: our zone files in their DNS server were removed because of failure to pay a bill.
Under normal circumstances I would say that this would make perfect sense from a business standpoint. Except for a few points:
- The amount we owed was $225. Our normal monthly bill is about $2,500.
- The last invoice they sent us was in May. For about $2,500.
- The last invoice was for bandwidth and domain name management services.
- Our DNS services were terminated because of an outstanding bandwidth invoice. Bandwidth remained online.
- They never sent a bill for the outstanding amount.
- They never attempted to contact us for the outstanding amount.
- They have been providers of our for more than 10 years.
- We are their biggest account.
That said I would say their handling of this case was an epic FAIL. Worse yet, the one guy that any answers for us (one of four remaining employees with the company) had no access to their systems, billing or otherwise, and was only able to add our zone files back. He was not able to transfer calls, take payments or offer any other insight into what was happening. He couldn’t even give us the phone number of their account person so we could contact her to straighten this out.
Once my director of IT was able to actually traverse their phone support system and get in touch with the woman that handles their billing and accounting it came to light that they never did send us an invoice and that we were apparently supposed to be mind readers when it came to paying for stuff we may or may not have used.
Ultimately the situation ended with my boss receiving confirmation that he would indeed receive an invoice that he could send to the AP department here so they could pay it and get us out of hock. And after a few minutes of downtime we were back on the air.
Now I know there are lessons to learn from all of this. But I think it would have been more useful and helpful if our service providers had taken to learning those lessons before shutting down a US$100M company for a stupid $225 invoice that was, get this, only 45 days late.
Wanna keep your customers happy? Try communicating with them. It would have made all the difference in this case.