How Losing Our ADSL Broke Our Websites

This is a cautionary tale, ladies and gentlephones, one which begins either late Friday night, or early Saturday morning, a few weeks ago, the exact timings are unclear. It may not be the most exciting story, nor is it the sort of revolutionary epic which inspires a new generation to rise against the hypocrisy of its government and reinvent politics of its time. It is however entirely true, and living through it was bloody annoying.

First a recap, at the beginning of September we began moving our websites from our own ageing servers which sat in a rack at TelecityRedbus in Docklands, to new rented machines at UKFast. We managed to migrate all our email, DNS and websites to the new machines with very little problems. So far, UKFast have been pretty much what every IT person wants from a hosting company.

Our actual domain names have been hosted with Easyspace for a fair few years, we originally had our main one on Demon but moved it to Easyspace after having problems with Demon's hosting of it. I seem to recall that whole fiasco peaked on a Friday with someone on their tech support promising to do something domain related, and then sodding off home for the weekend instead.

The Day We Lost The Internet

So this brings us up to about Friday last week, when we had two DNS servers running, our primary at UKFast, and a backup secondary running on my ADSL line at home. Everything seemed to be fine, until that is local contractors took out a few hundred telephone lines in my local area at the weekend. We have a lot of phone lines into this place of work, two of them were completely dead, one of those carried our Business ADSL connection.

What is a Business ADSL connection? Well generally one considers paying more money for the same thing as a domestic connection, but with the word Business in front of it, means if things are broke then they need to get fixed sooner rather than later. Needless to say this was also on a Business phone line, so you'd think the worst case would be 24 hours of downtime?

Think again! The damage done, cutting off hundreds of local people, would take best case 7 days to fix, but may even take up to 10 days. That's 10 days without internet access, the phone lines we rely on for our businesses, thank you very much BT, thank you very very much.

But our contract is with Demon, we still have other working phones in the house, maybe they can help us? Well the best they initially offer is to move our ADSL to another phone line which will take 7 working days. How is that any help? After talking to 5 different people for quite a long time, eventually they have a better offer, they can do it in 3 days but we have to pay about £150 and get locked into a new 12 month contract with a company we've been toying with leaving for the last few years because their support has gotten so bad.

Time to change ISPs, so long Demon, you used to be good (apart from the domain incident) and we've been with you since our very first leased line all that time ago (15 years is it?), but now your support has become so terrible and unhelpful if any issue pops up I despair at having to ring them.

A little tip for any budding ISP support people out there, if the person ringing you up is IT literate, do not repeatedly ask them to ping microsoft.com or power cycle their router when they ring up to report a line fault. And even if you insist they ping something for you, don't tell them exactly how to open the start menu, and browse for the command prompt.

So I ring up Zen, they can also do it in 3 days for an additional £150, which appears to be BTs standard charge. Since it was BT who are responsible for us having to do this in the first place, it seems BT will refund us that charge at a later date so all is good. Incidentally, a few friends of mine use Zen and find their connection excellent for gaming, and their support top notch.

Zen must be doing really well, because both times I ring the sales line they answer with a mere hello, no mention of the name of the person I'm talking to or any offer of how they might help me. Someone send these people on a training course, I wasn't even sure I'd gotten through to sales when they picked up. Unless they really are doing so well they can afford to be so nonchalant towards new customers.

So How Does All That Kill a Website?

A good question I hear me ask myself, what has websites hosted in Manchester got to do with our ADSL in London? Well, come Monday morning I'm reading threads on the photography forum my World of Warcraft guild (which includes many good friends) uses. It seems some people can't see a picture I posted that is hosted on our new servers.

Well that's odd, so I check the servers everything appears to be up and running. But also something even worse is apparent, I've been checking the traffic stats on Google Analytics almost daily the last week or two, checking to see if recent changes we made to our sites for search engine optimisation are having the effect I intended. Since Sunday our traffic has dropped to nearly zero, what the hell is going on? This is all I need, not only am I struggling to access the internet with a make shift 3G dongle due to no ADSL, trying to sort out our ADSL, now I have this to deal with?

I check the servers, they all seems fine, I get friends to check them and they have trouble resolving the domain names. It's a DNS issue, I bloody hate DNS issues, mainly because they can be troublesome at the best of times. Looking back, here is what I think happened...

The root domain servers think we have two domain name servers on ns0.reviewer.co.uk and ns1.reviewer.co.uk, despite me changing them a few times on Easyspace (who host our domain names) to ns1 and ns2. Now ns1 is currently at UKFast, with ns2 being on my ADSL at home, the latter obviously down and out of the game at this point.

However that should still be okay, the secondary backup server on ns2 is just that, the backup server. ns1 should still be accessible. At least it would be, if the root domain servers really thought that ns1 and ns2 were where they should be. It turns out that Easyspace's system for changing them may be buggy, or perhaps there is some other mystical process here I don't understand.

These root servers had wrong information for the reviewer.co.uk domain, which is the most important we own because not only does it host our most high profile and high traffic websites, primary email addresses, and so on, it also contains the ns1 and ns2 entries that all our other domains (such as MyReviewer.com) rely on for their DNS.

They believed that the name servers were at an IP we used in our racks at docklands (now down because of the move) and a backup on my ADSL at home (also down) and no amount of trying to change them with Easyspace's control panel would fix it. It seemed for the last few weeks, the ONLY name server which was working sat on my ADSL line. So even though at some point in the future I would have noticed and fixed this problem, for now the internet thought our backup DNS server was our only server, and death of our ADSL killed that.

As our domain data on name servers around the internet started to expire, traffic began to drop, amazingly one single person from our forums still resolved the old addresses. I don't know why or how, but he had become the Omega Man.

So anyway I sent a critical support ticket in, which takes them bloody hours to even reply to, after which they just send a mail asking me to confirm what I was asking them to do. I'm sorry guys, but when you are losing emails, website traffic and money, you do not want to spend 6 hours waiting for somebody failing to help you.

At this point I took the only sensible option available to me, bearing in mind it was now past 1am. I forked out £15 to have Easyspace transfer the domain to UKFast, despaired at the fact this could take 3 business days to happen, and went to bed. When I woke up, the domain transfer had gone through, and I could use UKFasts system to enter one correct nameserver entry. Minutes later traffic began flowing to the site, mail started appearing, and all was once again well.

Now what have I learnt from those few days? Nothing new unfortunately, but I can offer the following advice. If you have a service that you care about, make sure you are responsible for as much of it as you are competent enough to be responsible for. Then make sure anything out of your control is looked after by people who care, and you have some way to contact them.

This Week's Videos

Been a while since I did some of these, so here we go! First up The Axis of Awesome 4 Chords, which is a good lesson in how diverse a selection of music can originate from the same four triads.




And as I like to have related vids in these updates, here is the other side of the music coin, good old fashioned dancing. What better reason to dance for than Jesus?