When BT go down

British Telecom (BT) are a large if not the largest ex state run communications provider within the UK. In fact, with the exclusion of Hull run the telephone network in the country too. Imagine the joy and excitement that ensues when they experience an issue that effects their core infrastructure.

They get a bad rap. They really do. Some times it is the wonder of being a giant corporate, some times it is just being a bit pants. Their corporate network is behind the majority of the UK’s infrastructure at some point or other.

Yesterday – a morning they had an issue.

It doesn’t happen often, and it presented as failures, and as partial routing failures. The knock on was dramatic.

It took us about two calls to start joining the dots, and speaking with networks they confirmed seeing a 50% drop off of traffic at the BT peering points at 7am … it was 9am

As a result – our morning was mostly spent answer questions regarding “why does my X not work” – and “your service X is broken” – “fix this now” – “its not me its you…”*

(*you dont call, you don’t send me flowers….)

More than just about anyone else I had every confidence that BT would be ontop of the failure / issue, and would be working the solution as quickly as they were able – with their top people on the case. This was in no doubt at all – especially given the severity and wide reach of this particular issue.

As the calls and emails come in its apparent that no one has checked to see if there are any issues with their provider – or having done any diagnostics (keep in mind the majority of our interactions are with developers and ‘the people who look after the IT’):

“Okay, so its BT, not you, okay. How long will you take to fix this?”

“This is not acceptable, the rest of the Internet works … bar amazon and xbox and….”

“This is not but I don’t use BT – I am with [provider X who uses BT for routes and transit]”

… next up they announce they are off to phone up BT.

FOR THE LOVE OF GOD WHY.

They have noticed. They wont be in a situation to say…

“Certainly Mr. Smith we will have that fixed for you in 17 minutes time, and thank heavens you called in – we can prioritize you… in fact, I will get the guy who is heading up the engineering team on the line now so he can give you a break down of what is happening..”

…sometimes you just have to sit back and get on with something that does work. Maybe, at worse, get the grumpy people to check over contracts and SLA’s – short of that – they ARE going to be aware and they ARE going to be on it… and if its a solution driven by budget as opposed to delivery – then chances are you are just going to have to wait. Sorry.

Imagine my deep joy as I hear this this morning:

“We’re hearing of possible power issues within Telehouse North in London which has affected BT’s wholesale and DSL connectivity again today.”

…ground hog day.

I feel for them – I really do – they wont be scrimping on hardware, and it could happen to any of us.

On the up side – I did find the following site which is pretty good for these things of things… http://downdetector.co.uk/problems/bt-british-telecom/map/ … handy.
BToutage

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this:
Skip to toolbar