Operators Opinion

Do O2’s recent outages highlight the need to have two or more operator accounts for redundancy?

Now that the panic has subsided and those suffering from total disconnection thanks to O2’s recent outages are back online, I wondered if it was time to point out the need for redundancy.

In today’s always-on environment, is it good enough to simply stand staring at the wall whilst your phone doesn’t work?

I was aghast at the RIDICULOUS Twitter messages being published by all manner of celebrities, famous folk and nobodies. The press were having a field day weren’t they? Furious O2 customers from “important” TV anchors to members of the governing classes took to social media to express their outrage.

I couldn’t help but think they were all crazy.

Running a system as complicated as a mobile network isn’t child’s play. They spend billions on infrastructure, monitoring — the whole shebang. But it’s a very delicate thing. Yes there’s redundancy and backup built into everything — they don’t call it “carrier grade” for nothing — but problems happen. You can often plan for them, but there’s not much you can do when someone takes a JCB to one of your critical buildings (See Vodafone’s partial outage last year). 

Not many network assurance teams consider JCB thievery when they’re building redundancy. 

But problems happen.

And when they do in today’s world, it makes for a fascinating experiment. 

Just how reliant are people on their mobile phones?

And just how quickly does business come to a halt when you’re unable to communicate ‘personally’ for a length of time?

What’s astonishing is that there wasn’t a real problem. It was an inconvenience if you were an O2 customer unable to connect to their network. But Vodafone, Three, Orange & T-Mobile were perfectly fine.

Of course the vast majority of normobs (“normal mobile users) are accustomed to only having one phone. On one network. Even if they went out and bought a SIM card, that wouldn’t really help. Because it would have a different number. And that’s the sticking point for most people. Their phone number. That’s the one element of control the mobile operators still have and it’s a huge, huge barrier.

The legions of complaining celebrities were hardly likely to pop down to their nearest Carphone Warehouse and pick up a temporary sim — when this would have given them connectivity immediately. But no one would know it was ‘them’ calling. No one would be able to call them. They wouldn’t be able to (easily) pick up voicemail (checking your operator voicemail from a landline is easy but beyond most folk). 

For you and I, dear reader, the situation is different, no?

According to the last survey we did there, the bulk of Mobile Industry Review readers have 2+ operator accounts. For us, this occurrence would have been a minor inconvenience. I routinely use three phones. My main number diverts to the office if I can’t answer it, and the helpful ladies take a message and send it to me immediately via email. So I don’t miss a thing. I’ve got Hullomail running on another phone that I use for my consultancy work. That neatly separates things and also ensures that I can access (and get notifications about) the voicemails on any device I’m using — desktop, iPad, whatever. 

If you’ve spent a bit of time fiddling with your iOS devices, the chances are you’ve unified your iMessage communications to use a common email address (or similar) so that, actually, you’re more or less device independent for your messaging. 

What’s your device and service provider strategy? Would you be seriously hampered by your primary operator going down for a length of time?

By Ewan

Ewan is Founder and Editor of Mobile Industry Review. He writes about a wide variety of industry issues and is usually active on Twitter most days. You can read more about him or reach him with these details.

18 replies on “Do O2’s recent outages highlight the need to have two or more operator accounts for redundancy?”

Email is so reliable that I get a lot of ‘urgent’ messages that way. Maybe it’s the norm in my company, I suspect other orgs don’t expect 24/7 email monitoring and no, it shouldn’t be relied upon. But hey.

I’d be far more concerned about loosing email than voice. What I do routinely carry is a second *data* device, be it MiFi, iPad, laptop etc. I guess I know I can always borrow someone else’s phone, use a landline or – do these still exist – a payphone. But my email is me only, and in a corporate world with VPN-only access it’s locked to a few devices. No webmail for me.

All carriers fall over and go plop, but the degree in which the megalith gsm provider engages its customers, instead of dead silence, and denials, proactive consumer interaction. Also for contract customers, little gifts like a months free service or 50% off for the total failure of the network would at least in some way show and slow the appeasment. Making the down time into an up time positive story is not hard. Just takes someone with 1/2 a brain. UK Carriers are far to complacent to seem to really care about its customer base. Does being by telefonica spain have any influence on its dont give a fuck about customers attitude?

I’ve got O2 for work calls (wasn’t affected by outage) and additional O2 sim for data, T-Mobile for personal use and a 3 MiFi lying around somewhere. Lot’s of unused PAYG sims lying around too. I know it’s easy to have lots of kit and sims if you’re in the industry but is it really that difficult to get a new sim even if you’re not? I mean your friendly neighbourhood criminal probably does it quite frequently right?

I often see pix of celebs carrying an iPhone AND a Blackberry – but they’re probably often on the same network. Don’t have any sympathy for celebs, media journos etc who moaned about lack of service. Perfectly easy to buy a PAYG sim and even another PAYG smartphone and get back online. You can buy this stuff at network operator shops, phone retailers like CPW, Tesco/Asda/etc and all manner of other retailers. It’s not exactly specialist kit nowadays is it?

Why not just provide users with the option to ‘roam’ onto other networks, for a small charge, naturally. Of course, when a network is down that charge (to the customer) should be waived by the operator that is having trouble.

It’s not quite true that all the systems in the core network are fully redundant, the HLR which is the database of users and their rights on the network, is effectively a single system, so if this fails, or an upgrade does not work, then this kind of mass outage can occur. There are ways to to have multiple HLRs but it is very expensive, and carries its own risk, and I suspect this is not wide spread, partly due to the risk of the swap out to the more redundant system carrying the risk of a near total outage !
and no I do not know what took out the O2 network (I’m ex vodafone not ex O2)

Would I be hampered? Well, probably not (for a day or two), but many not even for a little longer. Like many others, social media and email are more used for communication than actual phone calls, which all work across multiple devices, multiple networks, even WiFi only tablets, so personally I would simply pick up a different device with a different network SIM and keep on going (actually, as I generally get SIM free devices, I wouldn’t even need to change device).

As for a strategy, for many years I kept my personal mobile on a different operator to my company-provided phone to provide that little extra redundancy for *personal* calls.

What’s interesting is that whilst the O2 outage was going on, I saw many ppl complaining about it via facebook, but I didn’t see anyone say “here’s a temporary number for me”, so maybe they were happy to be reliant upon facebook / twitter as well instead.

You say many people only have one SIM, but on the train each day I see many ppl with two phones (presumably personal / work), or a phone and an iPad, so again, social media is likely to still work even if calls don’t (the others have new Kindles so don’t benefit from social media access via Kindle that Amazon gave us early adopters) 🙂

Given our phones all work fine when we go abroad I don’t see technology being the challenge here; it’s about money (cost to agree terms) and the user experience (“I didn’t agree to you charging me”)

As a former employee of a Network infrastructure firm HLRs can be supplied fully geographically redundant with N+1 sites in whatever config the operator decides and yes it is more expensive, but so is the impact of having your network go down.

In a key upgrade like this the method would be to take one site out (with the 2nd site keeping the network live) – upgrade the 1st site – test it is working fine – then switch live traffic to it. Then take the 2nd site out and upgrade it.

If O2 has such a network architecture and despite precautions above it failed they should be furious at their vendor. If they decided to economise and go for a single site they have only themselves to blame. I know when the giffgaff network (their own low cost MVNO) went down recently the route cause was their single prepaid system got taken out by water damage in the switch building. So perhaps o2 policy is to play it fast and loose and reckon that any reputational damage caused is worth the extra cost of maintaining 5 9s reliability.

When the HLR goes down you can’t roam. The foreign network requires info from the home HLR and without the home HLR you can’t receive any calls.

Some of the custoner comments I heard on the BBC bordered on hysterical. One lady blamed O2 for putting her husband in mortal danger – really??!!

Moving on to more serious matters I was astounded by the number of well established companies that had no contingency or disaster plans. The mobile phone service from a carrier should not be allowed to
become a single point of failure for a business – as was the case for one
comapny shown on BBC News. It’s down to companies themselves to develop business continuity plans and put resources in place to cope with the very real prospect that they will have no landline or mobile or broadband service at some point in a five year cycle lasting from just a few hours to maybe several days.

I have a very low opinion of O2 and wouldn’t return as a customer. But personal feelings aside in this now.
I saw a lot of comments and outrage yesterday towards O2. but I didnt get involved. I think they are all crazy. I have a giffgaff account in a Windows phone. [not affected by outage] My main account is with 3. These things happen, It’s happend with 3. It happened when I was with T. It happend with vodafone. people just need to accept it and not rant and rave at the Operator as they need to foucs on getting it back up and running. Our Networks are Fragile things. Sure it was since midday Wed: thru to yesterday morning with intermittant periods. Its wasn’t just a few hours. SO….. Doesn’t that scream that it was proabably something BIG? use a spare account during an outage, a friends, relatives, whatever. or do something else. dont go on a war path, or demand money back cos that only happens if its been something like three days.
Right Rant Over!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.