RootMetrics: Operators Can’t Discredit ‘Scientific’ Mobile Network Tests

Choosing the right mobile network can be hard, especially when you want to find out which will provide the best coverage, speed and reliability at home, at work and in between.

All four UK operators make regular boasts about their service and infrastructure improvement plans, but they would, wouldn’t they? Ofcom has stepped in to try and provide some impartial advice through performance tests and complaints figures, but often, it is up to consumers and businesses to do their homework.

RootMetrics, a Seattle-based firm, has been doing network tests in the US since 2008 and has moved to the UK, providing reports for major cities and venues for the past few years. It has since grown to 183 staff, including 32 data analysts and statisticians.

It assesses networks using ‘RootScores’, a weighted metric, rather than pure stats in a bid to make them easier to understand, but operators (mostly the ones that don’t do too well) have disputed the firm’s findings and methods.

Scientific testing

But the company says its scientific approach gives the best indication of how well a network will perform, giving consumers the chance to make an informed decision and networks the opportunity to improve their service.

“Fundamentally what we do is empower the consumer with scientific information,” Scott Stonham, RootMetrics’ director of international strategic accounts, told TechWeekEurope. “We go about our testing in a way that, on the one hand is incredibly scientific so it can stand up to scrutiny and can be repeated, but on the other hand is distilled in a way consumers can understand.”

It is this scientific approach that separates RootMetrics from crowdtesting sites like Open Sure Signal and subjective advice that you might find in Carphone Warehouse.

“The most accurate way to find out the characteristics of a population is to use random sampling,” he explained. “However if we were to choose random samples by throwing pins at a map, there’s a good chance we’d find somewhere people don’t care about. People don’t make phone calls in the middle of fields.”

“We take a list of places and rank them by population density to make sure that when we apply our random testing, we do it in places where people spend most of their time because we’re trying to give people information that’s useful.”

Test process

In the UK, RootMetrics takes the 16 most populous ‘large urban zones’ to conduct its test, claiming these cover 50 percent of the population. For example, the London zone includes places like Slough, meaning test results can cover home, office and commute.

Each tester is given two ‘scouter kits’ – one which stays in the car at all times and another which is carried around by the tester – comprising the same two smartphones for each network. The smartphones come from the operators themselves and nothing is changed apart from that a RootMetrics app is installed.

“That application is there to instruct the phone to ‘do things’ – make and receive phone calls, send and receive texts, to download and upload data – just like consumers do,” said Stonham. “Right now our test sequence is seven and a half minutes long.”

“Each one of these phones is ready to go and we have another device that makes sure each of these devices is carrying out the tests at exactly the same time. With the scientific principle, if you’re not testing at exactly the same time and place, you can’t compare.”

Helping out networks

Call, text and data performance is then converted into a ‘Rootscore’ – a metric which is determined by the firm’s analysts in Seattle. The criteria changes as networks improve and consumer demand evolves, so an average speed of 7Mbps a few years ago might have resulted in a high mark a few years ago, but is considered poor in 2015. Individual Rootscores makeup the overall Rootscore, with individual elements weighted according to perceived importance.

“At the moment, call is weighted at 40 percent, mobile Internet is 55 percent and text is 5 percent,” said Stonham. “In the previous year, it was 45 percent, 45 percent and 10 percent, so we understand how things are changing.”

Operators are ranked by individual categories and overall and the findings are made public. But these reports are just a ‘small slice’ of the information acquired by RootMetrics. Much more detailed information is provided to operators and other bodies, such as financial institutions. Stonham says this helps operators identify issues they didn’t know about or thought weren’t affecting service.

While all operators carry out internal testing, this is often done by separate teams within an operator who “rarely” talk to each other or present it in a usable away, he said, and so patterns are more difficult to detect.

“I think operators are getting much better at it and you can see that in how quickly things are rolling out and how things are improving, but its silos that are the problem.

“We spot things that they were totally oblivious to because siloed operations didn’t alert them. If we can point out something that has a measurable impact on the consumer experience that they were completely blind to, that’s fantastic.”

Settling operator disputes

So if RootMetrics is helping the operators, why are some so quick to dismiss it? Three, which has improved significantly in the rankings, says it prefers YouGov and other tests that take into account other things like customer service, while O2 – which is currently rated the worst operator – complains RootMetrics isn’t sharing enough data and its tests aren’t accurate.

T-Mobile in the US also has its grievances, with its colourful CEO John Legere reportedly calling RootMetrics’ rankings “bullshit”

“RootMetrics doesn’t actually do testing that replicates what customers are doing,” said O2 chief operating officer Derek McManus. “I feel RootMetrics has had its day.”

Stonham says everything RootMetrics does is for the consumer and that if its statisticians and analysts in Seattle believe there are too many anomalies that can’t be explained, testers are told to start over again.

“We do everything from the consumer point of view and what consumers would do on the devices and networks they use, in the places they would be in a purely scientific basis and show the operators, warts and all, in the best possible light.”

How much do you know about UK mobile operators? Try our quiz!