Cloudflare Outage Downs Hundreds Of Websites

Tom Jowitt,
Oops computer error - Shutterstock: © alp33

Not a cyberattack. Swathes of the Internet knocked offline on Tuesday after ‘service issues’ at infrastructure provider Cloudflare

Internet infrastructure provider Cloudflare has resolved a problem that resulted in widespread online outages on Tuesday.

The ‘service issue’ actually resulted in hundreds of websites being knocked offline, including Discord, Omegle, Shopify, NordVPN, and National Rail Enquiries to name a few. Users of Microsoft Outlook and 365 also reported delays and connection issues.

Cloudflare is a US content delivery network, and its services include helping firms fend off denial-of-service (DDoS) attacks, which typically involve overloading a site with large amounts of junk traffic.

Cloudflare explanation

The firm admitted the problem on Tuesday in a series of tweets, saying that the outage impacted traffic in 19 of its data centres.

The firm also described what happened in a blog post.

It said that unfortunately those 19 data centres handle a significant proportion of its global traffic.

“This outage was caused by a change that was part of a long-running project to increase resilience in our busiest locations,” it said. “A change to the network configuration in those locations caused an outage which started at 06:27 UTC. At 06:58 UTC the first data centre was brought back online and by 07:42 UTC all data centers were online and working correctly.”

“Depending on your location in the world you may have been unable to access websites and services that rely on Cloudflare,” it added. “In other locations, Cloudflare continued to operate normally.”

“We are very sorry for this outage,” it concluded. “This was our error and not the result of an attack or malicious activity.”

The blog then went to provide more detail of the issue, which lasted approximately two hours.

Decentralisation needed

The fact that a configuration problem at one American cloud service provider could have such an impact on the Internet has been noted by an industry expert, who warned of over reliance on a select few service providers.

“The modern internet is approaching its 40th birthday,” said David Warburton, threat research lead at security specialist F5 Labs. “Despite its age, it still proves to this day how well it was designed, with layers of resilience and redundancy.”

“In fact, the web as a whole was intended to be decentralised,” said Warburton. “By not relying on any one central system, it meant that many different components could fail, and internet traffic could still find a way to get where it needed to go.”

“What we’ve seen over the past decade, however, is the unintentional centralisation of many core services through large cloud solution providers, like infrastructure vendors and CDNs,” said Warburton. “We can think of these cloud solution providers as the supermarkets of the web. Many of us appreciate the ease of buying groceries from one large store rather than visiting a dozen different ones on the high street.”

“Similarly, these cloud solution providers deliver many benefits, such as simpler application deployment, reduced management complexities and economies of scale,” said Warburton. “In a traditional internet app deployment model, an outage of a server or misconfigured application might take out a single website.”

“As we saw today, similar problems with a cloud solution provider can end up taking out all of their customers, resulting in not one website being taken offline, but hundreds or thousands,” he said. “The impact can potentially affect organisations’ digital experiences, revenues and reputations.”

“Cloud solution providers provide immeasurable benefits to their users, but we shouldn’t forget the lessons of the past,” said the man from F5 Labs. The “re-centralisation” of the internet through these cloud solutions is now causing the very problems the original design of the internet was intended to avoid through redundancy. It’s important we consider an approach that moves us away from single points of failure or we will likely see more issues like we did today.”