Categories: NetworksWorkspace

Router Bug Crashes Networks Across US, Europe

A bug in a firmware update for Juniper routers may have been responsible for a widespread outage throughout North America and Europe on 7 November.

The bug caused a network failure at backbone provider Level 3 Communications, resulting in service disruptions for Time Warner Cable, Research in Motion’s BlackBerry services, and various Internet service providers based in the UK, among others.

Wide effect

“There are confirmed reports that this event within Level 3 has affected most of North America,” hosting provider Phyber Communications wrote in a status report on 7 November, noting that information was “still limited”.

The culprit appeared to be a problem with Juniper routers that corrupted Border Gateway Protocol (BGP) tables, according to various reports on Twitter. The BGP bug was in the update for the Junos 10.2 and 10.3 firmware on Juniper routers, several members of the North American Network Operators Group posted on the mailing list.

“This outage has affected other networks running Juniper routers with the majority of them seeing their devices core dump and reload,” wrote Max Clark, managing director of Phyber Communications. A core dump refers to when a device crashes and writes the contents in memory to disk.

The router crash would inevitably disrupt network operations within organisations depending on particular Juniper routers for their Internet operations. Level3 Communications is a tier 1 network and one of the key Internet pathways in the United States. When routers went down in Level3 data centres, other organisations downstream were affected.

“This morning, Juniper learned of a Border Gateway Protocol edge router issue that affected a small percentage of customers,” Mark Bahaus, Juniper’s executive vice-president of services, support and operations, said in a statement. “A software fix is available, and we’ve been working with our customers to immediately deploy the fix.”

Bahaus did not provide any details, but a document purporting to be a Juniper advisory posted on the text-sharing site Pastebin claimed the issue was in Juniper’s MX router series.

Failing routers

Level3 posted an update, explaining its outage was due to failing routers, but declined to name a networking vendor or elaborate on the issue.

“Shortly after 9 a.m. ET today, Level 3’s network experienced several outages across North America and Europe relating to some of the routers on our network,” Level 3 said. “Our technicians worked quickly to bring systems back online. At this time, all connection issues have been resolved, and we are working hard with our equipment vendors to determine the exact cause of the outage and ensure all systems are stable.”

Reports of the outage hit Twitter shortly after the routers went down as ISPs and hosting providers took to the micro-blogging site to reassure users. Resarch in Motion assured BlackBerry users that this time it wasn’t to blame for the outage, but “a global Internet issue.”

Hosting provider HostVirtual said the failure at its ISP Level3 seemed to be “related to a rogue route(s) injected into the BGP table or a Juniper bug.”

“We just had a core dump on all our Juniper running Junos 10.3R2.11,” NeoTelecoms, a French telecommunications company, wrote on Twitter.

Fahmida Y Rashid eWEEK USA 2014. Ziff Davis Enterprise Inc. All Rights Reserved.

Share
Published by
Fahmida Y Rashid eWEEK USA 2014. Ziff Davis Enterprise Inc. All Rights Reserved.

Recent Posts

X’s Community Notes Fails To Stem US Election Misinformation – Report

Hate speech non-profit that defeated Elon Musk's lawsuit, warns X's Community Notes is failing to…

1 day ago

Google Fined More Than World’s GDP By Russia

Good luck. Russia demands Google pay a fine worth more than the world's total GDP,…

1 day ago

Spotify, Paramount Sign Up To Use Google Cloud ARM Chips

Google Cloud signs up Spotify, Paramount Global as early customers of its first ARM-based cloud…

2 days ago

Meta Warns Of Accelerating AI Infrastructure Costs

Facebook parent Meta warns of 'significant acceleration' in expenditures on AI infrastructure as revenue, profits…

2 days ago

AI Helps Boost Microsoft Cloud Revenues By 33 Percent

Microsoft says Azure cloud revenues up 33 percent for September quarter as capital expenditures surge…

2 days ago