A bug in a firmware update for Juniper routers may have been responsible for a widespread outage throughout North America and Europe on 7 November.
The bug caused a network failure at backbone provider Level 3 Communications, resulting in service disruptions for Time Warner Cable, Research in Motion’s BlackBerry services, and various Internet service providers based in the UK, among others.
“There are confirmed reports that this event within Level 3 has affected most of North America,” hosting provider Phyber Communications wrote in a status report on 7 November, noting that information was “still limited”.
“This outage has affected other networks running Juniper routers with the majority of them seeing their devices core dump and reload,” wrote Max Clark, managing director of Phyber Communications. A core dump refers to when a device crashes and writes the contents in memory to disk.
The router crash would inevitably disrupt network operations within organisations depending on particular Juniper routers for their Internet operations. Level3 Communications is a tier 1 network and one of the key Internet pathways in the United States. When routers went down in Level3 data centres, other organisations downstream were affected.
“This morning, Juniper learned of a Border Gateway Protocol edge router issue that affected a small percentage of customers,” Mark Bahaus, Juniper’s executive vice-president of services, support and operations, said in a statement. “A software fix is available, and we’ve been working with our customers to immediately deploy the fix.”
Bahaus did not provide any details, but a document purporting to be a Juniper advisory posted on the text-sharing site Pastebin claimed the issue was in Juniper’s MX router series.
Level3 posted an update, explaining its outage was due to failing routers, but declined to name a networking vendor or elaborate on the issue.
“Shortly after 9 a.m. ET today, Level 3’s network experienced several outages across North America and Europe relating to some of the routers on our network,” Level 3 said. “Our technicians worked quickly to bring systems back online. At this time, all connection issues have been resolved, and we are working hard with our equipment vendors to determine the exact cause of the outage and ensure all systems are stable.”
Reports of the outage hit Twitter shortly after the routers went down as ISPs and hosting providers took to the micro-blogging site to reassure users. Resarch in Motion assured BlackBerry users that this time it wasn’t to blame for the outage, but “a global Internet issue.”
Hosting provider HostVirtual said the failure at its ISP Level3 seemed to be “related to a rogue route(s) injected into the BGP table or a Juniper bug.”
“We just had a core dump on all our Juniper running Junos 10.3R2.11,” NeoTelecoms, a French telecommunications company, wrote on Twitter.
Fourth quarter results beat Wall Street expectations, as overall sales rise 6 percent, but EU…
Hate speech non-profit that defeated Elon Musk's lawsuit, warns X's Community Notes is failing to…
Good luck. Russia demands Google pay a fine worth more than the world's total GDP,…
Google Cloud signs up Spotify, Paramount Global as early customers of its first ARM-based cloud…
Facebook parent Meta warns of 'significant acceleration' in expenditures on AI infrastructure as revenue, profits…
Microsoft says Azure cloud revenues up 33 percent for September quarter as capital expenditures surge…