CrowdStrike Blames ‘Undetected Error’ For World’s Largest IT Outage

Texas-based cybersecurity firm Crowdstrike has provided a detailed update on Wednesday about the buggy update last week that triggered IT chaos around the world.

The firm on its website provided a ‘preliminary post incident review‘ on what is being called the world’s largest ever IT outage last Friday, and which triggered a public apology from CEO George Kurtz.

CrowdStrike blamed the chaos on “an undetected error” within its faulty content configuration update for its Falcon sensor tool, that caused an estimated 8.5 million Windows computers into a ‘Bue Screen Of Death’ reboot loop.

What went wrong?

Crowdstrike on Monday had stated that a “significant number” of affected devices were now back online, but the disruptions have impacted tens of thousands of flights, hospital appointments and business operations around the world.

But on Wednesday the firm offered a more in-depth review of what went wrong.

“On Friday, July 19, 2024 at 04:09 UTC (5.09am British Summer Time), as part of regular operations, CrowdStrike released a content configuration update for the Windows sensor to gather telemetry on possible novel threat techniques,” it stated.

These updates are a regular part of the protection mechanisms of the Falcon platform and happen on a regular basis. But the Rapid Response Content configuration update on Friday 19 July 2024 resulted in a system crash on Windows hosts running sensor version 7.11 and above.

The issue did not impact Mac or Linux based systems.

Earlier this week Microsoft blamed a 2009 agreement with the European Union that prevented it from operating a ‘walled garden’ approach like Apple, and which allowed firms such as CrowdStrike access to the Windows kernel.

CrowdStrike said the defect in the content update was ‘reverted’ on Friday, 19 July 2024 at 05:27 UTC (or 6.27am British Summer Time).

However despite fixing the issue within 90 minutes, the damage had already been done and IT chaos resulted around the world.

“The issue on Friday involved a Rapid Response Content update with an undetected error,” CrowdStrike stated.

The US firm in a section on how it intends to prevent this from happening again, said it would improve Rapid Response Content testing; add additional validation checks; and enhance existing error handling.

It will also implement a staggered deployment strategy for Rapid Response Content; improve monitoring for both sensor and system performance; provide customers with greater control over the delivery of Rapid Response Content updates; and provide content update details via release notes.

Financial, legal implications

But it is fair to say that CrowdStrike is facing a world of trouble in the aftermath of the IT outage.

CEO George Kurtz was summoned to testify Wednesday before the US House Committee on Homeland Security, in the wake of the chaos caused by CrowdStrike’s faulty update.

CrowdStrike shares have dropped more than 20 percent since the meltdown, knocking off billions of dollars in market value.

The BBC noted that CrowdStrike is also facing a backlash after giving customers a $10 UberEats voucher.

The firm reportedly said in an email to staff and “partners” that it recognised the incident had caused extra work.

“To express our gratitude, your next cup of coffee or late night snack is on us!” CrowdStrike wrote, directing people to use a code to access the $10 credit.

Meanwhile the Guardian quoted an estimate from insurer Parametrix that the global IT outage sparked by CrowdStrike’s faulty update will cost US Fortune 500 companies $5.4 billion.

The projected financial losses exclude Microsoft, the Guardian reported.

And there are sure to be many legal implications for CrowdStrike in the months ahead.