The Story Behind CrowdStrike’s “Largest IT Outage of All Time”

On Friday, July 18th, a global update—codenamed “Falcon”—to cybersecurity company CrowdStrike’s software triggered events that brought major infrastructures and societal needs to their knees… despite affecting less than one percent of all Windows systems.

Now that about one and a half weeks have passed, let’s check in and see what happened, both leading up to this event and in the days since.

Investigations Reveal Why the Bad Software Update Was Pushed at All

To understand the nature of the problem, we must first establish what CloudStrike Falcon is and is meant to accomplish.

In short, CrowdStrike Falcon is a cloud-based cybersecurity platform meant to provide endpoint security and identity management, utilizing AI, automation, and threat intelligence data. Keep the automation part in mind.

For Microsoft’s highly privileged software instances to be updated, these updates pass through Windows’ pipeline execution level, which can easily exacerbate any issues in the software.

This is, in effect, what happened with Falcon. When the Windows device attempted to access the files, Crowdstrike’s tools created errors that ultimately led to an operating system crash.

So, back to why automation is partially to blame: a bug in the part of CrowdStrike’s systems that tests and validates updates before they are released allowed the flawed version of the software through to be automatically pushed to any computer using CrowdStrike that just so happened to be powered on and receiving updates between midnight and 1:27 AM (when the flawed update was discovered and pulled) on July 19th. As a result, an estimated 8.5 million devices—many of which were on and actively being used in Europe and Asia by millions of workers during the workday—experienced the dreaded BSoD (Blue Screen of Death) that could only be resolved by manually deleting the problematic files… no small feat, particularly in its scope.

To its credit, Crowdstrike quickly provided a resource for those impacted to follow… or for their IT teams to follow, at least. 

Naturally, Scammers Are Jumping on the Opportunity

As expected, cybercriminals have swiftly adjusted their tactics to exploit the current issue. Phishing messages directing users to fraudulent “fix” websites have already appeared. Other sites promise to fix the problem, just to wipe your system entirely by replacing all your data with empty files… then announcing their actions over social media.

This makes it critical that you and your team know about these risks and how to avoid them.

The Impact of this Mistake Was Huge

From canceled flights to interrupted emergency services and medical facilities, financial companies thrown into disarray, and trains being paused, this issue (which, as a reminder, only impacted less than one percent of all Windows devices) caused massive challenges for many, many people because so many critical services were suddenly made unavailable.

Now, imagine the fallout if this happened to your organization (assuming it didn’t).

Undoubtedly, you would lose business as you failed to meet your clients’ expectations, and with word traveling so fast nowadays, fewer and fewer people would bother inquiring about your services. While very few people saw this issue coming, it only shows that even the most advanced technology needs to be under close watch.

While a disaster of this scale may have been unavoidable without prior knowledge, you can trust Cambium Data to closely monitor your business infrastructure, solving issues and resolving threats. Ideally, you won’t notice when we do so, either, except that you have fewer IT troubles overall.

Give us a call at (402) 514-3200 to learn more about fully managed IT support services.