CrowdStrike Update Halts the Systems: CyberSpace Ripple Effect

1.CrowdStrike Windows Outage: What Happened?

A recent CrowdStrike update caused severe disruptions, including high CPU usage on macOS systems and the Blue Screen of Death (BSOD) on Windows systems. These issues affected hundreds of systems globally, leading to operational standstills in various sectors, including broadcasting, aviation, and transportation.

2. When Did the Disruption Occur?

The issues became widely noticeable and disruptive on July 19, 2024, following the rollout of recent updates and patches.

3. How Did This Happen?

High CPU Usage on macOS:

• • Software Conflicts: Other installed software conflicts with the CrowdStrike Falcon agent.
  • Outdated Versions: Using older versions of the Falcon agent lacking optimizations.
  • Agent Misconfiguration: Incorrect settings within the CrowdStrike platform.
  • Malware Interference: Malware attempting to disrupt the agent’s operations.

BSOD on Windows:

• • A faulty channel file in the update led to the BSOD, causing systems to crash and services to fail. This significantly impacted services like Microsoft 365 and Azure.

4. Global Impact: Who Felt the Impact the Most?

Users and organizations worldwide were affected, including macOS and Windows users, IT administrators, security teams, and companies dependent on Microsoft 365 and Azure.

Most Affected Sectors:

Broadcasting:

• Sky News: Experienced significant outages and was unable to broadcast. This not only affected their regular programming but also their ability to provide timely news updates to the public.

Aviation:

• Airlines in the U.S.: Several planes were grounded, leading to flight cancellations and delays. The impact was felt deeply by passengers who experienced travel disruptions.
• Edinburgh Airport: Boarding scanners were affected, causing delays and operational challenges.

Transportation:

• Train Services in the U.K.: Train operations were disrupted, leading to delays and operational issues. This affected daily commuters and overall service reliability.

Businesses:

• Global Operations: Numerous businesses experienced significant downtime, resulting in productivity loss. The disruptions led to challenges in maintaining regular business operations, affecting everything from communications to service delivery.

Financial Impact on CrowdStrike

The disruption also led to a significant drop in CrowdStrike’s stock price as investors reacted to the widespread issues. The financial hit reflects the magnitude of the impact and the urgent need for the company to address and mitigate the problem swiftly.

How to Fix Blue Screen of Death and “Recovery” Loop Caused by CrowdStrike

Method 1: Use Safe Mode and Delete the Affected File

1. Boot into Safe Mode:
   • If on the Recovery screen, click “See advanced repair options,” then navigate to “Troubleshoot” > “Advanced options” > “Startup Settings” > “Restart.”
   • After restarting, press 4 or F4.
   • Alternatively, press F8 repeatedly during startup to access Advanced Boot Options and select Safe Mode.
2. Delete the Affected File:
   • Open Command Prompt (Admin) or Windows PowerShell (Admin).
   • Navigate to the CrowdStrike directory: cd C:\Windows\System32\drivers\CrowdStrike.
   • Find the file: dir C-00000291*.sys.
   • Delete the file: del C-00000291*.sys.

Method 2: Use Safe Mode and Rename the CrowdStrike Folder

1. Boot into Safe Mode:
   • Access “See advanced repair options” on the Recovery screen.
   • Navigate to “Troubleshoot” > “Advanced options” > “Startup Settings” > “Restart.”
   • After restarting, press 4 or F4.
   • Alternatively, press F8 during startup to access Advanced Boot Options and select Safe Mode.
2. Rename the Folder:
   • Open Command Prompt in Safe Mode.
   • Navigate to the drivers directory: cd \windows\system32\drivers.
   • Rename the folder: ren CrowdStrike CrowdStrike_old.

Method 3: Use Registry Editor to Block CSAgent Service

1. Boot into Safe Mode:
   • Restart Windows and press F8 until the Advanced Boot Options menu appears.
   • Select Safe Mode.
2. Edit the Registry:
   • Open Windows Registry Editor (Win+R > regedit).
   • Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CSAgent.
   • Find the Start entry, change its value from 1 to 4.
   • Save and restart the computer.

Explanation of Registry Changes

• Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CSAgent contains CSAgent service settings.
• Start Value:
  • 0: Boot start (rarely used).
  • 1: System start (loaded by I/O subsystem).
  • 2: Automatic start (loaded by Service Control Manager during startup).
  • 3: Manual start (requires manual start).
  • 4: Disabled (prevents the service from starting).

Changing the Start value to 4 disables the service, allowing Windows to boot normally.

5. How to Mitigate the Damage?

Details

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
• Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
• Windows hosts which are brought online after 0527 UTC will also not be impacted
• This issue is not impacting Mac- or Linux-based hosts
• Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
• Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version.

Current Action

• CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
• If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

• Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
  • Boot Windows into Safe Mode or the Windows Recovery Environment
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Boot the host normally.

         Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

• Detach the operating system disk volume from the impacted virtual server
• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
• Attach/mount the volume to to a new virtual server
• Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
• Locate the file matching “C-00000291*.sys”, and delete it.
• Detach the volume from the new virtual server
• Reattach the fixed volume to the impacted virtual server

Option 2:

• Roll back to a snapshot before 0409 UTC. 

Workaround Steps for Azure via serial

• Login to Azure console –> Go to Virtual Machines –> Select the VM

• Upper left on console –> Click : “Connect” –> Click –> Connect –> Click “More ways to Connect” –> Click : “Serial Console”

• Step 3 : Once SAC has loaded, type in ‘cmd’ and press enter.

  • type in ‘cmd’ command

  • type in : ch -si 1

• Press any key (space bar). Enter Administrator credentials

• Type the following:

  • bcdedit /set {current} safeboot minimal

  • bcdedit /set {current} safeboot network

• Restart VM

• Optional: How to confirm the boot state? 

  • wmic COMPUTERSYSTEM GET BootupState

For additional information, please see this Microsoft article.

Latest Updates

• 2024-07-19 05:30 AM UTC | Tech Alert Published.
• 2024-07-19 06:30 AM UTC | Updated and added workaround details.
• 2024-07-19 09:45 AM UTC | Updated

6. Threat Actors Set Their Sights on the CrowdStrike Incident

The recent CrowdStrike update has not only disrupted businesses but also attracted the attention of threat actors in the cybercriminal ecosystem. Discussions about the incident have surfaced on dark web forums, where attackers are exploiting the chaos to strategize new attacks. They are actively sharing information and publishing target lists of companies affected by the BSOD and high CPU usage issues. This development underscores the heightened threat landscape as cybercriminals seize opportunities presented by such vulnerabilities to launch targeted attacks, intensifying the need for robust cybersecurity measures.

7. How ThreatMon Can Help

ThreatMon offers comprehensive threat intelligence solutions that can help your organization stay ahead of emerging threats. By providing real-time monitoring of dark web activities and actionable insights, ThreatMon enables you to identify potential risks early and fortify your defenses against targeted attacks. Stay proactive and protect your assets with ThreatMon’s advanced threat intelligence platform and expertise.