Hundreds of services are out of action worldwide thanks to a Windows BSoD bug that has seen airlines, healthcare providers, banks, and more all go offline. While the true cause of the outage hasn't been discovered, it's thought that a company called CrowdStrike is the cause, with the company blamed for pushing out a faulty update. CrowdStrike is used by many businesses worldwide for Windows PC security management.
The issue was first noticed in Australian services as it was the middle of their workday when the update went out, but businesses in Europe are beginning their working days and are experiencing the same outage. Airports globally are experiencing havoc, and Ryanair, one of the biggest airlines in the world, is experiencing an outage relating to ticket booking and check-ins.
On top of that, broadcasters in Australia and in Europe are down too, with Sky News still being off air and Channel 10 and ABC being off air in Australia, too. Berlin, Schipol, and other airports are all experiencing massive delays at present. Even worse, while CrowdStrike has reverted the update and posted a fix, that obviously won't solve the problem for computers that have already installed the update.
11:27 PM PT:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Workaround Steps:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
What is CrowdStrike?
The cause of all the problems
CrowdStrike is a security suite that offers endpoint security and robust protection through its cloud-based Falcon platform. It offers antivirus, endpoint detection and response, and manages threat-hunting services to counteract ransomware, malware, and other attacks. The company has been around since 2011 and has been a major player in the space since then.
The company services are more than just detection, as they provide comprehensive intelligence to help aid organizations in understanding the motives of attackers. CrowdStrike proactively hunts out potential vulnerabilities and threats within a network and attempts to identify and neutralize attacks before they can actually commit any damage.
Because of this update, companies are expecting to need to manually go one by one in updating their servers and machines in order to get them booting again. Some companies can have hundreds or even thousands of machines to go through, meaning that we're still likely hours away from many of these services coming online. The steps involve booting into safe mode and manually deleting a file, as the machines can't be automatically updated to fix the issue given that they're incapable of booting normally.
At present, the update can already be seen to have far-reaching consequences. Hospitals are beginning to cancel appointments, with the NHS's booking system in the United Kingdom going offline as well. It's likely to be a long day of delays and problems as IT professionals scramble to fix the affected computers.
Some flights in the US are grounded
Delta and United are grounded, along with several smaller carriers
Different airlines are dealing with the problem in different ways. While some European airlines like Ryanair have resorted to manual check-ins in airports, U.S.-based airlines like Delta and United are grounded at present. American was grounded but is since back up and running. Globally, more than 1,300 flights have been canceled according to the BBC, and that number is expected to rise as the day goes on. This is a global outage event affecting systems in every country for any business that relies on CrowdStrike software.
Having spoken to friends of mine working in the tech industry, many of them are simply waiting for issues to be resolved. Affected servers and computers need to be fixed manually, as the problem has been caused by a problematic kernel driver that was rolled out as part of the CrowdStrike update. Because of that, machines can't boot to receive a new update, making it so that users have to remove the problematic driver, then boot the device and install the new CrowdStrike update. Machines that use BitLocker encryption will also likely need a recovery key, further complicating the process.
Since the outage, the CrowdStrike CEO released a statement detailing the issue, though it was notably devoid of an apology. He has since apologized on behalf of CrowdStrike. Companies around the globe are still working to fix the damage that's been done, and we're expecting the outages to continue for hours to come. Global payroll systems, medical dispensaries, and more are all caught in the crossfire, with millions of dollars in economic damages expected.