On July 19th, 2024, what started as a routine software update from cybersecurity giant CrowdStrike turned into a digital nightmare. The update, meant to bolster defenses, ended up causing a major malfunction that crippled an estimated 8.5 million Windows machines around the world.
This incident triggered widespread chaos across various industries. Airports faced gridlock as critical flight management and ticketing systems went haywire. Grocery stores struggled to process payments as checkouts malfunctioned, leaving frustrated customers in long lines. Even media outlets weren’t spared, with journalists left scrambling as their computers crashed, hindering their ability to report on the unfolding crisis.
Incidents like this show the importance of being prepared when your technology is not working. Whether it’s an issue with a vendor software update or a full-blown cyberattack. Does your business have a plan in place so that you can continue to operate in these situations? Let’s start with background on the incident and then cover what you need to take away from it.
What is CrowdStrike?
CrowdStrike is a leading cyber security company, founded in 2011 and based in the United States. Essentially, they act as digital bodyguards for businesses and large organizations, protecting them from cyber threats like ransomware, malware, and other online attacks.
CrowdStrike is trusted by a wide range of businesses, including more than 500 companies from the Fortune 1000 list. They have a solid reputation for responding quickly to cyber threats and have been involved in investigating major cyber incidents.
Their main product is called the Falcon sensor program. This cloud-based security system is designed to detect and stop cyber threats in real time.
What is the Falcon Sensor?
Think of your computer as a house. Regular antivirus software is like a security system that looks for specific types of bad guys (like burglars) that it recognizes from before. If it sees any of these known bad guys, it stops them from getting in.
Falcon sensor is something more, called an EDR (Endpoint Detection and Response). It’s like having a smart security guard for your house. This guard not only looks for the bad guys that the antivirus knows but also keeps an eye out for any strange or suspicious activity. The guard can also investigate unfamiliar situations and take action to protect your house, even if the threat is something new.
So, while an antivirus is good at stopping known threats, an EDR is much better at handling new and unexpected threats to keep your computer safe. The trade-off is that EDR requires a deeper level of access.
EDR requires rapid updates to stay on top of quickly changing threats. Unlike other software updates, these can’t be rolled out in stages.
Unpacking the Glitch: A Bug Bites the Watchdog
Let’s dissect what went wrong. On July 19, a routine software update for CrowdStrike’s Falcon sensor caused major disruption for many businesses around the world. The program, designed to be a vigilant defender against cyberattacks, received an update containing a critical error.
This update was intended to improve security by targeting specific tools used in cyber attacks. But the update contained a coding mistake, known as a “logic error.” This “bug” essentially caused the software to malfunction, leading to the dreaded “Blue Screen of Death” on affected Windows computers.
The impact was immediate and widespread.
Many businesses found their Windows computers unusable, resulting in significant disruption. Airports experienced chaos as their systems failed, grocery store checkouts malfunctioned, and journalists faced difficulties reporting on the issue due to their equipment crashing.
The problem affected millions of devices globally. People reported that their computers went into a reboot loop, making it impossible to use them.
CrowdStrike responded quickly. Within an hour of identifying the issue, they began working on a fix. By 5:27am UTC, they released an update to correct the faulty configuration files.
But the recovery process varied. For many, the issue could be resolved remotely by deleting the problematic file if the system was online. For those with offline systems, manual deletion of the file was necessary, which often required help from IT support.
Domino Effect: How Businesses Became Collateral Damage
The impact of this security snafu went far beyond individual computer crashes. Businesses heavily reliant on Windows systems experienced a domino effect. Productivity plummeted as employees were unable to access essential files and communication channels. Call centers overflowed with frustrated customers, further straining resources.
The CrowdStrike outage had a huge impact on businesses across many sectors.
Airports and airlines: The outage led to significant disruptions at airports. Systems that manage flight schedules, ticketing, and customer service were hit, causing delays and confusion. Passengers experienced long lines and delays as airport staff struggled to manage without their usual digital tools.
Grocery stores and retail: Many grocery store checkouts malfunctioned, making it impossible to process sales. This led to frustrated customers and lost sales as stores struggled to operate without their point-of-sale systems. Some retailers had to close temporarily until their systems were restored.
Media and journalism: Journalists and media companies faced major challenges as their computers crashed, leaving them without the essential tools needed to report on the incident. This disrupted news coverage and the ability to provide timely updates to the public.
Banks and financial services: The financial sector also felt the impact, with banks experiencing system outages that affected transactions and customer service. Online banking services were disrupted, leading to difficulties for customers trying to access their accounts or perform financial transactions.
General business operations: Across the board, businesses that relied on Windows systems experienced productivity losses. Employees were unable to access important files, communicate effectively, or perform their usual tasks. Many companies found it difficult to provide customer support as their systems were down. Call centers and online help desks faced increased volumes of queries and complaints, further straining resources.
Healthcare: While not as widely reported, healthcare institutions using affected systems could have faced delays in accessing patient records, scheduling, and other critical operations, potentially impacting patient care.
Lessons Learned: Building Technology and Business Resilience
Overall, the CrowdStrike outage demonstrated how reliable software tools are critical for business operations and serves as a reminder of our dependence on these tools and its interconnectedness with modern business. It highlighted how a single software issue can cause widespread impact for organizations.
So how can businesses build resilience and safeguard themselves from the unforeseen consequences of software mishaps?
Review Incident Response Plans
Organizations must develop and regularly test comprehensive incident response plans that cover a wide range of potential disruptions, including cybersecurity incidents. Effective incident response requires collaboration between IT, security, communications, and other departments to ensure a coordinated and efficient response. A well-defined communication plan is essential for keeping employees, customers, and partners informed during an outage, mitigating panic and maintaining trust.
Your business should test and update this response plan on a regular basis to ensure it stays up-to-date. One way to do this is through a “tabletop” exercise where the response team walks through various scenarios and each responsible person discusses how they would respond to a particular incident.
Review Data Backup Strategy
Regular and comprehensive data backups are crucial for business continuity and disaster recovery planning. System backups should be regularly tested to verify that the process is working correctly.
Your backup plan should include the Bitlocker keys for all of your Windows systems. Bitlocker is the encryption system built into Windows computers to protect your disks from unauthorized access. That key can be required to recover your disk if something goes wrong as in this event or something like replacing a motherboard in your computer. Having that key readily available will help speed up recovery.
Review Monitoring
Many businesses were not aware of this incident until they walked into the office on Friday morning. Put monitoring systems in place to alert key personnel when systems are not working properly. Having this process in place will help technical teams get a head start on resolving issues and minimize downtime.
Supply Chain Risks
Evaluate the security practices of suppliers and partners to identify potential risks. When evaluating new software vendors, conduct thorough due diligence on their security practices, incident response plans, and third-party relationships.
After onboarding a new vendor, implement ongoing monitoring of their security posture, including vulnerability assessments and compliance checks. Your incident response plan should include your vendors to ensure effective collaboration in the event of a breach or other issue.
Vendor contracts should clearly outline responsibilities and liabilities for issues that arise.
Test and Validate Updates to Critical Software Tools
Deploying updates and security patches for your business applications are critical processes to ensure your business stays protected from the latest threats. If possible, apply updates to test systems before installing on your production systems. If that’s not something you have available, have a rollback process in place in the event an update causes issues with your systems.
Cybersecurity Awareness
Cybercriminals will take advantage of major security events to trick employees into “quick fixes” that can compromise your IT infrastructure. They may create fake websites related to an issue to use for phishing or to distribute malicious content. Make sure your employees stay cyber aware through continuous security awareness training to help reduce the risk of human error. Only use contact information from the vendor website to discuss issue resolution.
Cyber Insurance
Cyber insurance can help protect organizations from financial losses due to cyberattacks and outages. If you have a cyber policy, regularly review that policy to ensure adequate coverage for potential risks. If your business does not have this coverage, talk to your insurance agent about protecting your business from these types of events.
How We Can Help Your Business
Many businesses are now reviewing their disaster recovery and business continuity processes. They want to be sure they have clear procedures to help mitigate the impact of future disruptions.
Z-JAK Technologies includes reviews for these recovery strategies as part of our regular managed services agreements. Schedule a call with us to review your current operations or to plan a strategy to make sure your business is protected.
Need help? Call us today at 502-200-1169 or use the contact form to get in touch.