I heard that CloudStrike is something that runs on Windows servers, and an error with it caused a bunch of Win Servers to crash. What’s the impact of the issue too?
I’m not a tech person, tho I do use Linux desktop, btw 😉
The correct name is CrowdStrike and it’s basically antivirus monitoring software that many organizations use on their computers/servers to warn of threats and provide protection through blocking the threats.
The issue was that CrowdStrike released an update that basically bricked Windows computers that were on and active. The driver file caused Windows to go into a constant “blue screen of death” that prevented people from logging into their computers/servers.
The main issue with this was the resolution which wasn’t easy. To fix, you had to reboot the machine into “safe mode” which is a mode of Windows that basically disables everything but the most essential Microsoft services and programs. Once in safe mode, you were able to delete the file and then reboot and then the machine would later pick up the newer update that wouldn’t brick your machine. The problem with this is that this had to be manually on each machine. No automated process was found (at the time of me looking at it, anyway) and many organizations have hundreds if not thousands or even millions, in some cases, of machines that this would have to be done on. Not only that, but this also affected user machines even those who work remotely. Imagine trying to walk a user with little to no technical experience through that process because you cannot use any sort of remote software to do this.
Because of all this, it led to mass chaos for many organizations. Turns out a lot of businesses were using this which caused the massive global impact.
They’re saying this is probably the worst global outage we’ve ever experienced as of yet because of this.
It should be noted that CrowdStrike is also compatible for Linux and Mac machines too but these weren’t affected because the bug was only compatible with Windows machines.
Great response, and matches with what I’ve been reading.
Holy shit, someone(s) at CrowdStrike have had a reeeeally bad end of the week!
Agreed. It is odd that CrowdStrike doesn’t seem to?? Please correct me. That they don’t have a test release and a general release.
Weird
It’s definitely weird. They might well throw one out two developers and As under the bus for this event (I’ve seen this happen at other organizations - and such decisions smack of shitty management). But no matter how you look at it this is a company-wide failure. Because they didn’t have the infrastructure and policies in place to test their changes properly.
What a disaster! Thanks for the thorough response 😊
For context, I do not work in anything remotely close to an IT department. I work in a hospital. This affected my work the other day too. I am a bit more tech savvy than some of my coworkers, so I was attempting to see if I could fix the issue on my own by reverting to the previous windows update in recovery mode.
However, doing so prompted me for a Windows product key, which I obviously didn’t have because I didn’t install Windows on that computer.
The IT department had to come around individually for every single affected computer. They had to manually look up and type out the unique Windows product key for every single affected computer in order to be able to fix the problem.
Not sure if most installs of Windows act that way or not, but it definitely made the process more manual and annoying than it had to be. I have no idea why many of the recovery options required me to look up and enter a Windows product key. Seemed very odd to me and just made the ordeal more manual and time consuming than it had to be.
I believe some hospitals even ended up having to cancel surgeries.
Are you sure this was the product key and not a Bitlocker key?
This was another part of the hurdle for the CrowdStrike issue where if the machine required a Bitlocker key, this caused an additional headache for those working to resolve these issues.
Speaking for my team, we didn’t have a lot of machines on CrowdStrike and none with Bitlocker enabled so this wasn’t a problem for us, thankfully.
The Bitlocker key is a key that will show up when a Windows computer is restarted and won’t allow you to actually log into Windows until you can provide this key. You either need to type it or you can use a USB key to have it entered on some machines too. It’s used for encrypting the hard drive and would make sense for a medical office where they deal with sensitive information that needs to stay protected like your work setting.
I suppose that’s possible, but I didn’t see the word “Bitlocker”. Would that not necessarily appear on screen? It just asked for a product key, which I thought was odd.
It wasn’t blocking me from logging into Windows (which would blue screen though). It was instead blocking me from using certain recovery options.
Edit: After some digging, that is likely what it was even thought it didn’t say Bitlocker on the screen. From screenshots, it looks like it occasionally doesn’t say that. Would make sense for security purposes and I’m sure many companies had something like that enabled. It made fixing the whole ordeal a much more slow and manual process though instead of just giving users some instructions!
Also sorry idk who downvoted you!
I was wondering why this didn’t happen to me, and I guess it’s because my company uses Carbon Black which seems to be a competitor to CrowdStrike. Phew!
Thanks for the good explanation.
Intrinsic trust in outside actors in the kernel is a bad idea.
State actors wish they could fuck shit up like Microsoft.
**Clown Strike
To add to these guys, what it looked like in hospitals was all computers going blue screen of death on a loop. You would reboot and it would get to the desktop and go BSOD again. Communications with windows servers also went down.
The problem is, the safety plans are contingent on having windows working. There’s little to no contingent on no windows, people just expect it to always be there.
I and my team and all the individuals we ended up responsible for were fine by morning, but it was not a safe time.
911 was affected. Ambulance dispatch was affected. Many medical based institutions rely on windows and these security systems as well as airlines and such.
That said, please check on grandma, your favorite old uncle, single parent living solo, and all the rest today. There is no telling how many people are dead or injured in homes right now who couldn’t get through to emergency services last night. Maybe still can’t today.
There is no telling how many people are dead or injured in homes right now who couldn’t get through to emergency services last night.
Waooooowww. I hadn’t realized how harmful this issue was. It sounds like CloudStrike’s future is bleak.
deleted by creator
No, the same security has all that shut off for anyone but IT.
Big time AV software used by a lot of conpanies issued an update that crashed all windows machines it was installed on. The only way to fix it is to boot into safe mode and then delete a file. On millions upon millions of servers and pcs. One by one.
Including Mercedes formula1 team lol.
Crowd strike is an antivirus program that is installed on servers and laptops/desktops. The update corrupted a file that caused the operating system not to boot. The impact is thousands of hours of manual labor to recover these servers and endpoints. You have to do it in person unless your user is tech savvy enough to get into safe mode and delete a file. And you need admin rights
deleted by creator
Edit: I got the name (Crowd Strike) and some basic facts wrong. There are other, better answers here.
I’m guessing the 33% vote rate is due to your intro, not your explanations?
I got the name (Crowd Strike actually) and some basic facts wrong, and the intro probably didn’t come off the way I meant it to.
