I’m one of the admins who manage CrowdStrike at my company.

We have all automatic updates disabled, because when they were enabled (according to the CrowdStrike best practices guide they gave us), they pushed out a version with a bug that overwhelmed our domain servers. Now we test everything through multiple environments before things make it to production, with at least two weeks of testing before we move a version to the next environment.

This was a channel file update, and per our TAM and account managers in our meeting after this happened, there’s no way to stop that file from being pushed, or to delay it. Supposedly they’ll be adding that functionality in now.

Yes, CrowdStrike says they don’t need to do conventional AV definitions updates, but the channel file updates sure seem similar to me.

The file they pushed out consisted of all zeroes, which somehow corrupted their agent and caused the BSOD. I wasn’t on the meeting where they explained how this happened to my company; I was one of the people woken up to deal with the initial issue, and they explained this later to the rest of my team and our leadership while I was catching up on missed sleep.

I would have expected their agent to ignore invalid updates, which would have prevented this whole thing, but this isn’t the first time I’ve seen examples of bad QA and/or their engineering making assumptions about how things will work. For the amount of money they charge, their product is frustratingly incomplete. And asking them to fix things results in them asking you to submit your request to their Ideas Portal, so the entire world can vote on whether it’s a good idea, and if enough people vote for it they will “consider” doing it. My company spends a fortune on their tool every year, and we haven’t been able to even get them to allow non-case-sensitive searching, or searching for a list of hosts instead of individuals.

Speaking as someone who manages CrowdStrike in my company, we do stagger updates and turn off all the automatic things we can.

This channel file update wasn’t something we can turn off or control. It’s handled by CrowdStrike themselves, and we confirmed that in discussions with our TAM and account manager at CrowdStrike while we were working on remediation.

Can’t speak for the person you’re replying to, but I’m a security engineer and stuff still makes its way to me that you would think would get filtered out by others (and isn’t my job to fix). It just takes the right person thinking “this is obviously a problem with $system, let’s just send it straight over to them so they can fix it quickly!” And then we get the fun job of proving it’s not us and has no relation to us.

We got a ticket today for packet loss between two systems, neither of which have any of our tools on them…

That’s great! Like I said, it’s dependent on your employment contract. But for people who aren’t as certain, separate work and personal devices as much as possible just to protect yourself.

Depending on where you work, your employer may be able to take that personal device you’re using for work in the event of a lawsuit against the company (where they need to retain anything that may be relevant to discovery), or in the event of a security incident (where they may need it for forensics).

I work in information security, and I practice strict isolation for that exact reason. Two laptops, two phones, because if anything ever happens they can and will take devices for analysis or evidence. If you are using an issued device, they’ll assign you a new one; if it’s a personal device you’ll get it back when they’re done with it, which could take years.

Edited to add this is dependent on your employment contract, but it’s better to be safe than sorry. Cover your camera and use your work computer.

I’m a security engineer, and encryption is great, but can be bypassed. Relying on encryption assumes it was implemented properly, that the system was shut down properly so all keys were flushed correctly, and the encryption algorithm doesn’t have weaknesses.

Generally if somebody dedicated enough can acquire physical access to a system, they can probably find a way into it given the right resources. Did that happen here? Probably not. Could it have? Absolutely. That’s why most enterprises or government hard drives are shredded rather than just relying on them being wiped or encrypted.

Encryption is part of the solution, but it’s not automatically the complete solution.

We don’t know what was on those servers, but it was apparently sensitive enough that the government redacted descriptions of the data in court filings.

The US government brief said the relocated servers were not wiped before being moved to a new data center. The type of data on the relocated servers was apparently so sensitive that it could not be described in the US court filing, which redacts the sentence that describes what the servers contained.

https://arstechnica.com/tech-policy/2023/09/us-government-slams-musk-in-court-filing-describing-chaotic-environment-at-x/

Yeah, policy is not made by economists. Which is both good and bad! Ideally you don’t want somebody who only looks at their own corner of the world running things, because they’ll end up sacrificing everything else to make that corner look good.

It’s more “if people quit trying to break the system to enrich themselves, and the politicians actually agreed to empower the agencies which are supposed to oversee and regulate large companies and financial institutions, and we actually listened to the data instead of the soundbites that sound good as long as you don’t think about them much, we’d be much better off.”

Economists are not in charge of anything, politicians and rich people are. And they aren’t incentivized to run things like an economist, because then they would make less money.

Just because the people with an incentive to blow up the economy to make money end up blowing up the economy to make more money every few years doesn’t mean economics is at fault for that. It’s like saying climate science isn’t real because earlier projections of global warming were more optimistic, when the real reason is the science was suppressed and downplayed by the people making boatloads of money off fossil fuels.

We have a fairly solid understanding of an ideal economy. If the economy was run according to current theory, we’d avoid a lot of issues (and find new ones we would address, of course).

However, the economy is run according to political whims, so most of the economic theory gets thrown out the window. It’s pretty easy to run into major issues when nothing stays consistent for more than a couple years, and the interests of those in charge do not include a stable and sustainable economy.

Willow looks like it’s a decent chunk of the way there, as soon as the hardware is more generally available.