Basically they fucked up and don’t like the criticism from other companies/ customers.
It’s not really criticism, it’s competitors claiming they will never fuck up.
Like, if you found mouse in your hamburger at McDonald’s, that’s a massive fuckup. If Burger King then started saying “you’ll never find anything gross in Burger King food!” that would be both crass opportunism and patently false.
It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.
But also Microsoft fucked up. And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too. Delta shut down, cancelling 4,600 flights. American Airlines cancelled 43 flights, 10 of which would have been cancelled even without the outage.
Like, imagine if some diners at McDonald’s connected their mouths to a chute that delivers pre-chewed food sight-unseen into their gullets, and then got mad when they fell ill from eating a mouse. Don’t do that, not at any restaurant.
All that said, if you fuck up, you don’t get to complain about your competitors being crass opportunists.
Well there’s a provocative anecdote if I’ve ever seen one. Well done.
In what way did Microsoft fuck up? They don’t control Crowdstrike updates. Short of the OS files being immutable it seems unlikely they can stop things like this.
Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.
CrowdStrike pushed an errant update. Microsoft allowed a single errant update to cause an unrecoverable boot loop. CrowdStrike is the market leader in their sector and brings in hundreds of millions of dollars every year, but Microsoft is older than the internet and creates hundreds of billions of dollars. CrowdStrike was the primary cause, but Microsoft enabled the meltdown.
Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.
They have to give that access by EU ruling:
Microsoft software licensing expert Rich Gibbons said: “Microsoft has received some criticism for the fact that a third party was able to affect Windows at such a deep technical level. It’s interesting that Microsoft has pointed out the fact this stems from a 2009 EU anti-competition ruling that means Microsoft must give other security companies the same access to the Windows kernel as they have themselves.”
It’s not really criticism, it’s competitors claiming they will never fuck up.
Not in all cases [podcast warning], sometimes it’s just them pointing out they’re doing silly things like how they test every update and don’t let it out the door with <98% positive returns or having actual deployment rings instead of of yeeting an update to millions systems in less than an hour.
It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.
Clownstrike deserves every bit of shit they’re getting, and it amazes me that people are buying the bullshit they’re selling. They had no real testing or quality control in place, because if that update had touched test windows boxes it would have tipped them over and they’d have actually known about it ahead of time. Fucking up is fine, we all do it. But when your core practices are that slap dash, bitching about criticism just brings more attention to how badly your processes are designed.
But also Microsoft fucked up.
How did Microsoft fuck up? Giving a security vender kernel access? Like they’re obligated to from previous lawsuits?
And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too
Customers can’t test clownstrike updates ahead of time or in a nonprod environment, because clownstrike knows best lol.
Redundancy is not relevant here because what company is going to use different IDR products for primary and secondary tech stacks?
Backups are also not relevant (mostly) because it’s quicker to remediate the problem than restore from backup (unless you had super regular DR snaps and enough resolution to roll back from before the problem.
IMO, clownstrike is the issue, and customers have only the slightest blame for using clownstrike and for not spending extra money on a second IDR on redundant stacks.
Yeah, it was an international fuck up. You’re going to get heat, and it’s 100 percent deserved. Go cry in a corner and fuck off into oblivion.
Cry me a half billion dollar river, maybe we can use that money to fix all the damages it did.
No, its good. They sent out $10 gift cards.
“Our industry is built on trust,” Sentonas said
And instead of following that statement with an apology to all the companies and people they royally fucked in the ass with their shitty business practices, they instead whined about other people pointing out what a massive, colossal, and completely preventable fuckup this was.
Good going sealing my resolve to never use crowdstrike.
Hey, they gave some people an Uber Eats coupon
A $10 UberEats voucher that most couldn’t use
When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”
https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/
So, fraud on to of everything else…? If they do some light treason, it’s a hat trick!
Crowdstrike didn’t just fuck up, they killed people. I personally had to postpone a blood test, but mine wasn’t critical and I’m alive to complain. Not everyone is.
It’s okay, they sent out $10 gift cards for Uber Eats to apologize (that they immediately cancelled).
I don’t see the other companies fucking up so badly though.
You don’t? Shit I see it all the time.
You see CrowdStrike level of bad all the time? Where? What? Who?
damn i haven’t used Windows in over a decade. are y’all ok?
Rofl, like Unix OSes never have problems. Even developers, who are among the most tech savvy users, tend to drag their feet on installing updates unless forced.
i was mostly making a joke about how this absolutely is not a common problem on any platform, not to this degree. and at least when my Arch and Nix systems go down i don’t have anyone to blame but myself. sure, systems have update issues, but a kernel level meltdown that requires a safe mode rescue? that’s literally never happened to me unless it was my fault
Even developers, who are among the most tech savvy users
Doubt
As a sysad, I’ll sign onto this.
Awww, let’s all feel bad for the rich, shitty company that has shitty quality control.
Hey, they’re trying their hardest. It’s hard because they had the joke build stored right next to the actual build so when they went to push it they clicked the wrong one.
CrowdStrike customers ‘unhappy’ with CrowdStrike
Sorry but some fields have no margin for error, just take the L and shut up.
Somebody should call them a wambulance since their fee-fees got hurt.
I do believe the number is still nine-wah-wah
Lol perfect
Stahp, you meany!
Nothing shady about that commentary after seeing how they screwed up. I couldn’t believe how amateur hour the cause of the crash was (the program not validating definition file contents, which spectacularly failed when fed a file consisting only of zeroes). They should rename themselves to ClownTrike.
That wasn’t what was in the file, it was actual stuff. I saved a copy of it.
What happened was the file directed their craptastic snake oil software, which did absolutely no sanity checking first, to access memory it wasn’t actually given which predictably resulted in it crashing, and since its dick was way up the kernels butt at the time they both went down together.
I’ve been calling them ClownStrike because they’re clowns and their incompetence struck everyone else hard.
Yup a null pointer reference for a boot time driver. Which Microsoft never should’ve signed and should revoke. But ya know… Money
Ah, thanks for the clarification on the details. Either way it boggles my mind that they didn’t have checks in place.
I wonder if they’ll end up doing a rename / rebrand if “ClownStrike” continues to haunt them (as it should).
If they do, I’m sure the new name will be some focus tested aberration they pay way too much for.
I appreciated the RiskyBiz episode with the Sentinel one guys where they go over all the ways this could have been prevented if they did real testing
Crowdstrike absolutely deserves the shit they’re getting.
Oh god. Sentinel one is horrible. If they’re taking issue with your testing, you’ve really screwed the pooch
Companies all over the world shutdown because of their incompetence. They do not deserve to be in the security business.
