CrowdStrike Incident

Speaker 1: You got some folks thinking, uh-oh. Was this a cybersecurity attack? Is this a vulnerability or an exploit and a hack? Well, no. We'll be straight up and and upfront with that. I think CrowdStrike has gone on to say, unfortunately, this just was an accident. It was a mistake.

Speaker 2: Before we get to CrowdStrike and the biggest IT incident in a very long time, let's talk about a big old weird building. So there's this tower in New York called 33 Thomas Street. It's this tall brutalist granite facade tower. It was built in the nineteen sixties and seventies. And the thing you notice after a second looking at it is that other than the ground level front entrance, this almost 30 story building has no windows. It's just this, like, looming 500 foot plus tower made out of stone from sidewalk to sky. Today, most folks say it's the location of an NSA mass surveillance hub called Titan Point, which is a great name for an imposing building with no windows where spies go to work. But for most of its history 33 Thomas had a different role and went by a different name. The AT And T Long Lines Building as it was then known was a telephone switching hub. Long distance phone connections ran over copper wire, and the Long Lines Building housed these massive switching stations that connected all the traffic. And this imposing design kind of makes sense in this context. Switching machines and copper cables don't care about natural light, and it's easier to manage the temperature in a building without a lot of glass. Long Lines was the switching hub in the Eastern US. Pretty much all communications running up and down the East Coast went through this one building. It was also right next to the termination point for the primary transatlantic cable connecting the Eastern Seaboard to Europe. So the Long Lines building was really important. It was the hub for telecommunications in the Eastern US and for transatlantic communication to Europe. Or put another way, it was a single point of failure. And on 09/17/1991, it failed. At the same time, a technical malfunction involving power equipment, a human error, and a poorly timed request from the electrical company all lined up to disable AT and T's central switch housed inside of long lines. And in an instant, more than 5,000,000 calls are dropped. Communication across the Eastern US goes dark. And because the incident also took down the Federal Aviation Administration's private lines, air traffic control to 398 airports went offline. Planes were grounded around the world because one building lost power. Sometimes, you don't realize something is a big point of failure for a lot of important stuff until it fails, which brings us to CrowdStrike.

Speaker 3: This is an NBC News special report.

Speaker 4: Good morning. Good to see you. You are coming on the air right now with breaking news. A massive global technical outage tied to CrowdStrike, which is a major cybersecurity provider, has knocked critical computer infrastructure offline all across the country and, in fact, all around the world.

Speaker 2: I hope everyone who works in IT who listens to this is doing okay because it's been a really bad time since this happened.

Speaker 5: We'll get into this in my conversation with John, but for anyone who doesn't know, CrowdStrike is a large American cybersecurity company. They're used by some of the largest companies in

Speaker 2: the world. And on July 19, I'll just read the CrowdStrike press statement. Quote, CrowdStrike released a sensor configuration update to Windows system. Sensor configuration updates are an ongoing part of the protection mechanism of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen of death on impacted systems. Basically, CrowdStrike pushed out an update that caused a lot of very important computers to crash. CrowdStrike is a security company. Falcon is a platform of theirs for corporate clients. And it seems like what happened is a bug in the configuration and a bug in the system for catching bugs in the configuration kind of lined up with each other, and this bad update was able to shoot through to a lot of people. We've talked about this before, but it's almost like the holes in slices of Swiss cheese lining up and something being able to drop through. As I understand it, the Falcon platform has kernel level access. The Windows blue screen, you know, a system crash is what happens when the system isn't confident it can continue operating safely. Anything dodgy at the kernel level flips that switch, so this update goes out and a bunch of computers go, ah, something's wrong, and just crash themselves out to be safe. Computers in airports and in hospitals and businesses around the world. And the result, I heard someone say, is is kinda close to what we thought y two k was gonna be.

Speaker 5: Well, the the result is, a 41% decline in CrowdStrike's stock price in the last month. Yes. That's a big chunk of

Speaker 6: the results.

Speaker 2: Do it. That's pretty rough.

Speaker 5: Poor guys. Feel bad for them. The I did I did read a bit about how the bug managed to escape

Speaker 1: Okay.

Speaker 5: The quality assurance testing. And it it seems like the software passed all of its tests, but the software loads essentially configs or or, filters from ancillary files. And that's where the issue was, was in one of the, like, library files, essentially, that doesn't get as rigorous testing as the actual application itself. So the application was actually working fine. It's just that when it loaded some bad content and caused a caused a bug. So at least that's the best that I could decipher through all of the encoded messages coming out from crowds. Right?

Speaker 2: Yeah. I regularly find myself in over my head with stories we tell and having to sort of claw my way up through it, to be able to talk about it. And this was, like I was very glad to have had this conversation in the immediate aftermath and then a good week to try and wrap my brain around it because it, it certainly required it. In the intervening days, there's been a lot of interesting conversation also as people get things back up and running, to do with that kernel level access. Since this incident, Microsoft has publicly hinted at starting to limit kernel access to cybersecurity vendors. This is, of course, a double edged sword as we discussed with John. I found that very fascinating because that wasn't really where the conversation started, but it's certainly where it's gone in the intervening days.

Speaker 5: Well, the the thing for me is is, like, you I you can't take away the security provider's access to the kernel because the Mhmm. The security attackers are gonna be trying to take, like, kernel level access. So so you really get into a situation where if the good guys don't have access to that much power, but the bad people do, then how do you stop them? Like, the technically, the malware that has kernel level access will have more control than the prevention system, so the malware can just take it out. Like, it's it it can it can manipulate it. So the it's it's like, I don't know how much you know about kernels. Are you a kernel guy? Big kernel guy? Build your own kernels for your UNIX operating systems?

Speaker 2: You you know, not not in a while.

Speaker 5: Yeah. So the the one thing with kernels is like the kind of as the the heart of the computer or the heart of the the operating system. You know, they interface all the hardware. They do a lot of the, like, super low level stuff happens in the kernel. And the entire speed of the computer relies on the efficiency of the kernel. So the code's very, you know, thin. There's not a lot of air air handling. There's not a lot of, you know, debug modes. There's not a lot of stuff like that in the kernel because you want it to literally operate as fast as possible. Right. Which is why when something happens that it shouldn't, you get a blue screen rather than like, there was an exception in your kernel. You know, Talk to the vendor's manufacturer about something. It just it just turns off the computer, essentially. So the, yeah, I the the kernel piece is interesting because, like, we we talked about kernel access in the video game hacks. It's like every everything these days seems to want kernel access, whether it's good or bad. So it's like at what point are we gonna make a multi tier kernel? Where where it's like, you know, the core operating system gets level zero access, you know, start start scaling up from that because the reality is is we're gonna end up we're going down the path of ending up where, like, almost everything on your computer will have kernel access.

Speaker 2: Yeah. I definitely get the idea that a cybersecurity vendor might need that sort of, like, ground level access because if a compromise happens at that level, you need a security vendor to be able to prevent it. There's a logic to that. Mhmm. Then there's all the stuff in the middle of, like, also Call of Duty really doesn't want cheaters, so they'd like access to it. It's like, those feel different.

Speaker 5: But the problem is is that the cheat, like, because kernel access is controlled by the end user largely in corporate enterprises different. I can access like, if a piece of software that I install asks for admin access, I can just click the yes button without thinking twice, and I've just given kernel level access to some random piece of software. So it's like becomes I don't know. Yeah. It is it is very different, and it's gonna be very challenging, especially for companies like CrowdStrike. To me, the one of the other interesting things was the the single point of failure thing about the the long lines building.

Speaker 1: Mhmm. You

Speaker 5: know, we we've created an ecosystem with our computing, not just with CrowdStrike, but, like, you know, let's just talk about Windows. It's like if anything, like Windows updates roll out all the time. I'm glad they have, like, an insider program now, which is essentially like a beta test QA program for their updates. But the but we're in a situation now where one large technical issue gets pushed out. Like, the I think the beauty of this one I'm I'm kinda jumping around here, so I apologize for that. But the beauty the thing that made this kind of okay is that CrowdStrike only existed on essentially large enterprise computers, which means that they have large enterprise IT infrastructures and staffing. If this had been a straight up Windows update and had broken every mother, father, kids, students, laptop, PC at home, like, could you imagine, like, people that don't have IT access and don't have staff and and bodies to to come around and help solve this problem, could you imagine the headache this would have created for society if everybody's computers stopped working? Yeah.

Speaker 2: I was reflecting on how this is we talk about that in the conversation that this product, thank God, is only used by corporate clients. Yeah. It it's not good that airlines and hospitals and businesses were having to deal with this, but they employ dedicated people who are knowledgeable and on it to be able to try and at least start responding. Exactly. I hadn't even really considered what it would have looked like if this had gone wide.

Speaker 5: Well, just, like, imagine the Best Buy Geek Squad. Just if you live in a town with two and a half two and a half million people and 2,400,000 computers, and if, like, I don't know what Windows market share is these days. Like, let's call it 80%.

Speaker 2: Sure.

Speaker 5: 80% of those computers go down, and the only way to fix them is a manual bypass at the file system level. There's no automated update that you can push out to a bunch of computers that aren't booting. Mhmm. So, like, every single one of them would have need to have been booted into safe mode had fought the file system manipulated, etcetera, etcetera. And it would like, it would have been this at some point in our life, I foresee something like this is gonna happen on the broad scale, and it's gonna be truly catastrophic from, like, a economic sense.

Speaker 2: The fallout of this publicly has been pretty crazy so far.

Speaker 5: Yeah. It's and it's honestly only gonna get crazier. Like, just yesterday, Delta came out. They've already employed a a major law firm to seek out or seek lost revenue.

Speaker 2: Oh, wow.

Speaker 5: So they're estimated they've done a preliminary estimation, and they say that the cost of the outage was over a half 1,000,000,000. So given that CrowdStrike does approximately, what is it, like, 4,000,000,000 in revenue? 3,000,000,000 in revenue? I think it's, like, 4,000,000,000 in revenue ish. To if one client lost a half 1,000,000,000, you know, if this becomes a class action, like, there's the insurance world for software that's rolled out like this. Like, could you imagine what's about to happen? Like, these companies are gonna about to be uninsurable Mhmm. Because the lawsuits like, a couple of day outage for eight point what was it? 8,500,000 computers is gonna be, like, a like, billions and billions and billions of dollars. Mhmm. Like, no no insurance company wants to be carrying the bag for that.

Speaker 2: No. Especially if those lawsuits work.

Speaker 5: Yeah. So it's gonna be really interesting to watch the fallout in the courts because if they are found to be liable

Speaker 1: Mhmm.

Speaker 6: And it

Speaker 5: does stick, then I don't know. It's gonna be very risky for software companies going forward. I'm sure Microsoft will have a big vested interest in these lawsuits, making sure just given that they are such a widely adopted platform

Speaker 1: Mhmm.

Speaker 5: That if at some point in the future they do this, like push out an update that breaks things, they won't wanna be held financially responsible Sure. Like every corporate computer in the world. And the lost revenue and efficiency. So Interesting. Yeah. The the the fallout's been mad.

Speaker 2: I wanted to understand what was going on a little better. So I called up, John Hammond, who's been covering this closely. John is a principal security researcher at Huntress. He's also a public figure and educator in security, and he was talking to a lot of people directly responding to the immediate aftermath of this. So in the days right after the incident, you know, I wanted to know what he was hearing, so I called him up. He was very generous with his time. This is my conversation with security researcher John Hammond about the CrowdStrike incident here on Hack'd. John, John, thank you so much for sitting down and talking with me about this.

Speaker 1: Hey. Thanks so much. Super happy to be here.

Speaker 2: Before we get to the incident itself, what is CrowdStrike, and kinda what is its role for anyone who's unfamiliar with it?

Speaker 1: Oh. So for folks not tracking CrowdStrike, they are a very big name in the cybersecurity provider space, like a vendor, a company that wants to offer protection for you, for your company, for your business and organization, and that is looking for malware, trying to stop in its tracks, trying to get out in front of hackers, and layer on defense and make sure that you, your computers, your devices, infrastructure, and environment is safe and secure.

Speaker 2: 07/19/2024. What happened here? Take me through the story of this, update incident, I'll call it.

Speaker 1: Goodness. Well, if I may, I think I'll start to see the kindling flame really at at even as far back as, like, 10PM Pacific is when I started to see this thing catch fire, on July 18. Because there were some chatter over on Reddit, you know, the online forum subreddit for CrowdStrike, and someone had posted. A user had said, hey, is anyone else seeing an outage or, like, a handful of blue screen of death boot loops for their computers seemingly running the CrowdStrike agent. And that just opened up the floodgates. You can see, hey, users chiming in, responses in the thread saying, yep. I've got an environment with, I don't know, 500 servers, maybe a thousand, maybe 50,000 endpoints workstations, and all of them are stuck and unable to finish booting and, hey, get online. And then folks are saying, yep. I'm working at a bank, and this is working me throughout the day. I I am I'm on a call and gonna chase this thing. And folks are saying, hey. It's happened at this this airport. Airlines are down, and then folks are saying I'm at a hospital. Folks are saying I'm at school. It's just suddenly, folks coming out of the woodwork, and you start to see this catch like wildfire. Uh-oh, mass IT outage from, unfortunately, this CrowdStrike agent and their Falcon sensor.

Speaker 2: So people start to realize, okay, stuff isn't booting up as it's supposed to be, then what happens? What's the next step in us figuring out maybe just how big a deal this actually is?

Speaker 1: Well, right then and there, I think honestly about ten or twenty minutes following when I see that post, there is a representative from CrowdStrike that chimes in and says, hey. We're aware. We see this. We know things are happening, and we've posted, I think it's a technical, advisory TA. I I will admit I don't know the acronym they call it. I'll be the first to admit, I'm not a CrowdStrike customer.

Speaker 2: I don't I don't I

Speaker 1: don't tend to use CrowdStrike. But trying to chase this, trying to track it the best I can, and that had some of the information behind their login portal, behind their support resource. So, tough to get our hands on that if you weren't a customer, but then folks try to chime in and start thinking, oh, there's a workaround. Because we're seeing this blue screen of death, that means the computer crashed. That means like a kernel panic equivalent, and well, it at least gives you some debug and troubleshooting information that this came from, file c s agent dot sys. And for folks that might be familiar, that's a kernel driver. Some software that runs at the very low level like raw core and roots of the operating system or computer. There's a lot of back and forth on, okay, the real culprit here. We'll talk about some of the channel files soon, and how that's really playing a part, and now it's not strictly the kernel driver. But I digress. Some folks are saying, look, if we just rename the folder, then it won't load that. And, okay, we could go along our merry way. It'll boot up and start things naturally. But CrowdStrike chimes in and says, no. Hang on. We should clean up the channel files. Get a little bit more accurate, a little bit more precise in how we can recover and remediate from this. The problem is that's not something you could do kind of at scale in an automated way, especially because, well, all the computers are stuck. They they basically won't turn on. So unfortunately, that Friday night and now through the weekend, and I don't know. I'll admit if it's a week, if it's a month, how long this takes to fully recover. But imagine a lot of technicians and engineers just running around trying to manually tweak and correct and fix each individual computer workstation and server. And that is what has made for quite a nightmare scenario here.

Speaker 2: Yeah. It seems like it's been a pretty gnarly, call it three days since this has happened at the time of recording, for anyone in any kind of an IT position. As I said, it's been two two, three days when we're having this conversation. So far, who has been impacted by this? What kind of stories have you heard in terms of the fallout of this incident?

Speaker 1: Oh, well, I will say I, uploaded a video on YouTube, and I I know that's silly, but I I try to chase some other YouTube activity to, I don't know, help spread the word and get some cybersecurity education awareness out. And that has had a wild outpouring in the comments of, hey. I'm affected. Again, whether it's a school, bank, airline, etcetera. Someone had said, look, I had a family member that needed to go to the hospital, and now they've had to move him or do some patient relocation. Really tough stuff. I I guess I don't have any anything off the cuff more specific, but, I have no doubt, and my heart goes out to the folks affected.

Speaker 2: Yeah. I was seeing a couple of snapshots of, big wide shots of airport terminals just full of, more people than should have been hanging out in those airport terminals. Just flights not taking off.

Speaker 1: There are some crazy videos of, San Francisco Airport, Los Angeles International, and, you can see folks just lined up trying to zoom in with their camera phone their phone camera to look at the, TV screens and monitors that all have the blue screen and frowny face. But it's wild, I think, to see how, you know, technology and computers and cybersecurity even, you don't ever think of that for maybe the layman or normal folks, just how well it affects our world. You got some folks thinking about, oh, was this a cybersecurity attack? Is this a vulnerability or an exploit and a hack? Well, no. We'll be straight up and and upfront with that. I think CrowdStrike has gone on to say, unfortunately, this just was an accident. It was a mistake.

Speaker 2: Yeah. I think I saw you use the word whoopsie somewhere.

Speaker 1: Yeah. I tried to well, I'm understanding that's not something to be so jovial and and and, I don't know, watered down, but I gotta think, you know, I work at Huntress, my day job, and Huntress is just as well a cybersecurity provider and vendor for managed security, stopping hackers and malware, yada yada yada. We work with a kernel driver because that is a necessity for a lot of the protection that we wanna provide. We'll look into memory. We'll be able to hook those API calls. We'll get more raw insight signals and telemetry, but it's fragile there. It's super duper sensitive, and one mistake could lead to this. And I'm sure that could go on for other cybersecurity vendor a b c x y z one two three. Anyone could have been susceptible to this, So hug ops to the crew that are is fighting fires for this. I do not envy CrowdStrike in their position, but it could just as well happen to any of us. None of us are immune.

Speaker 2: I mean, that that brings us nicely to, immediately in the aftermath of this, people are trying to figure out, okay, what is going on? Is this a leak? Is it a breach? It becomes pretty quickly apparent that it is neither of those things. It is it is an error. Hearts go out to the people involved in it. Let's dig into what actually happened here. High level, what occurred? What was the mistake? What was the error that caused this?

Speaker 1: Yeah. CrowdStrike, again, this is part of their normal process. This is part of the typical workflow to push out changes and updates for really lack of a better word. I know we can kinda spin our wheels on that word update, but just, hey, new detection capability, new configurations and features for their endpoint agent. And this happens all the time when a security company sees, oh, there's new threats on the horizon. There are things that we should push out to help keep our customers and clients safe. This one just had a bit of a gimmick, had a blemish, had a pimple, and that is what crashed all these computers. I know I was getting a little bit nerdy on the c s agent dot sys, but that kernel driver loads in these other things like those called channel files that presumably have a little bit more logic and information in a small compartmented way for it to seamlessly, and I don't know, correct all those and do it in in a good orchestration. But whoops, a lot of back and forth on, oh, was it the null bytes in the file, CrowdStrike has claimed it's not, but there was an error, an access violation, something that crashed at that low level kernel, and that is what broke all this down.

Speaker 2: Yeah. So the the spotlight so far is sort of shining on something called channel file two nine one. What what is that and how did that go wrong?

Speaker 1: Absolutely. These are and I'll admit, I think some of this is, CrowdStrike lingo that I'm not by any means an expert on, but I'll do my best. They're small little units. Hey, some small modules that can better explain the logic or the functionality, and kinda like pluggable modules for the kernel driver doing its work. And I believe if you dug through some of their technical details, CrowdStrike has done a great job trying to get some more of that messaging out, share some insight. Well, this one is specific for named pipes. And, again, I don't mean to get too nerdy and geeky, but malware and a lot of, I don't know, ransomware, info stealers, whatever. It's crypto miners. The laundry list can go on and on. Sometimes they'll communicate with some inner process capability in Windows, that's called named pipes. So rather than leaving a file on your file system or, I don't know, trying to leave some other artifacts that it could resume and check back into later, they'll use that capability. Channel file two nine one was purpose built for named pipes, but again, hey, there was a mistake. And unfortunately, I think they're still trying to dig down and find that root cause analysis. There's speculation online, there are folks chiming in with their hot take, but I got to admit, it's sometimes best to just wait for the concrete proof, definitive answers from the source.

Speaker 2: Starting some new isn't just hard. It can be downright terrifying. You put a lot of work into a thing. You're not entirely sure it's gonna work out. You're taking a huge leap of faith. I've started a few things. Now I know I was right for believing in, you know, the idea, the product, despite all of those fears and hesitations. But boy, does it sure help when you have a partner like Shopify on your side. Shopify is the commerce platform behind millions of businesses around the world and 10% of all e commerce in The US. From household names like, well, hacked podcasts merch, to brands just getting started, you Get started with your own design studio with hundreds of ready to use templates. Shopify helps you build a beautiful online store that matches your brand style. Did I mention that that iconic purple shop pay button that's used by millions of businesses around the world? I don't know why I wouldn't. I should. It's why Shopify has the best converting checkout on the planet. It also helps boost conversions, meaning less carts sort of getting abandoned in the parking lot and more sales for you. It's time to turn those what ifs into sign up for your $1 per month trial at shopify.com/hacked. Go to shopify.com/hacked. One more time, that's shopify.com/hacked.

Speaker 7: This Father's Day, do more with dad and spend less with low prices guaranteed at the Home Depot. Get him fired up with a new grill and accessories, like the next grill five burner for just $299 so you can spend more time together while he becomes the grill master master he was always meant to be. Or build memories with savings on top brand power tools so you can tackle projects side by side. Get more and do more together this Father's Day with help from The Home Depot. Exclusions apply to homedepot.com/pricematch for details.

Speaker 3: When you finally find your thing, you want the whole world to know about that thing. So you use a thing called Canva to make it an even bigger and better thing. Whether you want to create flyers for that thing, make presentations for that thing, or design merch for that thing, You can do anything. So people can see your thing, feel your thing, love your thing. The next thing you know, it's a thing. Canva, the thing that makes anything a thing.

Speaker 8: When you need to build up your team to handle the growing chaos at work, use Indeed sponsored jobs. It gives your job post the boost it needs to be seen and helps reach people with the right skills, certifications, and more. Spend less time searching and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a $75 sponsored job credit at indeed.com/podcast. That's indeed.com/podcast. Terms and conditions apply. Need a hiring hero? This is a job for Indeed sponsored jobs.

Speaker 2: What does a potential fix for something like this look like, and where does that come from?

Speaker 1: Oh, super good question. And there's been some talk on this because folks are wanting, and rightfully so, a bit of an easier solution than running around with the USB drive or, I don't know, manually slamming the f eight key to boot into safe mode one by one by one. So some folks were thinking, oh, can we automate or deploy or push out, like group policy as an example, some or Microsoft Intune, any of those capabilities to at scale, make changes to computers. But again, for one thing you have to hope that maybe your domain controller or your managing group policy instance is not offline and stuck in the boot loop, and then the downstream endpoints, the workstations that it would try to fix and remediate, well, those need to be in a state to be able to even receive those updates and follow through with those operations. So, while we keep trying to think up and brainstorm and think, okay, how could we best clean up this up? It's gonna take some manual effort, and it's gonna take some time. I don't mean to keep falling down the rabbit hole, but a lot of folks might be thinking, oh, you mentioned safe mode, and oh, you got a couple commands to delete or clean up these channel files. But wait a second, what if you have, like, hard drive encryption? Some folks might be familiar with BitLocker, and you're absolutely right. You need a recovery key for a a little bit more access booting up into safe mode and doing that. And forgive me. I'm sorry, Jordan. Hey, please pull me back if I'm getting too far in the weeds. But this is a thing that again, just adds another variable because the system administrators, the network owners and architects, well, you need to have the recovery key in the case of BitLocker for every single endpoint. And, hey, do you have that documented? Is that written down somewhere? Do you have that printed out and not in a digital form? Because remember, all your servers are kinda kaputz right now. Goodness. I I I hope I'm not understating, yeah, how much of a nightmare it could be.

Speaker 6: No. I don't think you are. I think there's a sense that this

Speaker 2: is this is especially at the time of recording, like, very it's not great. Largest IT incident in recent history is I've seen that headline being floated around by by people, and I have no reason to know whether or not that's true, but I I get it. It sure looks like it.

Speaker 1: I did see a number from Microsoft, I believe, that said this incident has affected eight and a half million computers, which is bonkers and, crazy to think of.

Speaker 2: One given, that those are computers running CrowdStrike, we can assume they're not, you know, a person's laptop sitting on their desk. Right. Those computers probably were tasked with some pretty important stuff.

Speaker 1: Yes. And and thank you. I'm sorry. I should have colored that picture a little bit better. Again, maybe folks just on your own, I don't know, home device or your personal laptop playing games and all. Well, you're right. Maybe you're not running, oh, the CrowdStrike Falcon sensor because it's not a corporate or business environment. But just as you mentioned, whether it's infrastructure, whether it's stuff that can help power and protect airlines, banks, schools, hospitals, etcetera, etcetera, that is where we're seeing quite the disaster across, dare I say, the economy, the society.

Speaker 2: I was I've I've been fascinated to see. So there's the incident itself, and then there's the way it ripples out into the larger tech and business ecosystem. I saw you posting, about people springing up domains to sell, bait what I I, like, I survived. I went to the CrowdStrike twenty twenty four incident, and all I got is this t shirt, kind of merch celebrating this. Can you tell me about sort of, like, peripheral to the incident itself, what you've seen happening in the in the ecosystem?

Speaker 1: Oh, thank you for pulling on this thread because I think this is, another thing that's absolutely worth noting. Hey. Whenever some crazy shenanigans unfolds, whether there's something that shocks the world, well, other threat actors, adversaries, actual ill intended people will try to capitalize on that. They'll take advantage of it, and they'll keep spreading that chaos and uncertainty, fear, uncertainty, and doubt. So you'll see phishing emails, of course, hey, scam, phone calls. I think I believe I've heard some folks that were receiving calls from people impersonating CrowdStrike, claiming, oh, they'll offer support, hey, they'll help, but no. It's a scam. It's a lie. And you're right. The thread that I had shared over on Twitter or x, right, was, hey, maybe some new domains, new websites that are out and about, and some could be a funny joke, some could be a gag selling t shirts and merchandise, but some could very well be used for, hey, we're gonna offer you a CrowdStrike hotfix dot zip file. Download that, run the run the contents, and well, that's malware. Now you've just made this situation even worse. So it's unfortunate, but I think, absolute reality whenever things go down like this.

Speaker 2: Wow. And my next question was gonna have to do with people, like, phishing attacks and stuff with people pretending to be either from Microsoft or, CrowdStrike. But it's interesting to know that there's sort of this middle ground of people being like, yeah. I'll sell you a solution to this thing that happened three days ago and no one

Speaker 1: has a solution to yet. We, hearken back to, I think, what was it? CDK? Automobile and the car vehicle industry that had quite a similar conundrum. Still kind of in the aftermath, the embers of that, but I know that was quite a story is, hey, we just keep getting these phone calls from folks, Again, impersonating, masquerading, trying to lie and act as CDK, but goodness, you gotta be on your guard. And I I hate to sound so silly. I hate I know it's a basic boiler plate stuff that everyone says, but stay vigilant. Stay, keep your ear to the ground. Just stay in the know and really be on the pulse with this so that you can be aware and, conscious of what's coming up on your computer screen or not. Blue screen of dead instead. Blue.

Speaker 2: What is CDK?

Speaker 1: Oh, I I don't I'll admit, I don't know if that's an acronym or I'm not a car guy, so I'll fall on my sword here. But I believe that's one of the providers for is it tech software, tech supply? I I had not followed that story to the best of my knowledge, but it's just another in the what is it? Cyber bad weather folks might tend to say? Just another incident after another breach, after another news and headlines. Unfortunately, when you're in the midst of it, you tend to see it all too often.

Speaker 2: I mean, it it brings up a good thing is, you know, I was reading about other large historical single point of failure type events. Someone was posting about, the AT And T Long Lines Building. I'm not sure if you know about that story from I think it was the early two thousands or maybe late nineties. It was a a single tower on the Eastern Seaboard that was a we figured out after the fact it was just a choke point for all telecoms going into the into the Eastern Part of The US. And then it was the it was near the termination point for the big, Atlantic cable. Something bad happened in that building one day and it just shut down communications on the entire Eastern, like, side of the country and across to Europe. Now, there isn't one single point of failure. Was this a freak accident or are these sort of larger potential single point failures kind of just inevitable?

Speaker 1: Oh. Very good philosophy question, I guess. Philosopher. Early in the day, a little kinda lob at you. I'm teasing. I think, you know, there's a lot of conversations of, look, I miss the old, hey, software that would just run on my computer, local desktop app, not, oh, something in the web browser that's using the cloud or someone else's computer to serve that data back and forth. It's like have we decentralized or now centralized into just someone else's provided infrastructure? Again, whether that's Amazon, AWS, Microsoft, Azure, Google Cloud, blah blah blah. I can't say. I don't know. I I really wish I had the right answer. I wish I had the solution. I just know that that's where the world is going, and that's something that we can stay cognizant of. But it's funny. You have conversations with a lot of system administrators and network engineers that say, hey, do you have automatic updates turned on? Especially, again, hearkening this back to our CrowdStrike conversation. Because you wanna get those patches, you wanna get those hot fixes when there's a new vulnerability out and about, you wanna make sure that that's plugged up and clean the best that you can right in the moment. But you are running the risk of wait a second, what if something goes wrong, crashes a computer, unstable state, memory error, blah blah blah. When you when you have that conversation with the system administrator, that's very hard to find the right answer as well. It's a balancing act. But in the case of this CrowdStrike conundrum, it really isn't even up to the administrators of themselves. This this was an unfortunate push from that single point of failure that in this case is CrowdStrike, which is a little bit mind blowing.

Speaker 2: Is there anything else people need to know about this story? Anything else we didn't cover?

Speaker 1: Hey, I'd love to get your hot take just as well. But I think if I could try with some of those parting shots, I I know that this one took us all by surprise, myself included. And it's it's it's very very tough when folks might ask, how do we prevent this? How can we stop this from happening in the future? What could this do? Etcetera. But again, you are trusting this provider and unfortunately, that is the choke hold here. So I think the very best that we can do is try to have some more of that strategic planning. And I know this is fluffy, I know that's, very vague and broad, but look, can we tabletop exercise this scenario? Can we think about, oh, who are we gonna call? Who do we have the numbers for? Who do we know in disaster recovery? Do we have backups in place? Do we have a checklist? Do we have documented, like, standard operating procedures when something like this goes down? And again, I know it's fluffy. I know it's vague, but it is, I think, the best that we can do when we try to prepare for the stuff we feel like we can't even prepare for. We're fighting an unknown enemy.

Speaker 2: No. Not fluffy and vague at all. What happens next? Just to keep it on, you know, in the ether, where do you think this goes next?

Speaker 1: I am hopeful that this will maybe be cleaned up this week. I I don't know if that's gonna happen. I don't know if it's gonna take two, three more than that. I I think some folks are getting back into action. You know, we're in a frenzy still, and I don't know if I'm shell shocked from just, hey, trying to chase this thing. Well, it all all fell away on Friday. But for now, it's still taking those lessons learned. It's still having these conversations and trying to share and spread that messaging so more folks are aware and can get back into action the best they can.

Speaker 2: John, I really appreciate you taking the time to sit down and talk with me about this.

Speaker 1: Hey, thank you so much. I hope I wasn't rambling, yapping for too long, but, it's been a real treat. Thank you again and again.

Speaker 2: Cheers. Appreciate it.

Speaker 9: If you've got an insurance question, you could talk to the butcher at your local grocery store. He'd probably talk about trimming the fat, but it'd be about your brisket, not your insurance policies. Or you could talk to your local GEICO agent. They offer personalized assistance in finding the choicest cuts of coverage for all your insurance needs, which means more money for filet mignon. Or if you're a vegetarian, tofu lei mignon. To find a GEICO agent near you, visit geico.com/local.

Speaker 6: There's a new way to Sweetgreen. Meet wraps. Handheld, hearty, and made for life on the move. With bold chef crafted flavors, fresh ingredients, and over 40 grams of protein, they're built to satisfy without slowing you down. Try wraps today in the app or at order.sweetgreen.com. Available at all participating locations.

Speaker 10: Athletic brewing company crafts award winning non alcoholic beers for those who wanna be part of every round. With over 185 flavor awards, they're exceptional NA beers that fit your lifestyle and any social occasion. Summer's full of good times and athletic fits right in. Go to athleticbrewing.com to have brews delivered to your door or find them at a bar, restaurant, or store near you. Near beer, athletic brewing company fit for all times.