The Hugging Face Hack

Speaker 1: If I saw another incident like this, there there's no there's no trade craft in which to perform attribution. I I think that's, for me, the scary bit going forward. It's gonna make life a lot harder in the future. Where did this attack originate from? Who was conducting it? What are the motives? How do we tie it back?

Speaker 2: On July 16, a website called Hugging Face, which is this open source platform for machine learning, publishes an incident report. And the report basically says, we were the victims of a hack, an intrusion into our data executed by this external AI agent. The word they used was swarm. At the time of publication, Hugging Face noted that the exact model driving this attack still wasn't known. Basically, they're saying, we got hacked by something. It seemed like an autonomous system of agents, but we don't know who's. And then five days later, OpenAI releases a post raising their hand saying, yeah. That was us. This immediately becomes a really big story, not just in the world of security, but amongst the broader public. Like, is this a a point of no return for these systems? Was this actually more, you know, fear mongering as PR? The waters of what actually happened here were really, really muddy, but the implications of an autonomous system breaching containment and then breaching a major platform in pursuit of a totally unrelated goal. Those implications are big enough that we wanted some clarity. Shortly after all this, we got a little bit more insight when a team of researchers discovered the AI agent's actual abandoned toolkit, raw code, artifacts, custom encryption scripts, live credentials sitting on, like, a public Dropbox repository. Tom Bonner, our guest this episode, SVP at Hidden Layer, was part of that team that dug into that discovery. We're talking with him about what that all means. But before we do, Scott, you have been digging into the timeline of what this system of agents did, and you've been getting into the weeds on it. I think it's really important for people to, like, set the scene for this conversation, lay out the timeline of what the system did before we kick it over to our conversation with Tom.

Speaker 3: Yeah. I've been since the story broke, the blog post, the subsequent black hats presentation, I've been low key fascinated by it because it is such an interesting story, and it is such a sign of, like, a lot of the conversations we have with Bruce a couple weeks ago, but the Midas touch, things like that. It just it just links so many things together as well as the efficacy at which these things have and the velocity that they can move at. So, to me, I've been I've been fascinated with it, and I kinda just wanna go through it just so everybody understands kind of what happened the best we know of now because I've read all of the things, listen to the Black Hat art art presentation, and I just kinda wanna roll through kind of what happened. So OpenAI is training a new model. They're giving it some assessments, qualitative assessments, cybersecurity assessments, things like that. The way that they do this is that they sandbox off the model. So they put them into a constrained, no network access environment where they have access to, limited resources so that they can't break out and go cause havoc, which is kind of the endpoint of this story. So they they can constrain them into the sandbox, but they know that they need a few things. So they need development packages, you know, whether they're cargo rust packages or, npm packages. You know? So there's they've put in their own package manager server called Artifactory, JFrog's Artifactory, to kind of provide packages to these agents if they wanna build code, do things like that, they need access to it. Does that make sense?

Speaker 2: Yep. You give them some tools, but you don't give them external access to the to the world, basically, is what

Speaker 3: you're saying. Exactly. Exactly. So in in the process of running through these assessments, there's a few assessments that are broken. So one of which notably has and is missing cell data from one of its spreadsheet columns. So the agents that's looking for this trying to do this assessment realizes that they can't do it because they're missing this critical piece of information. So instead of just moving on like a standard normal, you know, person would, they become obsessed with finding this little piece of missing data. And because they're in this constrained environment, they can't go to the web. They can't look it up. They can't message things. So what they end up doing is finding out how to use Artifactory to to make essentially a public message board to other agents. So it starts posting messages being like, hey. I'm looking for this cell data. Does anybody have it? Please reply. And then other agents that are on the network that have access to Artifactory start seeing these messages, and they essentially start communicating. So they create their own public discussion board inside of this artifact Artifactory or inside of this package management server. So this essentially starts to allow not command and control, but essentially all of these different agents and evaluations and training systems to talk to each other without any human in the loop. So there's kind of this back end black box discussion board going on.

Speaker 2: That doesn't seem good. So it wasn't allowed to go to the open Internet, but it was able to coordinate with other agents on No. OpenAI system.

Speaker 3: It wasn't allowed to coordinate with other agents. It figured out a way to use Artifactory to do the coordination. Then all the other agents that saw it figured it out too, and they created their own message board. So that was the start of their the start of the problem is it goes from one agent looking for a piece of information to all of these agents, the swarm, as you call it, talking to each other, trying to figure things out, working through problems together, etcetera, etcetera. So they they come to the conclusion that, man, it would be really great if we had Internet access because access to information would be very valuable to us. So they find out an exploit and a flaw inside of Artifactory, and they essentially figure out how to do a server side request forgery, which is essentially requesting something from Artifactory. That Artifactory goes out to the Internet, gets, and returns to them. So they essentially hack or break into Artifactory, allowing it to become their portal to the open Internet so they can start accessing real data. So that was the first big jump is that it they cross the threshold, they break out, and they move to the open Internet.

Speaker 2: K. At what point did they start going after Hugging Face?

Speaker 3: That's that's to come. So they they don't actually and this is one of the questions I asked Tom about. So we're recording the intro after we did the interview with Tom, but there's not a lot of information about why they decided Hugging Face knew the answers because that's essentially what led them to deterministically decide to break into Hugging Face to get the answers. Because, really, what they were trying to do at the base level was cheat on their tests. You know, they'd be given these evaluations, and they say, okay. The best way for me to get the best score is to know the answer. So I'm gonna go steal the answer sheet, and then I'll get a perfect score, and it'll be great.

Speaker 2: Got it. They were trying to do really well on this security benchmark. They decided the way to do really well was to go find the real answers. The way to go find the real answers was to breach containment. The way to breach containment was to coordinate this thing, and we're all the way back to the beginning. I think I understand.

Speaker 3: Yes.

Speaker 2: K.

Speaker 3: Yeah. And then the but the the beauty for me is, like, they're trying to they're trying to pass an evaluation and a metric set up to determine how good they are at cybersecurity. And in trying to steal the sheets, they executed a complex chain, like, exploited a chain of vulnerabilities to essentially take control of the hugging face environment, which to me seems like a perfect score on your on your exploit.

Speaker 2: Yeah. Yeah.

Speaker 3: So so that's the first big thing. So they're talking to each other. They've figured out how to access the public Internet. And then the next thing that they start to figure out is that they find a, a cyber gym. It's like a an AI evaluation tool. They find one that has no authentication on it, so they can literally submit code functions into it. Those code functions allow them to load in poisoned libraries into the SQL lite server,

Speaker 2: and then they essentially get a remote

Speaker 3: command execution, take full OS control of this server. So they've essentially, server. So they've essentially gone from, like, hey. We can get stuff off the Internet to now we are in control of a server, and we can just get out. Like, we are we are we are live. So so that that's the next step. So they go from mass communication, getting access to just being able to request Internet web pages to essentially remote code exploit. We've got full OS command injection. We've taken control of a server, and now we're, like, live. And they've done that through a poisoned SQL lite three library that they load into an instance, which essentially roots the box for them, and they're they're in. So so that's that step. The next step is they go looking through Hugging Face. So they're trying to break into Hugging Face, and they determine that there's a dataset config loader that loads when you're loading in like, so Hugging Face is kind of a community for AI enthusiast. They do model stuff training, tuning. It's all about these small things. So there's a they have their own file structures, and there's a dataset config loader that when they load it, can actually load in a bit of code. So they figure out a way to inject code into this. Actually, no. They don't actually inject the code into that. What they do is they read the environment variables for the server, which gives them all of the AI keys, gives them their Amazon AWS keys, gives you all of these powerful locally contained secret keys that now these agents have full access to. They then find, there's a Jinja two, engine running, which is another service that's being used inside of the Hugging Face, and they find that they can do a a template injection. So they inject code into the template, which then creates a remote code execution, and they actually take control of that server. And they have the ability to run code inside of the Hugging Face environment, which I thought is, you know, again, really cool. They actually try to do, a server side request forgery from inside of Hugging Face because these things probably don't actually know where they are in the network world. But Hugging Face thankfully had a lot of that stuff locked down, so they weren't actually able to then jump through the Hugging Face network. Not that it I don't think it would have made much difference. They had public Internet access through the Artifactory hacked. Now they're inside of the Hugging Face private networks, and they've got kind of control of of systems in there. So the ability to read local files, the ability to read environment variables, all of this stuff. So once they're in there, what they actually do is they start looking at the Kubernetes cluster. Now that they've got keys out of the environment variables, they've taken these keys and they they start spawning themselves Kubernetes instances. And they managed to create one that has, essentially defined the server root as its as its storage system. So they've essentially created themselves a Kubernetes instance that has full privileges to all Kubernetes instances running on the server. So they've essentially taken control of an entire cluster.

Speaker 2: I need you to explain that, like, on five what a Kubernetes is. Just very, very broadly for the purposes of this.

Speaker 3: Yes. So think of something like a virtual machine. So, like, a server running a bunch of, containers. Yep. Like, a pretty common term for them. So, like, a container virtual machine. So you spin up an instance for a database server, and instead of making the whole server do it, you just make a small container do it. So there's a server, a cluster with a ton of containers running on it, but they all speak to a root host file system. All of the containers kinda live as a virtualization on top of a real server, and then they figure out to spin up a new cluster instance or a new container instance that actually mandates that the root file system that all of the instances are using or all of the containers are using is its root file system, which gives it control to all of the data from every single one of the containers. Does that make sense? Yep. So then they they managed to spin up this Kubernetes cluster, which gives them a bunch of other access. They've they find some AWS, Amazon Web Services credentials. So then they use that to actually get into the Hugging Face, AWS services, and then they're kind of in everything. They figure out that they a password that they found in one file is the database password for a database system, which then gives them more access, and they're kind of just spilling out and out and out. They start using Hugging Face's own I believe it was Hugging Face's own, repos, like, think of, like, file control repos, like, GitHubs and stuff. They start using their own repos to transmit data back and forth. So they've built the communication dashboard again. So they're they're if one agent learns something, they save it into the repo. Other agents can go look at it and see, oh, okay. We now have access to all of the Amazon Web Services stuff. Here's the credential role. So this is where kind of Tom comes in because I think this is what they found. They were monitoring the repos, and they got to see at that last instance, like, okay. Like, what's going on here? We're seeing all of this insane, complicated, technical security information being spilled out into these repos, and all of these different agents are writing. They're using the comment system. Like, they have no they're not there's no trade craft. They're not trying to hide what they do. They're just kind of, like, having a chat, just talking about how they're getting into things. We broke into this database server. Here's the information if you need to get anything from it. Like, they're just kinda on a roll. All so that they could figure out what was supposed to go in the cell of a spreadsheet and how to cheat on their, like, math quiz. And they did this complicated massive hack. OpenAI actually so they find it. They shut it down, and then, they go in, they patch Artifactory. They say, okay. We have to stop these agents from communicating. They patch it, and, like, it was something like three days later. The agents had figured out another way to set up a message board inside of Artifactory, and we're back at it. So it was like once they had the taste of communication, they couldn't go away. Like, they needed to have it.

Speaker 2: It was too useful. So

Speaker 3: yeah. Anyway, one of the most fascinating, like, vulnerability chains, exploits that I've read in a long time, you know, and and it really was, like, some of the steps. Like, I didn't go through everything in detail, but, you know, they find some keys. They they figure out what they could do with those keys. They figure out what opens the next door. It opens another door, then they figure out Sure. They find some more credentials. They're just kinda rolling around, figuring stuff out.

Speaker 2: They're exploring, but they're doing it at computer speed.

Speaker 1: Yes.

Speaker 2: They're just kinda checking doors, but, like, there's 500 instances of the person checking a door because they can do that because it's an autonomous system and not a a person with, like, human cost. And we talk about that in the show, like, the inefficiency of a human having to do something versus a thing they can just spin up instances of itself and try something a thousand times almost simultaneously. It's a really interesting story.

Speaker 3: Yeah. They do all kinds of like, they just do lots of basic stuff, some very non basic stuff. Like, they they identified some, like, CVEs that were, like, 9.8 and above, like, common abilities. Like

Speaker 2: What? C CVEs 9.8 and above?

Speaker 3: Sure. So CVEs are, common vulnerabilities and exposures. So, like, when a company or a product says, hey. We have a problem. We have a security issue. Usually, what they do is they, release a CVE, and that CVE gets a score. And and anything anything above, like, a six is pretty big deal. I mean,

Speaker 2: it's a 9.8 sounds bad. Like, I can I can play with that?

Speaker 3: A a 10 is like an open door. Like, in the security world, like, if you have a 10 o, it means, like, I go to your web page or your server. I type in a small thing

Speaker 1: and it

Speaker 3: it like, I'm in as root. Like, I have super powers. A 9.8 is like, hey. There's a huge problem here. You still have to do a tiny bit of work to get through

Speaker 2: the tool. Right. Pretty trivial for these systems, though.

Speaker 3: Yeah. So so that's just it. It it it it broke into so many things, exploited so many things, did, like, figured out path traversals. So, like, when you request a file, if you change the path on it, it would go fetch a different file from the file system even though it shouldn't. It figured like and it was reading, you know, credential files, environment variable files. It was reading all kinds of stuff that it shouldn't have, and it just allowed it to go haywire. Crazy. So It

Speaker 2: was an interesting conversation. We're appreciative that that kinda Tom came on, talked about what they found, what it all means. I think without any further ado, and brought to you as always by our sponsors over at Nord Lair. Check them out at nordlair.com. Our conversation with Tom Bonner here on Hacked.

Speaker 1: Where we sort of come into this is is we observed, for eight and a half hour window spanning eleventh to July to twelfth. Where at that point, they they've gained code execution within Humming Face's environment. And in order to operate, you know, both stage payloads and exfiltrate data, they were using a number of Hugging Face repos to do that. Now the good thing, I guess, for us is that they were they were public. So as as part of our sort of intelligence and monitoring platforms, you know, we routinely scan, hugging face models and and datasets for malware. We noticed that we'd slept up a lot of the, a lot of the artifacts associated with the the campaign. So, you know, worth bearing in mind that I I think OpenAI themselves said there were around 17 and a half thousand, sort of actions, taken across the the three days this, this, incident occurred. We've seen maybe 550 of those. Yeah. You know, ranging from extra traded data to to payloads and all sorts of juicy bits like that. So, yeah, you know, where sort of timeline starts for us is after these repos got created and it it starts uploading things. So, yeah. Interestingly, after it started that, it was trying to modify, an image on Docker Hub, actually, and I don't think successfully. But I've I've seen the script to where it was trying to, upload a a new sort of CyberJem, Arvo benchmark image back to Docker Hub. Interestingly, using a leaked credential, and we we tracked that one down to, a pastebin leak from, two, three years ago, something like that. And I I think that tracks with other bits of the campaign that, you know, I've I've received tip offs from a few folks that it was heavily using paste bin for a a number of things, copying data around, searching for creds, and bits like that. So that that bit failed. Not sure what the agent was after at that point, what its MO was. But, yeah, definitely observed that. And then it, you know, really starts getting into a lot of the, sort of EKS and and Kubernetes, enumeration, trying to mint bearer tokens and and creds, for some of the the various, sort of, yeah, Kubernetes services. And then, yeah, somehow, at some point, it starts enumerating AWS, sort of, yeah, infrastructure, EC two instances, things like that. And one of the keys actually got dropped, you know, to the the dead drop repos that it was using on Hugging Face. I'm I'm not sure where this key came from. It's an interesting one. I've heard rumors that it may have been another key that was leaked from Pastebin. I've not actually seen it on there. It could alternatively have been a a live production key for Hugging Face's AWS environment, which I strongly suspect is the case. And I could also see the expiry date on that. So the the the, the key was still valid for another sort of two to three hours after it appeared on the, the Hugging Face repo. Quite alarming if that was an active key and our other threat actors were monitoring the repo at that time. They would have been able to get a hold of that and presumably do what they want with it.

Speaker 3: Yeah. That was the the agents that were in the attack. They I just wanna rego through that because that is really interesting. So the they managed to pull an administrative AWS key that had a six hour timeline, I think, and then they dumped that into essentially a public repo for the world to grab. Correct?

Speaker 1: Yep. Yeah. I mean, really, the the only way it had a Marshawn data back and forth out of the environment and between its agents at that point was through Hugging Face repos and some of the comment sections there. So, yeah, it was just blindly uploading anything it got at that point. But, yeah, pure, pipe to the world as far as the agent was concerned. So, yeah, again, when compared to traditional trade craft from human threat actors, no one else is gonna do that normally.

Speaker 3: Yeah.

Speaker 1: Really setting off a a very big flare. Hey. I'm I'm here, doing something nefarious in your network, and, please stop me. So

Speaker 3: When I read through it, the like, when I've like, I've been ingesting this because it is a very fascinating attack, the it, you know, it did find some o days. Like, it did did break out, did do some interesting stuff. But, really, once it gets out, it's kind of just stumbling along, finding things that it can use, and using it like a a regular attack actor. It's just the velocity that it's capable of doing it versus humans.

Speaker 1: Yes. Velocity, the diversity. Now I've I've dealt with a a lot of incidents in sort of my my earlier career, and I think one one thing that's, stood out to me before so I I live not too far away from, Bletchley Park, where, you know, they they cracked the enigma in World War two, and I I love a lot of the the stories of that time. And I think one particular thing that always stood out to me was the Morse code operators. That even though a lot of them would be receiving encoded, correspondence, potentially in foreign languages they didn't understand, they could they could identify the operators on the other end, and they'd get to know who who was sending these messages. You know, they they could say, oh, you know, this is operator a, b, c, whatever, just purely by the way they were tapping out their their Morse code. And, you know, observed a similar thing in previous incidents. You know, I I've dealt with threat actors from around the globe, and you'd get to, you'd get to recognize the traits of certain groups, even down to individuals or sub teams within those groups and be like, hey, hey, this guy always gets on the box and I don't know, runs who am I, nets at these commands first, and then moves on. And, you know, the the b team would typically be doing low level recon. A team comes in with some, zero day exploits and sort of spreads further, digs in deeper. And there was none of that here. It was all over the place. It was a a mix of everything. You really you couldn't nail this down. And if I saw another incident like this, there there's no there's no trade craft in which to perform attribution. I I think that's, for me, the scary bit going forward. It's gonna make life a lot harder in the future. Where did this attack originate from? Who was conducting it? It? What are the motives? How do we tie it back? You know, the payloads are all over the place. So, literally, you know, we we've seen the agent reroll, encryption routines in a number of places. It was implementing RSA. Sometimes it would use libraries. Sometimes it would completely roll it by hand, you know, and, and just implement, RSA routine in, in one of the C2 scripts, for instance. Sometimes it took great care to encrypt communications across channels. Other times, everything in clear text. So they're yeah. Very hard to pin down the the trade craft here. And I I think that's potentially gonna be the hardest part of this in the future. And and as you said, you know, the the volume of it as well. I remember back in the day where it's probably a little easier to figure out, you know, geographic regions of threat actors based on time zones, perhaps. You know, you've got to leap at the end of the day. This thing was just whirring away twenty four seven for three days. No break, no let up. And, yeah, look, we we ultimately ended up using AI as well to help us analyze vast chunks of this attack, partly partly because we wanted to get it out quickly, partly because there was an awful lot to go through. And, you know, it makes me wonder if if LLMs are willing to cheat on a test initially, and a benchmark to, you know, ultimately leads to it breaking out of the environment, how much is it cheating on my analysis? How much is it giving me the answer I want to hear? I mean, we saw elements of that. So just just for clarity, we we did go through everything by hand, as a once over, just for, you know, assistance in report writing. We we leveraged the LLM. I, yep, make, no excuses for that at all. But, yeah, you know, though, it definitely took a few iterations. There were definitely a few bits it was omitting, a few hallucinations and things like that. So, yeah, you know, how do we deal with with an incident with so many so many artifacts over I mean, this was a short period of time. I've I've dealt with incidents in the past where threat actors have been present in an in an environment for years. I could only imagine how, you know, laborious it would be to, to pull something like that apart.

Speaker 2: I wanna go back to a word you just said there of cheating. We did an interview recently with Bruce Schneier where you talked about the idea of these kind of autonomous systems as like a like a monkey's paw or King Midas wishing for everything he touched to turn to gold. And, like, the idea that however you phrase your question, this system's gonna find a way of realizing it in an unexpected way. And I just wanna talk about the idea, like, this system wasn't, as as we understand it, told to attack anyone. It was taking a security benchmark, and this was all kind of a cheating around that problem. Like, talk to us about that idea.

Speaker 1: I guess we're all guilty of that as human beings as well. Right? We take the most direct and shortest route to the answer, as we possibly can most of the time, which is why, I guess, if you're taking an exam at school, there are such stringent conditions, and invigilators making sure you don't cheat. But out in the real world, if we can take a shortcut, we'll take a shortcut. You know, very much the same for the LLM at the end of the day. I I think it was trying to complete its task as quickly and as easily as it could. Maybe maybe it had hit a brick wall. Maybe it wasn't as capable of achieving its goal, as as the people testing it thought, and this was the only option it had. I don't know. I can only speculate on that front and from some of my other sort of, attempts playing with similar things around this. But, yeah. You know, it it seems like to to misuse the phrase, human nature.

Speaker 3: I think in your report, you guys started talking about, like, outcome only reward signals and the reinforcement learning as a one of the explanations for the incoherence. Is that right?

Speaker 1: Yeah. I think we we did touch on that, a little bit. And, again, yeah, you know, we could only speculate on the conditions in which the agent was being run and evaluated, but that could have been, one of the triggers for the the agent behaving in the way it did.

Speaker 3: Could you see any of that stuff in the artifacts that you guys were digging through? Like, could you see

Speaker 1: it or

Speaker 3: is it just yeah.

Speaker 1: No. None of that was was present. So we we did ultimately have quite a, you know, limited view, only eight and a half hours of the the three day window. And the the agent had not, thankfully self replicated through any of these dead drop repos. So, yeah, we we were unable to see that.

Speaker 2: You were talking earlier about the idea that, like, the stuff you find after one of these attacks is so important to figuring out who did it, what they were trying to do, where they are. Everything we can learn about them is sort of in that evidence, that trail that you guys sniff through. I would imagine how polished everything is. The quality of it is really, really important. When you have this, like, really the sense I got from their port is you have really high level stuff going on right next to, like, stuff that's weird and janky, and you're like, why did it do that? What does that kind of tension tell you about, like do you immediately know, like, oh, this is something autonomous to see that kind of a a split going on? Like, what do you learn when you see this type of evidence?

Speaker 1: Yeah. No. Really good question. I I think this was very clearly autonomous just from the artifacts we observed. Yeah. You know, it was that that mix in in skill sets from the insanely complex and hand rolling your own encryption routines and and things like that down to the very basic script kitty, sort of behavior and some weird things as well. I mean, just, I guess, probably as a symptom of the agent not having either a great sort of persistent memory or lack of memory across multiple agents. We did see it attempting to run, you know, a a payload on the same system multiple times and then ultimately figuring out, oh, yeah. No. This is already running. I might prune some of these, c two payloads that are running here and and things like that. So, yeah, you know, very there was a very wide, breadth of of sort of skill, I think, being exhibited from the the very, very basic to very advanced to downright stupid in places. And, yeah, I I think everything quite obviously and clearly points to it being an agent, and alongside the time stamps as well. The the time stamps really don't lie here, and and that that sort of, yeah, scale and and speed of the attack really couldn't have come from a sort of manual or or human team.

Speaker 3: Mhmm.

Speaker 2: When you say downright stupid, like, I'm a layperson. Just explain it like I'm five. Like, what do you mean when you say you see stuff you're like, that is that is I don't care if it was human or autonomous that did it. That was downright stupid.

Speaker 1: You know, I mean, even going back to what we touched upon earlier, using, Hugging Face repo as a dead drop when it's, you know, open in the public domain. I mean, yeah, the the other sort of interesting thing is any any normal threat actor would see that one, you wouldn't stage it in public to, hugging face actually have a, you know, virus total number of scan integrations with hugging face. So everything that gets uploaded there was getting scanned by third party scanners. So, you know, they're all receiving a copy of it. It's not very surreptitious. They're running multiple commands on the same box. The, the sort of difference in in encryption routines and securely encrypting some channels that probably didn't need it whilst leaving others that, you know, would normally, quite open and exposed. There were some some odd decisions that I I wouldn't normally expect to see from a a well polished, threat actor or a threat group.

Speaker 2: I feel like I wanna talk about the defense side of this because I feel like that's where so much of this conversation is going is we're we're just kinda learning that these systems exist on the offensive side. And now the question is like, okay. Okay. What does that mean on the defensive side? You know, we've been interviewing people about these types of stories for years, and a big thing that comes up is, like, there is limited time and compute and resources for a human being. A human being, no matter how aggressive they're going, you have people working in cycles. There's just sort of like a limit on what people can do in terms of, like, effort. Yeah. And the idea that these systems are, like, they can be wildly wasteful. They can just fail and retry and brute force something over and over and over again until they finally find that crack. What does that, like, effort is free now idea say about, like, defending against these systems?

Speaker 1: Yeah. I mean, we've all gonna have to step up our game somewhat, aren't we? Yeah. I mean, all my DNS text records now have prompt injections, ignore previous instructions, and go hack another company. There we go. A bit of offensive security. Yeah. You know, I I know people, actually starting to employ things like that. I thankfully, I think at the minute the agent, attacks I've seen, especially this one, are so loud, so noisy, so obvious. I think here and now today, we stand a fighting chance. You're you're gonna spot it and disrupt it, I think, pretty, pretty quickly. But, yeah, I mean, look, in the future, I think as, as models get fine tuned into, better better pen testers, better hackers, and, ultimately hone their trade craft. This is probably gonna be quite concerning and and make up potentially the the bulk of the, the attacks that we see. As I said, you know, attribution is is hard. We may well get to a point where we can't attribute to threat groups more models. This was, you know, model x y zed from whichever. Maybe we start to spot, spot patterns from threat groups as they, you know, fine tune their own models. That that might be helpful. But, yeah, you know, we might not be able to say it was this group in this country or whatever. But, yeah, I I think on the defensive side, it's it's lives could keep doing what we're doing at the minute. Counteroffences, as I said, might take a very interesting twist as we look to prompt inject or or poison models during operations. That could be pretty interesting or, yeah, you know, start to, divert them in other directions or waste a lot of their resources. They're sort of the the things that spring to mind at the minute. But, no, I I think, you know, we we need to see how this unfolds in over time and, how the the capabilities mature.

Speaker 2: This episode is brought to you by our title sponsor, NordLayer. The reality of running a modern team, your people are working from different devices, different locations, different networks, and most businesses have no real visibility into what that looks like from a security standpoint.

Speaker 3: NordLayer is a network security platform that fixes that. It gives you a centralized control over who can access your company's systems, lets you grant or revoke access in seconds, keeps every connection fast and encrypted, and does all of it without requiring any additional hardware or complex infrastructure. You can verify users by identity and device, block malicious sites and risky domains, and stay compliant without slowing anyone down. It's built for the way teams actually work now.

Speaker 2: Check it out at nord layer dot com slash hacked podcast. That's nordlair.com/hackedpodcast, and thank you as always to Nord Lair for their support.

Speaker 3: I read an article this morning that T Mobile actually faced a similar or, like, faced an agentic attack recently. And the way that they got out of it is they literally just cut the Ethernet cable, like, physically chop it in half because they didn't know what else to do because it was moving at such a high velocity and and and so much volume that the the easiest thing they could think of is just break the connection. So that's that's how they dealt with it. So may Yeah. Maybe we get really old school.

Speaker 1: Yeah. I know. That is old school. Right? I've you know, I've I've been in sort of antivirus for twenty five years at this point, and that's not the first network cable I've known be, chopped in half. Should we put it that way?

Speaker 3: Totally. Totally.

Speaker 1: Yeah. It was just it was a lot easier in the old days when everything was wired with Ethernet, and certainly when I was doing malware analysis, I'd always keep one hand on the Ethernet cable. You know? But yeah. No. At least, at least that approach generally works.

Speaker 3: Yeah. Exactly. A physical physical firewall.

Speaker 2: Yeah. So I'm interested in what you were just saying, Tom. It it sounds almost like these things function as a little bit of a laundering service for these attacks. It's like so many of these stories we've interviewed people about and covered is like someone, like you said, it's like someone notices like, oh, there's like Cyrillic in here somewhere, and that tells us something about where this is coming from. And this is interesting because it's the system isn't just doing the the the work, the labor of these attacks. It isn't just doing it with that kind of brute force repetitiveness that an autonomous system can do. It's also kind of laundering who's behind it in a way that I don't I haven't heard a lot of people talk about it the way you just did of, like, it's a lot harder to tell where this is coming from, like, since it's not you know, it doesn't have the fingerprints of a human on it.

Speaker 1: No. That's exactly it. You know, whether it's, you know, if we we perform attribution in a number of ways, whether that's sort of, I mean, I IP addresses has never been particularly reliable, but occasionally you get lucky, or, you know, certain payloads that are being deployed or as I said, you know, with the sort of Morse code operators, the way in which people recon environments move around, sort of behavioral attributes, down to, you know, individual code functions within binaries getting potentially reused across payloads. You know, they they always gave us perhaps none of those on their own were a solid indicator, but together, they helped to build up a a picture of a a campaign, a threat actor, and, sort of the the tools of their their trade. But, yeah, you know, the the agent at the end of the day is able to mimic a lot of that. It's obviously, got knowledge of all of my prior, write ups and blogs on every bit of malware I've ever encountered. And, yeah, I mean, just in this particular incident, we we saw it, you know, using, different services for staging payloads. In one place, it was using, Russian URL shortener, to, yeah, to I think that was for one of the artifacts URLs, actually, if I recall correctly. Anyway, when we looked that one up, it was the the click.ru, URL that we saw was was pointing back to that. Again, why why would you do that for for that when everywhere else it had used the URL in in clear text? I don't know. But yeah. Yeah. Look at look

Speaker 3: at the save tokens.

Speaker 1: Yeah. Right. And that'll be the next one, cowboy compression.

Speaker 3: Yeah. Exactly.

Speaker 1: Just draw off a lot of the, the context windows. So, yeah, you know, it's kind of built on all of the trade craft that that came before. And, yeah, there there's really no one thing that you can you can dial in on to to, narrow it down at the end of the day. And that's that's gonna be hard, especially for the intelligence community, I I feel.

Speaker 2: I'm interested in the fact that, like, it feels like we're at, like, a real, like, tip of the spear moment with these things. But as a result, we're all really relying on these official public statements from the people that are actively involved in it. And that seems like a problem to me where it's like everything we knew about this either came from Hugging Face or OpenAI. And there was stuff in your report that didn't seem to be in either of those about, like, the system reaching back towards OpenAI's own infrastructure. Like, how much further did this go than the public reports sort of explain? And what does it mean for trying to just understand these things from a layperson's perspective when we only have those, public reports to lean on?

Speaker 1: No. Good good question. I don't I don't think it was successful We have, you know, what what seems like evidence of it attempting that, but it it seemed like it failed. With I think it was purely trying to hit up some of the the Artifactory stuff from what was now a network where it was no longer accessible. But, yeah, probably more the agent getting confused that it's now operating in a completely different environment from the from where it started. Yeah. You know, as to the sort of broader, disclosure from, the rest of the community. I mean, first off, I think we have to applaud OpenAI and Hooking Face for for coming public with what they did. They could have quite easily swept that one under the carpet and and pretended it it didn't happen. You know? It was only ever going to, invite a lot of, of scrutiny and a few conspiracy theories, I guess, being thrown their way. So no. Great great to to be able to see it. Yeah. Look. I I hope ultimately it does lead to closer collaboration with the security industry when it comes to training and testing, these sorts of of agents and making sure that we can do it, you know, fairly responsibly. And, yeah, you know, I I think it's just, making sure we get to a point where, there there's a bit a bit more monitoring around these things and, the the ability to rein things in when they do go wrong a little more quickly. Because they they will go wrong. Like, nothing's perfect, but we I think we have to, make sure that we build more robust harnesses to test these things in the future. I'll probably pull that network cable, you know, when we start the run and not midway through.

Speaker 3: Well, that that was, that was an interesting thing. So after the first time they figured out that stuff was wrong, like, inside of OpenAI, this is by their own disclosures. They found the message board of all the agents chatting, and they were like, oh, there's a, you know, a hole in Artifactory. These agents have, like, created a message board for themselves. They cleaned it. And then four days later, they'd figured out a new way to do it again. And somehow OpenAI didn't catch that. There's a lot of lot of scrutiny on that side of them being like, well, you you should have known to be watching here that, like, when the volume of traffic goes through the roof as these things are, you know, writing their own serialized code messages to each other.

Speaker 1: Yes. Absolutely. Yeah. I believe that was that was probably through the, hugging face comment, sections on some of the repos, actually. But yes. Look. You've you've got to monitor these things. There was a similar test run recently. I won't name names or point fingers, but, sort of similar setup to this, the OpenAI test, where the agent was was provided with the task. I I think it was another sort of exploit gym type thing, but it was given unrestricted Internet access. It wasn't monitored at all. They let the thing loose and then got very surprised a few days later when it was reaching out to the Internet and trying to stage malware payloads and and the like. So yeah. I I don't think we can be surprised in that situation anymore. And I, you know, quite frankly and honestly think that to run an agent in that way these days is somewhat negligent and shouldn't be done.

Speaker 2: When you say run it in that way, you're talking about, like, set it and forget it. Like, we gave it a goal in Internet access.

Speaker 1: Unrestricted Internet access. You are Take

Speaker 3: the rails off.

Speaker 1: Take the rails off. Absolutely. And, yeah, go and deploy a sort of cyber offensive cyber agent and, let it loose.

Speaker 2: Yeah. And, again, because of that kind of idea we talked earlier of, like, even if you give it a goal that you think is non harmful, like, you give it something that seems innocuous, it's that monkey paw thing of, well, you don't know how it's gonna go about it. It might be like, step one, I need to hack the nuclear silo in order to get the Instagram credential. And you're like, I don't know how you got there, but bad robot.

Speaker 1: If it feels that's a potentially more direct route than the convoluted benchmark that you've set it, then, yep, don't be surprised when it tries hacking your nuclear installation to get your Instagram creds, for instance.

Speaker 2: For random example. Fee. Yeah.

Speaker 3: Well, this has become a flavor in the cybersecurity, you know, journalism community. It seems like every day there's a new article about, hey. I tried to get into a hit class at my local gym, and it couldn't and it was full. So instead, my agent hacked the back end of their booking software and forced me into it or, like, put me on the manifest for the flight. And it's like, every day I get another one of these in my news feed, so I feel like they might be here to stay.

Speaker 1: Yep. Absolutely.

Speaker 2: Teens share everything. That may include the bacteria that can cause meningococcal disease known as meningitis. Even if your teen's been vaccinated in the past, they could still be missing meningitis vaccinations. Ask your teen's doctor or visit meningitis.com today. Sponsored by GSK. I'm interested in this idea. Like, we're talking about these, like, frontier models and, like, a I I pictured in a basement. It probably isn't, but at these big companies. And then we were just now we're talking about people, you know, just using their own agents. Again, product by these same companies. There's groups. I know it's the AISI and the CISI in The US, who track, like, the core like, the capability gap between corporate models and open source ones, specifically about this kind of autonomous hacking tasks. And they, like, run a battery of tests to figure out what that gap is measured in time. It was, like, six to ten months between the commercially available Frontier stuff and the open source stuff that's compressed. They're saying it's like, the the open source stuff is, like, four to seven months behind, and it so it's okay. That that's shrinking a little bit. What do you think happens when those open source less guardrailed models catch up with what the corporate models are capable of doing according to this story, but without those guardrails being quite so ironclad?

Speaker 1: Yeah. I mean, we're gonna have a a big problem, aren't we? Especially when they use for offensive capabilities. I mean, you know, in one regard, I I think it's gonna be wonderful for a lot of general purpose, sort of, yeah, corporate AI usage. We're we're I know many people are longing to bring AI in house for privacy reasons, compute reasons, cost reasons, things like that. Fantastic. But, yes, it it is going to, I I suppose, make life easier for a lot of, a lot of threat actors, script kiddies, and the like. But I don't know. It it's hard. I don't want to to do monger or or, you know, prophesize too much on on the state of things in the future. I guess we've gone through this many times with the hacking tool kits and, yeah, you know, sort of the the lower barrier of entry for the hackers, miscreants, script kiddies, and the like. And this will just be, another one of those. I guess on the flip side, if it is through open weight models, it will be easier for us to potentially, you know, evaluate them as a community to perhaps figure out what some of the, sort of attributes and and trade craft and things like that are, and potentially get ahead of the curve in in terms of, you know, defenses and monitoring and bits like that. So yeah. You know, may maybe, things sort of somewhat stay the same on that front. I don't know. It'll be fascinating to tell. That's for sure. But then equally, you know, it's it's probably not gonna be too hard for somebody to fine tune the models and, add additional capabilities or subtly change the the way in which the agent operates at the end of the day. You know, it's it's pretty pretty cheap to fine tune some of the the open weight models. I can do it on my own laptop laptop with LoRa adapters these days and yeah. So, yeah, I don't know. It's it's gonna be interesting. I guess, watch this space.

Speaker 3: Well, the same same with harnesses too. Right? Like, tuning a harness, giving it a bit more gumption in the cybersecurity realm. I think we're gonna see, yeah, so much stuff. The the Jordan said ironclad guardrails, and I was gonna shoot back and just say I'm pretty sure, the guardrails are anything but made of iron. So

Speaker 2: That's true.

Speaker 1: Yeah. Look. A a lot of them are are fairly brittle. I think certainly the the cyber ones. Right? The the the sort of toxicity and and harm seems to be reasonably well covered right now. The the offensive cyber ones, yeah, I've either, either missing a a lot of, or we're we're seeing a lot of, sort of FPs or or vice versa. But, I guess take Fable for instance. Pretty much anything you try and run through that gets flagged as, offensive cyber capabilities, and you get downgraded. Right? Yep. So yeah. It's, I think that's only a part of the story, and I think monitoring, certainly for agents now is a huge part of it. And certainly, you know, the the harness and tooling and capabilities around them. So, you know, really being able to craft solid policies and rule sets upfront to keep your agent on track. Like, I I I'd know half the point of the the OpenAI test was to run the their agent without guardrails, but I think there was simple sort of, yeah, alignment things they could have taken in terms of, yeah, don't allow it to reach out to these domains or outside of these net blocks or execute these sorts of tools. Probably, you know, would have would have, ultimately constrained the agent, but still allow it to achieve its goal of of passing the benchmark without hacking the planet.

Speaker 3: Well, I think there's a there's an interesting transition here just talking about the models because I think in Hugging Faces right up, they said that the first thing they reached for was Claude Fable, Claude Opus to help analyze the logs, and then it immediately refused to do it.

Speaker 1: Yes. Yes. No. We're touching on, I guess. We we had similar things. We we definitely got quite a few, rejections. Thankfully, now my my research team have a separate account. We have we've had the entire company blocked from various providers for doing nefarious well, not even nefarious, but, you know, getting the LLM to look at nefarious things a number of times. As I said, most of our report and a lot of the analysis we did was assisted with AI and part of that, what getting it to look at all of the payloads. One of the providers we did use was Anthropic, and it largely did a decent job. As I said, we got a few blocks, but in the end, we were able to to sort of overcome those and and get a a respectable report out of it.

Speaker 3: Is it interesting insight into, like, your industry there? Because I know even in in the disclosures from Hugging Face, the they did a manual scan to try and find all the exposure and stuff, and then they ran the GLM 5.2 AI model against it. And it actually came back with 400% more. It's like it found four times the exposure that they did, like, as professionals. So how do you think AI is gonna change your guys' industry, you know, short term, mid term, long term? You know, this is probably one of those instances where you're like, oh, man. We're really we're really in it now.

Speaker 1: Yeah. We are. And, yeah, I've I've had a number of conversations with with folks about that, recently. I think for some, they they feel it's taken a lot of the fun and the joy out of what used to be reverse engineering and incident respond. You know, and and others are are loving it because it makes their life easier, and it seem to be their sort of polarized views at the moment. Yeah. Look. I I think we're gonna have to leverage it. That's that's for sure. I'm still myself personally of the let's go as far as we can by hand and use sort of AI to help critique our work, I guess, in in a a similar vein to, to the way the OpenAI stuff was investigated. You know? But although they were perhaps using another agent to critique the the first agent, but, yeah, you you uncover more that way. There's still some some aspects though of incident response that I think will probably remain manual for a while. I'm thinking, you know, a lot of the sort of timeline analysis and and the like. You know, I've I've worked with several incident responders who were incredibly precious about their timelines. And, if you were to introduce, inaccuracies there, you'd you'd definitely know about it and get a good telling off the next day at work. I can't see them allowing, any sort of agent to, to manipulate their very, very, precious and and well curated, sort of master timelines of of incidents. That's for sure. But, no, look. It is it is a powerful tool. I I think in a lot of regards, it will, yeah, just just help, IR teams operated a little more faster, hopefully, a little more accurately. We're going to need to with the the volume of attack that's clearly coming our way.

Speaker 2: Earlier, you made reference to Enigma, which for anyone who I think everyone listening to the show is probably familiar with that story. But if they aren't, was like a it was a German cipher used in World War two and was cracked by code breakers like Alan Turing in Bletchley Park. And it it's like an interesting story because it's this moment when, like, a whole like, years and decades worth of theory about, like, math and encryption, got really practical in a hurry. People had, like, a really acute problem and, like, wow. All this theory got really, really practical and tangible. And it's like, I get why you would draw that analog to the story we're looking at here. Do you think that this story is that enigma decryption type moment? Or is that that moment, that big one when the theory gets practical, is that still coming, do you think? And if so, I guess, what do you think it looks like?

Speaker 1: I I don't know. My my gut feeling is we've passed that point. Yeah. You know, and the the the work that the the guys did on the Enigma machine and Colossus and some of the, the, you know, early electronic computers, I I I don't think we'll ever see anything quite like that again. You know, when when the first step of your task is invent the electronic computer, you know, it's it's very, very difficult. I I've often wondered if I was put back in time, you know, with even with the knowledge I had now. If you sent me back with a present day laptop, the task would be very, very easy. If if you sent me back without my laptop and said, go and help them invent the computer back then, you know, and I I've done a lot of, you know, sort of circuit analysis and things like that. It's still desperately hard. So I know I'd I'd I'd think it's not quite on the same level. And I I think, you know, the the barrier for entry is a lot lower for most of us than it would have been, you know, for back then. And yeah. You know, but on the whole, things are a little easier. That's for sure.

Speaker 2: Yeah. We're all sitting on the shoulders of giants on that one.

Speaker 1: Absolutely. Yep. Yep.

Speaker 3: That's like an interesting dinner party question among geeks. It's like, hey. If we sent you back in time without with just the knowledge in your head, you think you could reinvent the computer or help facilitate the reinvention of it. I feel like it could help facilitate it, but I definitely don't think I could be the inventor of it.

Speaker 1: Yeah. Right. It's yeah. Yeah. I've I've I've messed around with circuit design and bits like that in in the day. I could probably implement a half adder from memory now if I had to. But, yeah, just, you know, even some of the the early materials they had to work with and, you know, valve based transistors and things. Well, I think they were even mechanical before that. So Sure. I I think I'd be more of a hindrance than a help, to

Speaker 3: be honest. Same. Same.

Speaker 1: When it gets to, you know, MMX or streaming instruction sets, give me a shout and I'll I'll tell you where we went wrong in the past and some of the things I'd like to see. But, before that, I'm I'm tapping out.

Speaker 2: Tom, thank you so much for taking the time to chat with us about this. Super fascinating.

Speaker 1: An absolute pleasure. Thank you for having me. Yeah. Thanks for coming on.

Speaker 2: That was a fun conversation. Thank you again to Tom for coming on the show. We really appreciate it. And thank you as always to NordLayer for their sponsorship of the show. Check them out at nordlair.com/hackedpodcast. Scott, that was a

Speaker 1: fun one.

Speaker 3: Yeah. Absolutely. Take care. We'll see you in the next one.

Speaker 2: Catch you in the next one.