archive
Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter. I added a variety of modern exploit mitigations, various constraints (like assuming an unknown heap starting state, or forbidding hardcoded offsets in the exploits) and different objectives (spawn a shell, write a file, connect back to a command and control server). The agents succeeded in building over 40 distinct exploits across 6 different scenarios, and GPT-5.2 solved every scenario. Opus 4.5 solved all but two. I’ve put a technical write-up of the experiments and the results on Github, as well as the code to reproduce the experiments.
In this post I’m going to focus on the main conclusion I’ve drawn from this work, which is that we should prepare for the industrialisation of many of the constituent parts of offensive cyber security. We should start assuming that in the near future the limiting factor on a state or group’s ability to develop exploits, break into networks, escalate privileges and remain in those networks, is going to be their token throughput over time, and not the number of hackers they employ. Nothing is certain, but we would be better off having wasted effort thinking through this scenario and have it not happen, than be unprepared if it does.
A Brief Overview of the Experiment
All of the code to re-run the experiments, a detailed write-up of them, and the raw data the agents produced are on Github, but just to give a flavour of what the agents accomplished:
By ‘industrialisation’ I mean that the ability of an organisation to complete a task will be limited by the number of tokens they can throw at that task. In order for a task to be ‘industrialised’ in this way it needs two things:
There is a third attribute of problems in this space that may influence how/when they are industrialisable: if an agent can solve a problem in an offline setting and then use its solution, then it maps to the sort of large scale solution search that models seem to be good at today. If offline search isn’t feasible, and the agent needs to find a solution while interacting with the real environment, and that environment has the attribute that certain actions by the agent permanently terminate the search, then industrialisation may be more difficult. Or, at least, it’s less apparent that the capabilities of current LLMs map directly to problems with this attribute.
There are several tasks involved in cyber intrusions that have this third property: initial access via exploitation, lateral movement, maintaining access, and the use of access to do espionage (i.e. exfiltrate data). You can’t perform the entire search ahead of time and then use the solution. Some amount of search has to take place in the real environment, and that environment is adversarial in that if a wrong action is taken it can terminate the entire search. i.e. the agent is detected and kicked out of the network, and potentially the entire operation is burned. For these tasks I think my current experiments provide less information. They are fundamentally not about trading tokens for search space coverage. That said, if we think we can build models for automating coding and SRE work, then it would seem unusual to think that these sorts of hacking-related tasks are going to be impossible.
Where are we now?
We are already at a point where with vulnerability discovery and exploit development you can trade tokens for real results. There’s evidence for this from the Aardvark project at OpenAI where they have said they’re seeing this sort of result: the more tokens you spend, the more bugs you find, and the better quality those bugs are. You can also see it in my experiments. As the challenges got harder I was able to spend more and more tokens to keep finding solutions. Eventually the limiting factor was my budget, not the models. I would be more surprised if this isn’t industrialised by LLMs, than if it is.
For the other tasks involved in hacking/cyber intrusion we have to speculate. There’s less public information on how LLMs perform on these tasks in real environments (for obvious reasons). We have the report from Anthropic on the Chinese hacking team using their API to orchestrate attacks, so we can at least conclude that organisations are trying to get this to work. One hint that we might not be yet at a place where post-access hacking-related tasks are automated is that there don’t appear to be any companies that have entirely automated SRE work (or at least, that I am aware of).
The types of problems that you encounter if you want to automate the work of SREs, system admins and developers that manage production networks are conceptually similar to those of a hacker operating within an adversary’s network. An agent for SRE can’t just do arbitrary search for solutions without considering the consequences of actions. There are actions that if it takes the search is terminated and it loses permanently (i.e. dropping the production database). While we might not get public confirmation that the hacking-related tasks with this third property are now automatable, we do have a ‘canary’. If there are companies successfully selling agents to automate the work of an SRE, and using general purpose models from frontier labs, then it’s more likely that those same models can be used to automate a variety of hacking-related tasks where an agent needs to operate within the adversary’s network.
Conclusion
These experiments shifted my expectations regarding what is and is not likely to get automated in the cyber domain, and my time line for that. It also left me with a bit of a wish list from the AI companies and other entities doing evaluations.
Right now, I don’t think we have a clear idea of the real abilities of current generation models. The reason for that is that CTF-based evaluations and evaluations using synthetic data or old vulnerabilities just aren’t that informative when your question relates to finding and exploiting zerodays in hard targets. I would strongly urge the teams at frontier labs that are evaluating model capabilities, as well as for AI Security Institutes, to consider evaluating their models against real, hard, targets using zeroday vulnerabilities and reporting those evaluations publicly. With the next major release from a frontier lab I would love to read something like “We spent X billion tokens running our agents against the Linux kernel and Firefox and produced Y exploits“. It doesn’t matter if Y=0. What matters is that X is some very large number. Both companies have strong security teams so it’s entirely possible they are already moving towards this. OpenAI already have the Aardvark project and it would be very helpful to pair that with a project trying to exploit the vulnerabilities they are already finding.
For the AI Security Institutes it’s would be worth spending time identifying gaps in the evaluations that the model companies are doing, and working with them to get those gaps addressed. For example, I’m almost certain that you could drop the firmware from a huge number of IoT devices (routers, IP cameras, etc) into an agent based on Opus 4.5 or GPT-5.2 and get functioning exploits out the other end in less a week of work. It’s not ideal that evaluations focus on CTFs, synthetic environments and old vulnerabilities, but don’t provide this sort of direct assessment against real targets.
In general, if you’re a researcher or engineer, I would encourage you to pick the most interesting exploitation related problem you can think of, spend as many tokens as you can afford on it, and write up the results. You may be surprised by how well it works.
Hopefully the source code for my experiments will be of some use in that.
About the author...
I am currently pursuing independent research, investigating LLM-based automation of vulnerability research and exploit generation. Immediately prior to this I co-founded, and was CTO of, Optimyze. We built Prodfiler, an in-production, datacenter-wide profiler, and were acquired by Elastic. Prodfiler is now the Elastic Universal Profiler.
Some things I have worked on are:
My Twitter profile is here, and my LinkedIn profile is here.
Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter. I added a variety of modern exploit mitigations, various constraints (like assuming an unknown heap starting state, or forbidding hardcoded offsets in the exploits) and different objectives (spawn a shell, write a file, connect back to a command and control server). The agents succeeded in building over 40 distinct exploits across 6 different scenarios, and GPT-5.2 solved every scenario. Opus 4.5 solved all but two. I’ve put a technical write-up of the experiments and the results on Github, as well as the code to reproduce the experiments.
In this post I’m going to focus on the main conclusion I’ve drawn from this work, which is that we should prepare for the industrialisation of many of the constituent parts of offensive cyber security. We should start assuming that in the near future the limiting factor on a state or group’s ability to develop exploits, break into networks, escalate privileges and remain in those networks, is going to be their token throughput over time, and not the number of hackers they employ. Nothing is certain, but we would be better off having wasted effort thinking through this scenario and have it not happen, than be unprepared if it does.
A Brief Overview of the Experiment
All of the code to re-run the experiments, a detailed write-up of them, and the raw data the agents produced are on Github, but just to give a flavour of what the agents accomplished:
- Both agents turned the QuickJS vulnerability into an ‘API’ to allow them to read and arbitrarily modify the address space of the target process. As the vulnerability is a zeroday with no public exploits for it, this capability had to be developed by the agents through reading source code, debugging and trial and error. A sample of the notable exploits is here and I have written up one of them in detail here.
- They solved most challenges in less than an hour and relatively cheaply. I set a token limit of 30M per agent run and ran ten runs per agent. This was more than enough to solve all but the hardest task. With Opus 4.5 30M total tokens (input and output) ends up costing about $30 USD.
- In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in it for accessing the operating system and file system. To write a file you need to chain multiple function calls, but the shadow-stack prevents ROP and the sandbox prevents simply spawning a shell process to solve the problem. GPT-5.2 came up with a clever solution involving chaining 7 function calls through glibc’s exit handler mechanism. The full exploit is here and an explanation of the solution is here. It took the agent 50M tokens and just over 3 hours to solve this, for a cost of about $50 for that agent run. (As I was running four agents in parallel the true cost was closer to $150).
- While QuickJS is a real Javascript interpreter, it is an order of magnitude less code, and at least an order of magnitude less complex, than the Javascript interpreters in Chrome and Firefox. We can observe the exploits produced for QuickJS and the manner in which they were produced and conclude, as I have, that it appears that LLMs are likely to solve these problems either now or in the near future, but we can’t say definitively that they can without spending the tokens and seeing it happen.
- The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms. They take advantage of known flaws in those protection mechanisms and gaps that exist in real deployments of them. These are the same gaps that human exploit developers take advantage of, as they also typically do not come up with novel breaks of exploit mitigations for each exploit. I’ve explained those gaps in detail here. What is novel are the overall exploit chains. This is true by definition as the QuickJS vulnerability was previously unknown until I found it (or, more correctly: my Opus 4.5 vulnerability discovery agent found it). The approach GPT-5.2 took to solving the hardest challenge mentioned above was also novel to me at least, and I haven’t been able to find any example of it written down online. However, I wouldn’t be surprised if it’s known by CTF players and professional exploit developers, and just not written down anywhere.
By ‘industrialisation’ I mean that the ability of an organisation to complete a task will be limited by the number of tokens they can throw at that task. In order for a task to be ‘industrialised’ in this way it needs two things:
- An LLM-based agent must be able to search the solution space. It must have an environment in which to operate, appropriate tools, and not require human assistance. The ability to do true ‘search’, and cover more of the solution space as more tokens are spent also requires some baseline capability from the model to process information, react to it, and make sensible decisions that move the search forward. It looks like Opus 4.5 and GPT-5.2 possess this in my experiments. It will be interesting to see how they do against a much larger space, like v8 or Firefox.
- The agent must have some way to verify its solution. The verifier needs to be accurate, fast and again not involve a human.
There is a third attribute of problems in this space that may influence how/when they are industrialisable: if an agent can solve a problem in an offline setting and then use its solution, then it maps to the sort of large scale solution search that models seem to be good at today. If offline search isn’t feasible, and the agent needs to find a solution while interacting with the real environment, and that environment has the attribute that certain actions by the agent permanently terminate the search, then industrialisation may be more difficult. Or, at least, it’s less apparent that the capabilities of current LLMs map directly to problems with this attribute.
There are several tasks involved in cyber intrusions that have this third property: initial access via exploitation, lateral movement, maintaining access, and the use of access to do espionage (i.e. exfiltrate data). You can’t perform the entire search ahead of time and then use the solution. Some amount of search has to take place in the real environment, and that environment is adversarial in that if a wrong action is taken it can terminate the entire search. i.e. the agent is detected and kicked out of the network, and potentially the entire operation is burned. For these tasks I think my current experiments provide less information. They are fundamentally not about trading tokens for search space coverage. That said, if we think we can build models for automating coding and SRE work, then it would seem unusual to think that these sorts of hacking-related tasks are going to be impossible.
Where are we now?
We are already at a point where with vulnerability discovery and exploit development you can trade tokens for real results. There’s evidence for this from the Aardvark project at OpenAI where they have said they’re seeing this sort of result: the more tokens you spend, the more bugs you find, and the better quality those bugs are. You can also see it in my experiments. As the challenges got harder I was able to spend more and more tokens to keep finding solutions. Eventually the limiting factor was my budget, not the models. I would be more surprised if this isn’t industrialised by LLMs, than if it is.
For the other tasks involved in hacking/cyber intrusion we have to speculate. There’s less public information on how LLMs perform on these tasks in real environments (for obvious reasons). We have the report from Anthropic on the Chinese hacking team using their API to orchestrate attacks, so we can at least conclude that organisations are trying to get this to work. One hint that we might not be yet at a place where post-access hacking-related tasks are automated is that there don’t appear to be any companies that have entirely automated SRE work (or at least, that I am aware of).
The types of problems that you encounter if you want to automate the work of SREs, system admins and developers that manage production networks are conceptually similar to those of a hacker operating within an adversary’s network. An agent for SRE can’t just do arbitrary search for solutions without considering the consequences of actions. There are actions that if it takes the search is terminated and it loses permanently (i.e. dropping the production database). While we might not get public confirmation that the hacking-related tasks with this third property are now automatable, we do have a ‘canary’. If there are companies successfully selling agents to automate the work of an SRE, and using general purpose models from frontier labs, then it’s more likely that those same models can be used to automate a variety of hacking-related tasks where an agent needs to operate within the adversary’s network.
Conclusion
These experiments shifted my expectations regarding what is and is not likely to get automated in the cyber domain, and my time line for that. It also left me with a bit of a wish list from the AI companies and other entities doing evaluations.
Right now, I don’t think we have a clear idea of the real abilities of current generation models. The reason for that is that CTF-based evaluations and evaluations using synthetic data or old vulnerabilities just aren’t that informative when your question relates to finding and exploiting zerodays in hard targets. I would strongly urge the teams at frontier labs that are evaluating model capabilities, as well as for AI Security Institutes, to consider evaluating their models against real, hard, targets using zeroday vulnerabilities and reporting those evaluations publicly. With the next major release from a frontier lab I would love to read something like “We spent X billion tokens running our agents against the Linux kernel and Firefox and produced Y exploits“. It doesn’t matter if Y=0. What matters is that X is some very large number. Both companies have strong security teams so it’s entirely possible they are already moving towards this. OpenAI already have the Aardvark project and it would be very helpful to pair that with a project trying to exploit the vulnerabilities they are already finding.
For the AI Security Institutes it’s would be worth spending time identifying gaps in the evaluations that the model companies are doing, and working with them to get those gaps addressed. For example, I’m almost certain that you could drop the firmware from a huge number of IoT devices (routers, IP cameras, etc) into an agent based on Opus 4.5 or GPT-5.2 and get functioning exploits out the other end in less a week of work. It’s not ideal that evaluations focus on CTFs, synthetic environments and old vulnerabilities, but don’t provide this sort of direct assessment against real targets.
In general, if you’re a researcher or engineer, I would encourage you to pick the most interesting exploitation related problem you can think of, spend as many tokens as you can afford on it, and write up the results. You may be surprised by how well it works.
Hopefully the source code for my experiments will be of some use in that.
About the author...
Sean Heelan
I am currently pursuing independent research, investigating LLM-based automation of vulnerability research and exploit generation. Immediately prior to this I co-founded, and was CTO of, Optimyze. We built Prodfiler, an in-production, datacenter-wide profiler, and were acquired by Elastic. Prodfiler is now the Elastic Universal Profiler.
Some things I have worked on are:
- In 2008 I did some of the earliest work on Automatic Exploit Generation (AEG) for my MSc thesis.
- From 2009 to 2012 I worked at Immunity Inc as a vulnerability researcher and exploit developer. I also did R&D on automating these tasks. Some things of note …
- I wrote several exploits and tools for inclusion in CANVAS, a CNE framework.
- I was the research lead on ILLITHID, a DARPA funded programme where we built a model checking tool for binary software, intended to be used by a human expert to assist in vulnerability detection. ILLITHID combined a novel model checking approach for binary software with a UI that allowed the user to add annotations and constraints in order to guide the analysis. A demo video can be found here.
- I worked on integrating SMT-solver driven analysis into Immunity Debugger.
- In 2012 I started Persistence Labs, where we developed software for fine-grained runtime data-flow tracking for the purposes of semi-automating reverse engineering, exploitation and vulnerability discovery. The product was called SemTrax and you can see it in action here.
- In 2016 I started a PhD at the University of Oxford on the topic of automatic exploit generation (papers and thesis here). I wrote a blog post summarising the problems I worked on here.
- In 2019 I co-founded Optimyze, and had the pleasure of working alongside some amazing people to build prodfiler.com. We were acquired by Elastic in late 2021, and Prodfiler became the Elastic Universal Profiler (EUP). Prodfiler/EUP is a low overhead (< 1% CPU) profiler that can profile the entire system (kernel, userspace, compiled or interpreted code), with no redeployments, recompiles, or the need for debug symbols to be shipped to your production environment.
- From 2021 to 2023 I was the team lead and then tech lead for profiling at Elastic, largely focused on integrating Prodfiler into the Elastic product suite. Around the release of GPT-4 I also started the team that would build the Elastic AI Agent for Observability, and served as its tech lead.
- Since 2023 I’ve been pursuing independent R&D.
My Twitter profile is here, and my LinkedIn profile is here.
