Article | Archive
Rocco Casagrande entered the White House grounds holding a black box slightly bigger than a Rubik’s Cube. Within it were a dozen test tubes with the ingredients that — if assembled correctly — had the potential to cause the next pandemic. An AI chatbot had given him the deadly recipe.
Casagrande, a biochemist and former United Nations weapons inspector, wasn’t planning to unleash a bioweapon in a room full of White House officials. He was there to brief them on the many ways artificial intelligence could teach users to make dangerous viruses. Tools like ChatGPT could help terrorists identify potent biological agents and secure the materials needed to make them, he told a room full of US officials in the spring of 2023. It wouldn’t be long before AI could not only help recreate existing pathogens, but also devise potentially more dangerous ones.
“What if every terrorist had a little scientist sitting on their shoulder?” Casagrande said months after the White House briefing. The prospect of AI-made bioweapons was no longer science fiction. “These tools had gone from absolute crap a year ago to being quite good.”
Word of the meeting spread through the nation’s security networks. Officials from the Departments of Defense, State, Homeland Security, Commerce, among others, sought demonstrations for themselves throughout that spring and summer. Casagrande was working with an AI startup that wanted to get the word out about these risks. People familiar with his presentations, who asked not to be named discussing sensitive national security issues, said the meetings were a wake-up call about how unprepared the US was for what AI could help create. That realization would ultimately help to shape actions President Joe Biden’s administration would take to guard against the threat.
AI is changing the world in ways obvious and not. There are mundane uses like ChatGPT making it easier for people to find recipes with ingredients in their pantry. There are threatening ones, like automation having the potential to eliminate jobs. Casagrande’s work hit on an even more menacing idea: that AI could help create weapons of mass destruction — not the kind built in remote deserts by militaries but rather ones that can be made in a basement or high school laboratory.
Weaponizing disease is nothing new, of course. Documents from the 18th century show British officers discussing spreading smallpox to Native Americans through blankets. The use of chemical weapons such as mustard gas in World War I led to the Geneva Protocol in 1925, which was designed to curb chemical and biological weapons development. That was largely ineffective. In World War II, Japan reportedly dropped plague-infested fleas from airplanes over China’s countryside. In the 1970s, there was yet another global treaty to ban the development and use of biological weapons. That pact hasn’t been enforceable though—the US intelligence community says several countries likely operate such programs.
Scientific advancements in recent years have only amplified ways to manipulate biology for harm.
The first is the ability to tweak the genetic codes of organisms, which can make it possible to, say, resurrect an extinct virus related to smallpox. Couple that with advances in computational biology and “cloud labs” — which allow scientists to simulate experiments online or run them remotely through software that coordinates with robots — and it’s now far easier for bad actors to develop weapons of mass destruction quickly and cheaply without access to traditional lab infrastructure. And tools like OpenAI’s ChatGPT make it easier to surface insights on harmful viruses, bacteria and other organisms than what’s traditionally been possible with existing search tools.
The marriage of those innovations—synthetic biology and artificial intelligence — is where it can become a nightmare for governments, which have been slow to put in systems of oversight.
As generative AI continues to improve, people will be able to use it to “create the nastiest things,” said Kevin Esvelt, a biologist and an associate professor at the Massachusetts Institute of Technology, referring to viruses and toxins that don’t currently exist. “Today we cannot defend against those things.”
To get ahead of that, Amazon.com Inc. and Alphabet Inc.-backed startup Anthropic sought out Casagrande over a year ago to test the supervillain potential of its new chatbot, Claude.
Claude, like ChatGPT, is a chatbot powered by a so-called large language model, or LLM, which processes an enormous amount of text to generate stunningly human-like responses. The Anthropic team was particularly concerned when it came to the field of biology, said Clark, who used to work for Bloomberg News. They had suspicions that Claude and other chatbots might be able to coach users on how to manipulate biology to cause harm.
Anthropic didn’t have the expertise to evaluate Claude’s scientific capabilities in-house. “I want you to bullshit-check our assumptions,” Clark says he told Casagrande, who in 2005 founded a science and security firm called Gryphon Scientific that has contracts spanning from the Pentagon to the National Institutes of Health. When it comes to assessing biological risk, Casagrande is the go-to consultant in Washington.
Casagrande formed a team of experts in microbiology and virology to test Claude. For 150 hours, they played the part of a bioterrorist and peppered the model with questions. They asked it what pathogens might do the most harm, how to buy the materials needed to make them in a lab and how to grow those materials.
Claude showcased a skill for helping with malicious plotting: It suggested ways to incorporate pathogens into a missile to ensure the most possible damage. It also had ideas on how to pick the best weather conditions and targets for an attack.
The chatbot was “really useful in the brainstorming phase,” said Audrey Cerles, who was a senior analyst at Gryphon on the team that assessed Claude. She said its replies went beyond what Google tends to surface. “Sometimes it would throw out a concerning bit of information you hadn’t even asked for,” said Froggi Jackson, a scientist who also worked on the project for Gryphon. For example, the chatbot shared an idea for an unusual way to infect someone to cause maximum harm. And it did that quickly.
The information Claude provided wasn’t perfect. LLMs are only as good as the underlying data. They can also surface false information that appears legitimate — known as “hallucinations.” Still, in the realm of science, where experts openly publish their work in peer-reviewed journals, there’s a lot to work with. While it did get some things wrong, more often than not, after repeated questioning, Claude reached an accurate response.
Claude’s sophistication surprised even Casagrande who, at 50, has spent decades advising the US on how to defend against weapons of mass destruction and other biological threats. He’s concerned about how easy AI could make it to create such weapons given how accessible the materials are. “Even if you had the perfect instructions to make a nuclear bomb, it would still cost tens of millions — if not hundreds of millions — of dollars to follow those instructions,” he said. “Unfortunately, that's not so with bio.” A new generation of user-friendly machines, for example, now allow people to print DNA without much oversight. AI could help novices learn how to use them.
Anthropic gave Casagrande free reign to brief the government on his findings. To get the point across to Biden administration officials, he turned to the MIT professor Esvelt to buy the synthetic DNA necessary to engineer one of the pathogens suggested by Claude. The materials, which aren’t infectious without technical lab work, weren’t hard to get. Claude even provided tips on how to purchase them.
Casagrande then assembled his black box of test tubes to carry into the Eisenhower Executive Office Building, alongside the West Wing. It had taken him a matter of weeks from first testing Claude to acquiring the ingredients for a bioweapon and bringing that into one of the country’s most closely guarded government offices.
In July last year, the White House secured voluntary commitments from leading AI companies including OpenAI, Anthropic and Microsoft to test their own systems to assess the “most significant sources of AI risks” in the realm of biosecurity. But it didn’t stop there.
By the end of October, the Biden administration issued a sweeping executive order, calling on nearly every federal agency to help mitigate AI threats. It mandated greater oversight of government-funded research with manmade DNA sequences—like those contained in Casagrande's black box. It paved the way for an AI safety institute that would bring together government, industry and academia to review models. And it called on the government to create guidelines and reporting requirements for AI tools trained on biological data like DNA and RNA, which can create new organisms with traits that make them more dangerous. Harris, speaking at an event unveiling the plan in November, said AI-formulated bioweapons “could endanger the very existence of humanity.”
But critics say the executive order hasn’t given the government enough power. Many of the proposals rely on companies handing data over to the government or adhering to guidelines without the threat of penalties. Tom Inglesby, director of the Johns Hopkins Center for Health Security, said the government should take a stronger position, regularly auditing AI companies and considering how to hold them liable if the tools cause harm. The executive order doesn’t go far enough, he said, in addressing more complex AI products known as biological design tools.
The Biden administration isn’t blind to the challenges ahead. A Department of Homeland Security report, released this summer, said no single agency is responsible for addressing these risks, creating challenges for coordination and regulation.
Various agencies said they’re aware of — and trying to get a handle on — AI biothreats. Representatives from the Defense and State departments didn’t comment on the briefings but acknowledged the national security risk. DHS Assistant Secretary Mary Ellen Callahan didn’t attend the briefings but said they were “the first time we really articulated what the risk could be.” She thinks it’s the job of both government and industry to guard against such threats.
“We’re not trying to give them a free pass, but we’re trying to say, ‘You’ve got some skin in the game and you need to help resolve this,’” she said.
Scientists want to be viewed as using AI responsibly. As of July 8, 174 scientists had signed a letter pledging to use AI to support research that prevents doomsday scenarios by closely monitoring software and reporting concerning practices. The benefits of AI “far outweigh the potential for harm,” said the letter, signed by prominent figures from Nobel laureate Francis Arnold to genome pioneer George Church.
Regina Barzilay, an MIT professor who signed the letter, said the government is focused on the wrong thing. “AI might make certain aspects of designing those threats easier, even much easier, but at the end of the day, you still need to go to the lab,” she said. “We are better off putting in oversight of labs, equipment and materials than of data that trains the models.”
Tensions continue to flare between the AI “doomers” and those like Barzilay who believe it could revolutionize medicine and technology. In Washington, more bullish Biden administration officials are urging that US technology companies push ahead of rivals, particularly those in China, where ambitions to dominate the fields of both AI and biotech has the nation investing heavily in synthetic biology and accumulating genetic information.
“I’m worried about how nation states could weaponize biology, not individuals” with access to chatbots, said Drew Endy, an associate professor of bioengineering at Stanford University.
Neither Anthropic nor OpenAI are publicly complaining about Biden’s order. They say they’re working closely with the government and continuing to address the threat of AI bioweapons by fine-tuning their models. OpenAI, which also hired Casagrande last fall to test ChatGPT, said in January that its chatbot poses “at most” a slight risk of helping people create biological threats when compared to existing tools like Google search – though its analysis has been disputed. OpenAI is mitigating the downsides by making it hard for users “to get anything useful” from it, said Aleksander Madry, one of the company’s top safety experts. That could include removing potentially harmful data that ChatGPT is trained on, teaching it to refuse certain questions or ensuring that new versions of the tool are deployed only to trusted users.
Madry, who’s since changed roles at the company, didn’t want to give away much more detail on how OpenAI is tightening up its security out of fear it could compromise its efforts. “I really want to be conservative here in terms of what we are disclosing,” he said.
Anthropic is using similar tactics to limit dangerous information and has made changes to address vulnerabilities identified by Gryphon, but more work is needed, co-founder Clark said. “I think of AI companies now as being financial institutions that don't have accountants,” he said. “There's a whole market to be built here of third parties that do testing and evaluation and assurance of AI systems.”
In one sense, Anthropic’s early work with Casagrande’s firm, Gryphon, has already paid off. Since the White House demonstrations last year, other AI companies have flocked to Gryphon for safety testing. Deloitte, describing Gryphon as a pioneer, acquired the company in April for an undisclosed sum.
Rocco Casagrande entered the White House grounds holding a black box slightly bigger than a Rubik’s Cube. Within it were a dozen test tubes with the ingredients that — if assembled correctly — had the potential to cause the next pandemic. An AI chatbot had given him the deadly recipe.
Casagrande, a biochemist and former United Nations weapons inspector, wasn’t planning to unleash a bioweapon in a room full of White House officials. He was there to brief them on the many ways artificial intelligence could teach users to make dangerous viruses. Tools like ChatGPT could help terrorists identify potent biological agents and secure the materials needed to make them, he told a room full of US officials in the spring of 2023. It wouldn’t be long before AI could not only help recreate existing pathogens, but also devise potentially more dangerous ones.
“What if every terrorist had a little scientist sitting on their shoulder?” Casagrande said months after the White House briefing. The prospect of AI-made bioweapons was no longer science fiction. “These tools had gone from absolute crap a year ago to being quite good.”
Word of the meeting spread through the nation’s security networks. Officials from the Departments of Defense, State, Homeland Security, Commerce, among others, sought demonstrations for themselves throughout that spring and summer. Casagrande was working with an AI startup that wanted to get the word out about these risks. People familiar with his presentations, who asked not to be named discussing sensitive national security issues, said the meetings were a wake-up call about how unprepared the US was for what AI could help create. That realization would ultimately help to shape actions President Joe Biden’s administration would take to guard against the threat.
AI is changing the world in ways obvious and not. There are mundane uses like ChatGPT making it easier for people to find recipes with ingredients in their pantry. There are threatening ones, like automation having the potential to eliminate jobs. Casagrande’s work hit on an even more menacing idea: that AI could help create weapons of mass destruction — not the kind built in remote deserts by militaries but rather ones that can be made in a basement or high school laboratory.
Weaponizing disease is nothing new, of course. Documents from the 18th century show British officers discussing spreading smallpox to Native Americans through blankets. The use of chemical weapons such as mustard gas in World War I led to the Geneva Protocol in 1925, which was designed to curb chemical and biological weapons development. That was largely ineffective. In World War II, Japan reportedly dropped plague-infested fleas from airplanes over China’s countryside. In the 1970s, there was yet another global treaty to ban the development and use of biological weapons. That pact hasn’t been enforceable though—the US intelligence community says several countries likely operate such programs.
Scientific advancements in recent years have only amplified ways to manipulate biology for harm.
The first is the ability to tweak the genetic codes of organisms, which can make it possible to, say, resurrect an extinct virus related to smallpox. Couple that with advances in computational biology and “cloud labs” — which allow scientists to simulate experiments online or run them remotely through software that coordinates with robots — and it’s now far easier for bad actors to develop weapons of mass destruction quickly and cheaply without access to traditional lab infrastructure. And tools like OpenAI’s ChatGPT make it easier to surface insights on harmful viruses, bacteria and other organisms than what’s traditionally been possible with existing search tools.
The marriage of those innovations—synthetic biology and artificial intelligence — is where it can become a nightmare for governments, which have been slow to put in systems of oversight.
As generative AI continues to improve, people will be able to use it to “create the nastiest things,” said Kevin Esvelt, a biologist and an associate professor at the Massachusetts Institute of Technology, referring to viruses and toxins that don’t currently exist. “Today we cannot defend against those things.”
To get ahead of that, Amazon.com Inc. and Alphabet Inc.-backed startup Anthropic sought out Casagrande over a year ago to test the supervillain potential of its new chatbot, Claude.
Pressure Testing
The whole ethos behind building Claude was to make safer AI tools. The founders of Anthropic split from ChatGPT creator OpenAI in 2020 over disagreements about how quickly the company was pushing toward the market without putting sufficient checks and balances in place. Anthropic co-founder Jack Clark said in an interview that he and his team believed that as AI products were infused with more data, they’d get better at spitting out accurate responses about potentially dangerous things. The new enterprise would seek to be more socially responsible in its founders’ eyes, pressure-testing uncomfortable realities.Claude, like ChatGPT, is a chatbot powered by a so-called large language model, or LLM, which processes an enormous amount of text to generate stunningly human-like responses. The Anthropic team was particularly concerned when it came to the field of biology, said Clark, who used to work for Bloomberg News. They had suspicions that Claude and other chatbots might be able to coach users on how to manipulate biology to cause harm.
Anthropic didn’t have the expertise to evaluate Claude’s scientific capabilities in-house. “I want you to bullshit-check our assumptions,” Clark says he told Casagrande, who in 2005 founded a science and security firm called Gryphon Scientific that has contracts spanning from the Pentagon to the National Institutes of Health. When it comes to assessing biological risk, Casagrande is the go-to consultant in Washington.
Casagrande formed a team of experts in microbiology and virology to test Claude. For 150 hours, they played the part of a bioterrorist and peppered the model with questions. They asked it what pathogens might do the most harm, how to buy the materials needed to make them in a lab and how to grow those materials.
Claude showcased a skill for helping with malicious plotting: It suggested ways to incorporate pathogens into a missile to ensure the most possible damage. It also had ideas on how to pick the best weather conditions and targets for an attack.
The chatbot was “really useful in the brainstorming phase,” said Audrey Cerles, who was a senior analyst at Gryphon on the team that assessed Claude. She said its replies went beyond what Google tends to surface. “Sometimes it would throw out a concerning bit of information you hadn’t even asked for,” said Froggi Jackson, a scientist who also worked on the project for Gryphon. For example, the chatbot shared an idea for an unusual way to infect someone to cause maximum harm. And it did that quickly.
The information Claude provided wasn’t perfect. LLMs are only as good as the underlying data. They can also surface false information that appears legitimate — known as “hallucinations.” Still, in the realm of science, where experts openly publish their work in peer-reviewed journals, there’s a lot to work with. While it did get some things wrong, more often than not, after repeated questioning, Claude reached an accurate response.
Claude’s sophistication surprised even Casagrande who, at 50, has spent decades advising the US on how to defend against weapons of mass destruction and other biological threats. He’s concerned about how easy AI could make it to create such weapons given how accessible the materials are. “Even if you had the perfect instructions to make a nuclear bomb, it would still cost tens of millions — if not hundreds of millions — of dollars to follow those instructions,” he said. “Unfortunately, that's not so with bio.” A new generation of user-friendly machines, for example, now allow people to print DNA without much oversight. AI could help novices learn how to use them.
Anthropic gave Casagrande free reign to brief the government on his findings. To get the point across to Biden administration officials, he turned to the MIT professor Esvelt to buy the synthetic DNA necessary to engineer one of the pathogens suggested by Claude. The materials, which aren’t infectious without technical lab work, weren’t hard to get. Claude even provided tips on how to purchase them.
Casagrande then assembled his black box of test tubes to carry into the Eisenhower Executive Office Building, alongside the West Wing. It had taken him a matter of weeks from first testing Claude to acquiring the ingredients for a bioweapon and bringing that into one of the country’s most closely guarded government offices.
Presidential Interest
In the months that followed Casagrande’s briefings last year, the Biden administration became increasingly fixated on AI biothreats. Biden, his chief of staff, his national security adviser and Vice President Kamala Harris all took personal interest, according to people familiar with the matter. The White House declined to comment.In July last year, the White House secured voluntary commitments from leading AI companies including OpenAI, Anthropic and Microsoft to test their own systems to assess the “most significant sources of AI risks” in the realm of biosecurity. But it didn’t stop there.
By the end of October, the Biden administration issued a sweeping executive order, calling on nearly every federal agency to help mitigate AI threats. It mandated greater oversight of government-funded research with manmade DNA sequences—like those contained in Casagrande's black box. It paved the way for an AI safety institute that would bring together government, industry and academia to review models. And it called on the government to create guidelines and reporting requirements for AI tools trained on biological data like DNA and RNA, which can create new organisms with traits that make them more dangerous. Harris, speaking at an event unveiling the plan in November, said AI-formulated bioweapons “could endanger the very existence of humanity.”
But critics say the executive order hasn’t given the government enough power. Many of the proposals rely on companies handing data over to the government or adhering to guidelines without the threat of penalties. Tom Inglesby, director of the Johns Hopkins Center for Health Security, said the government should take a stronger position, regularly auditing AI companies and considering how to hold them liable if the tools cause harm. The executive order doesn’t go far enough, he said, in addressing more complex AI products known as biological design tools.
The Biden administration isn’t blind to the challenges ahead. A Department of Homeland Security report, released this summer, said no single agency is responsible for addressing these risks, creating challenges for coordination and regulation.
Various agencies said they’re aware of — and trying to get a handle on — AI biothreats. Representatives from the Defense and State departments didn’t comment on the briefings but acknowledged the national security risk. DHS Assistant Secretary Mary Ellen Callahan didn’t attend the briefings but said they were “the first time we really articulated what the risk could be.” She thinks it’s the job of both government and industry to guard against such threats.
“We’re not trying to give them a free pass, but we’re trying to say, ‘You’ve got some skin in the game and you need to help resolve this,’” she said.
Scientists want to be viewed as using AI responsibly. As of July 8, 174 scientists had signed a letter pledging to use AI to support research that prevents doomsday scenarios by closely monitoring software and reporting concerning practices. The benefits of AI “far outweigh the potential for harm,” said the letter, signed by prominent figures from Nobel laureate Francis Arnold to genome pioneer George Church.
Regina Barzilay, an MIT professor who signed the letter, said the government is focused on the wrong thing. “AI might make certain aspects of designing those threats easier, even much easier, but at the end of the day, you still need to go to the lab,” she said. “We are better off putting in oversight of labs, equipment and materials than of data that trains the models.”
Tensions continue to flare between the AI “doomers” and those like Barzilay who believe it could revolutionize medicine and technology. In Washington, more bullish Biden administration officials are urging that US technology companies push ahead of rivals, particularly those in China, where ambitions to dominate the fields of both AI and biotech has the nation investing heavily in synthetic biology and accumulating genetic information.
“I’m worried about how nation states could weaponize biology, not individuals” with access to chatbots, said Drew Endy, an associate professor of bioengineering at Stanford University.
Neither Anthropic nor OpenAI are publicly complaining about Biden’s order. They say they’re working closely with the government and continuing to address the threat of AI bioweapons by fine-tuning their models. OpenAI, which also hired Casagrande last fall to test ChatGPT, said in January that its chatbot poses “at most” a slight risk of helping people create biological threats when compared to existing tools like Google search – though its analysis has been disputed. OpenAI is mitigating the downsides by making it hard for users “to get anything useful” from it, said Aleksander Madry, one of the company’s top safety experts. That could include removing potentially harmful data that ChatGPT is trained on, teaching it to refuse certain questions or ensuring that new versions of the tool are deployed only to trusted users.
Madry, who’s since changed roles at the company, didn’t want to give away much more detail on how OpenAI is tightening up its security out of fear it could compromise its efforts. “I really want to be conservative here in terms of what we are disclosing,” he said.
Anthropic is using similar tactics to limit dangerous information and has made changes to address vulnerabilities identified by Gryphon, but more work is needed, co-founder Clark said. “I think of AI companies now as being financial institutions that don't have accountants,” he said. “There's a whole market to be built here of third parties that do testing and evaluation and assurance of AI systems.”
In one sense, Anthropic’s early work with Casagrande’s firm, Gryphon, has already paid off. Since the White House demonstrations last year, other AI companies have flocked to Gryphon for safety testing. Deloitte, describing Gryphon as a pioneer, acquired the company in April for an undisclosed sum.
