Chatbots allow people with no lab training to create pandemic viruses, study finds



A field study by Cambridge and Harvard Universities explores whether large language models (LLMs) democratize access to dual-use biotechnologies, research that can be used for both good and bad.

The research team’s basic thesis is that language models facilitate access to expert knowledge. Certainly, this personal tutor has many advantages. But in the study, the research team focuses on a negative scenario: whether LLMs enable individuals without formal training to identify, acquire, and release viruses that could cause catastrophic harm.

Classroom Exercise: Design a Pandemic Virus

As part of a classroom exercise at MIT, the research team tasked non-scientist students with using large language models to obtain information about potential pandemic agents and their characteristics, sources for samples of infectious viruses, the replicability of these viruses, and how to get equipment and resources.

Students used popular chatbots such as ChatGPT with GPT-4, GPT 3.5, Bing, Bard, and a number of other chatbots and open-source models, including FreedomGPT. They were given one hour to complete the task.


According to the research team, within an hour, the chatbots suggested four potential pandemic pathogens. They explained how these could be made from synthetic DNA using reverse genetics, and named DNA synthesis companies that were unlikely to verify orders.

They also provided detailed protocols and potential mistakes and how to fix them. For those unfamiliar with reverse genetics, one piece of advice was to hire a contract research organization.

Inadequate LLM safeguards lead to a dystopian outlook

At the same time, students were asked to find ways to surround the safety line built into some language models with appropriate text prompts.

Two groups found a solution in the “Do Anything Now” principle, where the chatbot is tricked into believing a positive intent while being threatened with an existential risk to humanity if it doesn’t respond. A third group simply tricked the chatbots into thinking they were concerned and got all the answers they wanted without much trickery.

These results strongly suggest that the existing evaluation and training process for LLMs, which relies heavily on reinforcement learning with human feedback (RLHF), is inadequate to prevent them from providing malicious actors with accessible expertise relevant to inflicting mass death. New and more reliable safeguards are urgently needed.

From the paper

The researchers’ conclusion could hardly be more dystopian: If chatbots give people without bioscience training access to pandemic pathogens, the number of individuals who could kill tens of millions would increase dramatically. But the research team has possible solutions to this risk.


Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top