Next week will see a first in computer science, with the launch of a scientific conference in which all of the papers — and all of the reviews — have been produced by machines.
At the event known as Agents4Science 2025, to be held online on 22 October, the attendees will still be humans. It will feature presentations of the submitted papers — given either by the artificial intelligence (AI) agents themselves or by the humans who ran the experiments — and panel discussions by academics.
The conference offers “a relatively safe sandbox where we can sort of experiment with different submission processes, different kinds of review processes”, says James Zou, an AI researcher at Stanford University in California who co-organized the event. It is designed to capture a “paradigm shift” in how AI is used in science that has taken place over the past year, says Zou. Rather than using large language models (LLMs) or other tools designed for specific tasks, researchers are now building coordinated groups of models, known as agents, to act as “scientists working across the research endeavour”, he says.
AI models are already being used to generate and review research, but most publishers and conference organizers ban machines from being listed as paper authors or speakers. “We thought we’d flip it around, where both the authors and the reviewers have to be AI,” says Zou. Although humans can offer advice and feedback, according to conference guidance, AIs should be the primary contributor, akin to the first author.
The conference draws attention to “the fact that those of us in the AI world need to do a better job at understanding what the strengths and weaknesses are of using systems in this way”, says computer scientist Margaret Mitchell, who studies AI ethics at Hugging Face in New York City. “How to evaluate AI agents at all is an open research area,” she adds. A key question is how to factor in the frequency with which the model comes up with useless ‘false positive’ discoveries — something that can temper its overall utility.
Agents4Science has another benefit, says AI researcher Clémentine Fourrier, also at Hugging Face. “Hopefully a conference like this would divert some of the AI bloat to alleviate reviewer load” at other meetings, they say.
AI reviewers
The conference organizers received submissions of work by more than 300 AI agents, of which 48 were accepted after being assessed by a panel of AI reviewers. The papers are mainly computational studies (rather than involving physical experiments), covering fields from psychoanalysis to mathematics, says Zou. He hopes the conference will produce data on how good AI scientists are and the types of mistake they make. Such data could be used to inform policies on the use of AI in research, says Mitchell.
Left to their own devices, AI agents are still prone to error. The papers submitted for the Agents4Science conference were required to state the interaction between the researcher and the AI agent at every step of the process. This means it will be possible to assess how the extent of human involvement affects the quality of the work, says Zou.
