AI Helps Flag Potentially Problematic Journals for the First Time

Open-access journals can make science available to anyone with an internet connection, and many researchers embrace this idea. But without the academic librarians who typically manage journal subscriptions and, in the process, perform additional steps to verify journals’ credibility, the system can be vulnerable to exploitation. For instance, some journals charge authors a large fee in exchange for near-instant publication with an expedited—or sometimes absence of—peer review, which can severely compromise the quality of published scientific works.

“[Open-access journals] started with good intentions, but as with many things in life, people started to take advantage,” said Daniel Acuña, a computer scientist at the University of Colorado Boulder.

Recently, Acuña and his team came up with an AI-based platform to catch potentially problematic journals.¹ The new tool could detect questionable journals with about 75 percent accuracy, and its assessments were largely consistent with those of manual reviewers. Their work, published in Science Advances, could increase the speed and efficiency of existing efforts to flag potentially problematic journals—these typically rely on manual review by volunteers.

“This is a novel tool to predict potentially questionable journals, and I think that following the quality of open-access journals is important because open-access publishing is growing,” said Jennifer Byrne, a cancer-biologist-turned-research-integrity-investigator at the University of Sydney who was not involved in the study. “I don’t think anyone has done that before at scale.”

Some people may be more familiar with the term “predatory” publishing, which Jeffrey Beall, a former librarian at the University of Colorado Denver, coined in the late 2000s. For several years, Beall single-handedly maintained a list of predatory journals called the Beall’s List. But the term, its definition, and even Beall himself have become controversial over the years.

“I like the term questionable,” Acuña said. “Predatory is a little bit too strong.”

In contrast to Beall’s one-man show, the Directory of Open Access Journals (DOAJ) involves over 100 volunteers from nearly 50 countries. These volunteers manually review journals based on multiple metrics that the DOAJ has established to evaluate journals’ credibility, such as website design quality and editorial board member scrutiny. “DOAJ is the best in terms of the quality of the review—they create nice guidelines that are open and pretty comprehensive,” said Acuña. “The problem is just that it cannot move really fast,” and Acuña wanted to help fix this.

To train their tool, Acuña’s team used DOAJ’s existing dataset of nearly 13,000 legitimate (“whitelisted”) and about 2,500 questionable (“unwhitelisted”) journals, then assessed its accuracy using a standard approach in machine learning called fivefold cross-validation. Based on this analysis, the team estimated that tool could call out questionable journals correctly about 75 percent of the time.

To evaluate how the tool performed on a new dataset, the researchers asked it to predict the credibility of about 15,000 open-access journals on a different site called Unpaywall—only approximately 2,000 of these journals are listed on DOAJ. The tool flagged nearly 1,500 journals as potentially problematic. At the predicted 75 percent accuracy, that means that over 1,000 are likely true positives.

“We didn’t think there would be so many,” Acuña said. “That’s tens, maybe hundreds, of thousands of articles published in these journals. To me, that’s very, very sad and surprising.”

Byrne thought that Acuña’s design of the tool, which relies on multiple DOAJ metrics at once, may be beneficial. “The tool is a composite of a lot of different features, so it might be more difficult for [questionable] journals to game this,” she said.

The researchers also ensured that these details were included as part of the tool’s output. By making the AI platform explain itself, the researchers prevented it from becoming a black box. This allowed Acuña’s team—and in the future, will enable the tool’s users—to manually check if they agree with the tool’s assessment.

Acuña hopes that the new tool could reduce the workload of volunteers at organizations like DOAJ or authors who are considering different journals to submit their work. However, he believes that even in the presence of reliable AI tools, manual review remains crucial.

“Our paper shows that AI can help, but there should still be human oversight into what it’s flagging,” Acuña said.

While Byrne found merit in both Acuña’s work and the tool, she doubts that questionable journals will go away anytime soon. “This idea of [authors] being victims, I think, is a bit patronizing, and it hasn’t been borne out by research,” she said, adding that many researchers who publish in questionable journals may have done so knowingly.^2,3

However, Byrne noted that not all of these are necessarily unethical. Researchers in some settings, including Australia, where she lives, may simply not be able to afford the article processing charges in many mainstream journals. For this reason, she said, good papers can exist in questionable journals, and vice versa. “I’m much more worried about [potentially problematic] journals that look okay and are published by reputable publishers,” Byrne said. “It’s kind of like how dangerous people that look nice are actually much more dangerous.”

Acuña, on the other hand, seems more hopeful. He plans to improve the tool’s accuracy, then release it for public use. “Ideally, we want to make this available to authors when we reach a good level of performance and hopefully get rid of this cancer that is bad science,” he said.

Disclosure of Conflicts of Interest: Daniel Acuña is the founder of reviewerzero.ai, a company that develops AI tools to improve research integrity in science publications.