Researchers rushing to adopt AI without understanding the limitations risk committing scientific misconduct.

“AI” is everywhere in scientific research right now. The US is directing federal research funds into the Genesis Mission to “harness AI.” Top research institutions are offering classes and certificates in “agentic AI.” And many recent scientific papers contain chatbot-generated text. While researchers have been using AI and machine learning for decades, the current enthusiasm is very different. It’s about generative AI (GenAI) like ChatGPT, Claude, and Gemini. But the fundamental principle behind GenAI, statistical inference, guarantees it will always be unreliable. As important as reliability is in science, researchers need a way of clearly understanding what GenAI can—and can’t—actually do.

This post is the first in a series on GenAI in research. It reflects my perspective, based on my professional experience. For those who don’t know me, I’m a scientific consultant at CAT Lab. I help collect, prepare, and analyze our research data. We work with some very complex data and a big part of my job is making sure we maintain its scientific integrity throughout the analysis process. My background is in information science and computer science (including AI), as well as in math and physics.

GenAI favors plausible falsehoods over unlikely truths.

When thinking about GenAI, we can borrow a helpful analogy from the world of physics: the perpetual motion machine. Over the last several hundred years, many have searched for, or even claimed to have discovered, a machine that produces more energy than it consumes. Such a machine is called a “perpetual motion” or “free energy” machine. But perpetual motion violates the law of conservation of energy, one of the most fundamental and well-tested physical laws. So if you understand the underlying principles it’s obvious that no such device could ever exist, no matter how cleverly designed.

GenAI as a scientific tool is very much like perpetual motion. Scholars like computational biologist Dr. Jennifer Listgarten have made this connection before. A deeper look at the analogy is worthwhile because it can help clarify thinking about GenAI. Just as physics is built on the conservation of energy, GenAI is built on statistical inference. I’ll say more about statistical inference in a later post, but in ordinary terms, GenAI is a machine for producing statistically plausible output. And here’s the key: statistically plausible is different from correct. Expecting correctness from GenAI is like expecting perpetual motion from a machine. Gen AI has no concept of correctness. And even if it did, GenAI favors plausible falsehoods over unlikely truths. And unlikely truths are the heart of scientific discovery. 

Determining exactly how these claims are incorrect is difficult and costly. But determining if they’re incorrect is easy.

When it comes to GenAI, there’s a lot to learn from the historical claims of perpetual motion. Sometimes perpetual motion is “proved” using a complex theoretical framework. Sometimes it’s “observed” experimentally. The complex theories just bury logical fallacies under pages of equations. And the elaborate experimental apparatuses just hide measurement errors. Determining exactly how these claims are incorrect is difficult and costly. But determining if they’re incorrect is easy. If energy isn’t conserved, the claim is false. It’s the same with the latest GenAI models and apps. Regardless of the details, if the output needs to be reliably correct, GenAI can’t deliver.

Any good-faith research use of GenAI needs to include systems to verify correctness, fix errors, and mitigate harms.

What does this mean for researchers? If we’re using GenAI to analyze data or write papers, we should expect the output to misrepresent the research. GenAI will omit, alter, or fabricate data in the name of statistical plausibility. And that is the very definition of research misconduct. So any good-faith research use of GenAI needs to include systems to verify correctness, fix errors, and mitigate harms. Perhaps there is a place for GenAI in science, but it may be that once the hidden risks and costs are considered, that place is considerably narrower than is currently practiced.

In future posts, I’ll elaborate on the abilities and limitations of GenAI, and their implications for scientific integrity:

Part 1: Reliable GenAI and Perpetual Motion
Part 2: Statistical Inference and (In)correctness
Part 3: Inference Plus Guardrails
Part 4: Methodology and the Replication Crisis
Part 5: Fragile Systems and Their Consequences
Part 6: Principles for Evaluating GenAI