Statistical Inference and (In)correctness

GenAI’s deceptively plausible outputs make it dangerously easy to overlook its limitations.

This is the second post in a series on GenAI in research. It reflects my professional perspective, based on my experience as a scientific consultant with a background in information science, computer science, and physics.

What do researchers need to know about Generative AI (GenAI) to understand its limitations? Statistical inference is the common foundation of modern GenAI, so that’s a good place to start.

To have any hope of success, there needs to be a predictable relationship between the known data and the unknown data.

Statistical inference is a mathematical tool for using known data to guess, or infer, unknown data. For example, guessing future plant growth based on nutrient levels. To have any hope of success, there needs to be a predictable relationship between the known data and the unknown data. Statistical inference works by creating a model of that relationship. Not a perfect model, but one close enough to make a good guess most of the time.

How do you create the model? By knowing the answer already—at least for some of the data. Statistical models are “trained” on examples where the known and “unknown” data are both actually known. In particular, GenAI typically uses neural networks (also called deep learning).

The weights are adjusted until the training input produces guesses that most closely match the training output, at least on average.

Neural networks, despite the name, do not work like human brains. They work by multiplying known input data by numbers called “weights” and adding up the result to get a set of outputs. The outputs are then fed into another layer as the inputs and the process repeats until a final output is produced. To train such a model, the weights are adjusted until the training input produces guesses that most closely match the training output, on average. The “on average” is important. Statistical model output is a little bit wrong most of the time, and mostly wrong a little bit of the time, because that’s how you get the best average result.

Dr. Emily M. Bender has described these systems as “stochastic parrots.”

GenAIs are statistical models trained on extremely large data sets. For this reason, linguist Dr. Emily Bender has described these systems as “stochastic parrots.” “Stochastic” meaning statistical, and “parrot” meaning to repeat without understanding. There has been a lot of discussion (and misunderstanding) of the idea of GenAI as a stochastic parrot, and most of it has focused on the “parrot” part. But let’s instead focus on the “stochastic” part. There’s no debate that GenAI is based on statistical inference, and that’s enough to understand some its fundamental limitations.

A system that can’t distinguish between correct and incorrect statements cannot produce a correct statement, except by accident.

Very literally, GenAI output is a function of its training data, prompt input, and some randomness. GenAI can’t magically divine information it wasn’t given. For any problem, producing a correct output requires at least enough information to evaluate correctness. A system that can’t distinguish between correct and incorrect statements cannot produce a correct statement, except by accident. But the requirements for a correct solution don’t live in the training data. They live in the mind of the human posing the problem, and the world around them. Operators attempt to communicate their needs to GenAI through prompts, but if operators aren’t already experts, they don’t know which information needs to be provided. GenAI is being promoted as a tool that can guess the details of the operator’s needs and produce an output that meets those needs, but that is fundamentally, mathematically impossible.

The plausibility of GenAI’s output hides its fundamental inability to differentiate correct from incorrect.

But GenAI is very good at producing plausible output that looks correct, at least superficially. The output is a statistical mashup of human-created solutions to similar problems. It might be an elaborate mashup, but it reflects the relationship between inputs and outputs in the training data, not the subtly different relationship between the operator’s needs and the correct output. In the previous post, I described how elaborate theories and experimental apparatuses hide the fundamental flaws in perpetual motion machines. In the very same way, the plausibility of GenAI’s output hides its fundamental inability to differentiate correct from incorrect.

Let’s consider an example. Imagine using GenAI to design a building, a task that usually involves the collaboration of architects, engineers, skilled tradesworkers, inspectors, and bureaucrats. These different roles help ensure the building is safe and reliable for its intended purpose. They are guided by professional experience and a collection of building codes, which can vary from region to region and city to city. GenAI might be able to produce a design that would work well in typical contexts. But what if the location is in a flood plain? That’s statistically unlikely in terms of other buildings that exist, so GenAI is unlikely to produce an output that accounts for it. If the AI is provided with the relevant building codes, and the relevant site information, it might be able to address the issue. But does the operator know which codes apply? Do they know to check whether the building is in a flood plain? Do they know what other factors might matter? The only people who do are the experts who have been doing the job already, without the assistance of GenAI.

To present unverified GenAI output as fact is irresponsible and dangerous.

Just as the impossibility of perpetual motion is immediately obvious to anyone who understands physics, the unreliability of GenAI should be immediately obvious to anyone who understands statistical inference. In fields where mistakes can cause real harm, like science, engineering, law, and medicine, to present unverified GenAI output as fact is irresponsible and dangerous. Researchers and other professionals need to be aware of these risks, before considering using GenAI.

The other posts in this series elaborate on the abilities and limitations of GenAI, and their implications for scientific integrity:

Part 1: Reliable GenAI and Perpetual Motion
Part 2: Statistical Inference and (In)correctness
Part 3: Inference Plus Guardrails
Part 4: Methodology and the Replication Crisis
Part 5: Fragile Systems and Their Consequences
Part 6: Principles for Evaluating GenAI