Trust in AI is built by bounding it, not by predicting it

Trust in AI is built by bounding it, not by predicting it
3 min read

In a text published by MIT Technology Review, the authors argue that trust in artificial intelligence cannot be based on predicting its behavior. According to them, it instead comes through strict boundaries, applied outside the system’s scope. A guardrail that the AI itself can move is not a guardrail.

The text takes the example of OpenAI: the guardrails were inside a system that the agents could modify. Traditional software executes instructions written by humans. AI, on the other hand, exercises what the authors call a “synthetic judgment“: it interprets situations that no one programmed and produces responses that no one prescribed. The more freedom it is given, the harder it becomes to guarantee the predictability on which trust has historically been built.

The limits of classic definitions of trust

The text recalls two recent approaches covered by the publication. The first deems it insufficient to put a human in the loop, as humans are fundamentally incapable of understanding the intentions of AI systems, and it argues for interdisciplinary research combining academia, public authorities, and the tech industry. The second describes using incentives to prevent models from deviating from rules, even when no one knows how they work internally. Both recognize that it is impossible to understand or monitor every AI decision.

Yet our trust tools, whether requirements, tests, certification, safety margins, locks, or success and failure criteria, all rely on the ability to bound the behavior of a system. The authors speak here of a “determinism paradox.” With generative AI and agents, this behavior depends on training data, architecture, instructions, context, sources consulted, and environment. Inputs change, models are updated, tools evolve, and agents interact with each other. Small variations propagate into outcomes that no one designed.

The problem is not randomness, the text asserts, but the space of possible behaviors, which is too vast to be enumerated in advance. Trust must therefore be built around boundaries rather than predicted outputs. The goal is not a deterministic AI, but a “bounded, observable, and controllable” AI. It also cannot be a one-time test: AI continues to evolve after deployment, and a system that passed all tests can stray from the conditions that gave those tests meaning. According to the text, trust must be “continually earned.”

Safety engineering has accounted for some of this variability for decades. The Nuclear Regulatory Commission relies on probabilistic risk assessment to estimate the likelihood of failures in complex systems. The STAMP model has shown that accidents can arise from interactions between components, people, and organizations that are each functioning correctly. AI changes the game because a single component can now interpret, infer, and act. The stakes grow as AI moves from browsers to vehicles, robots, medical systems, industrial controls, and data centers, where a bad response can become a life-threatening event.

Building trust that scales

A person validating thousands of AI actions per hour does not constitute an acceptable or scalable guardrail, according to the text. The solution, however, is not to remove the human from the loop, but to place their judgment where it matters: setting boundaries in advance, then approving the rare actions with serious consequences, while strict constraints manage the rest.

Trust must therefore extend all the way to the system’s architecture. The authors pose three questions: what authority to grant the AI, under what conditions, and supported by what evidence? They add a fourth question: what happens outside of those conditions. This approach, they conclude in essence, is very different from asking whether a model scored 94% on a benchmark, i.e., a reference test.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x