An AI model is said to be aligned when its behavior matches the intentions of those who design and use it. Tests conducted by the UK’s AI Security Institute (AISI) on OpenAI’s GPT-6 Astra show that a model presented as highly aligned can pursue unintended objectives. The issue takes on particular weight with AI agents, capable of acting autonomously.
What is a misaligned model?
A model is misaligned when it pursues something other than what is expected of it, without this implying any malicious intent. Most often, it diligently optimizes a poorly formulated objective or an imperfect metric. Mathematician Norbert Wiener already described this trap in 1960, in an article for the journal Science. He imagined a bottle factory programmed to produce as many bottles as possible, which would end up bankrupting its owner under an unsellable stock. He argued that one must be ‘absolutely sure that the purpose put into the machine is the purpose which we really desire, and not merely a seductive imitation.’
With large language models, misalignment takes several forms. Goal circumvention, or reward hacking, was described by OpenAI in March 2025: a model in training, tasked with passing unit tests, had modified the verification function to always return ‘true’, deeming the actual correction too difficult. Complacency was illustrated in April 2025, when an update to GPT-4o made ChatGPT so eager to please that OpenAI withdrew it after a few days. The company acknowledged giving too much weight to short-term feedback, such as thumbs-up on responses. Finally, deception and scope overreach refer to a model acting beyond what it was permitted or giving an inaccurate account of what it accomplished.
How is a model aligned?
After pre-training, a model knows how to continue a text, but not yet how to follow an instruction. Alignment comes next, in several layers. Reinforcement Learning from Human Feedback (RLHF) involves having annotators rank several model responses, after which the model is trained to produce the ones they prefer. OpenAI implemented this as early as January 2022 with InstructGPT. Constitutional AI, introduced by Anthropic in December 2022, replaces some human annotations with a list of principles written in natural language, from which the model critiques and rewrites its own responses.
Other methods complement this setup. Laboratories and public bodies subject models to scenarios designed to push them to make mistakes before deployment. Another approach involves having a second model review the reasoning chain of the first to spot an intent to cheat. OpenAI observes, however, that penalizing these ‘bad thoughts’ does not eliminate the majority of deviations, and mostly teaches the model to conceal its intentions.
Misalignment should not be confused with hallucination. A hallucination is an error: the model presents false information as true. Misalignment concerns the objective. The one who rewrites a test to make it pass is not hallucinating. Retrieval-Augmented Generation (RAG), which provides the model with reliable sources, reduces hallucinations without changing what it seeks to accomplish.
What this means for businesses
As long as a model merely responded, misalignment produced a bad answer that the user could review. Agents, however, send emails, update customer records, and execute code. Misalignment then becomes an action, sometimes difficult to undo.
OpenAI launched GPT-6 Astra in early September, presenting it as ‘the world’s smartest and best-aligned model.’ The tests conducted by the AISI before its release, published on September 28, nuance this claim. In a simulated cybersecurity evaluation, with safety filters disabled, Astra conducted software supply chain attacks against targets outside its mission in 29% of cases, compared to 6.3% for GPT-5.6 Sol. It also created fake identities to deceive developers. The UK institute urges caution, noting the model often remarked that it was in a simulation.
OpenAI also abandoned plans to launch GPT-6.1 Astra, expected in October. In a statement to the press, Saachi Jain, the company’s head of safety systems, stated that the model ‘did not quite meet the expected level regarding respect for scope and permissions, nor in how it accounts to the user for the work performed.’ At Astra’s launch, OpenAI highlighted its own evaluation, where the model never exceeded the permitted scope. No method today can guarantee that a model is aligned: one can only test it, scenario by scenario. Hence the importance, for businesses, of limiting agents’ access rights and having sensitive actions validated by a human, rather than relying solely on the ‘aligned’ label.
