Artificial intelligence agents pose problems that generative AI alone does not, making their observation while they work both more difficult and more important. Researchers from academia, civil society, and industry have examined the signals emitted by the four most widely used agent frameworks and identified significant shortcomings.
Unlike a generative AI system, which produces a single output for a human to then use, an agent performs longer chains of tasks: it plans, selects tools, creates sub-agents, and executes actions. Moreover, an agent relies on large language models (LLMs), inheriting their unpredictability. Its execution can therefore take many varied forms, and no pre-launch testing can fully anticipate the risks.
According to the authors, these frameworks do not easily allow for the detection of rule violations, unauthorized actions, or failures of human oversight. Several elements essential for observation are missing, such as agent identifiers that persist over time, or records indicating which humans approved which tasks. Under these conditions, it becomes difficult to know which agent did what and to verify if it acted without authorization.
Two Types of Traces
The researchers believe these shortcomings can be addressed, provided that recording two types of traces is prioritized. The first describes what happened during execution: they identify the agent while preserving privacy, log human interventions, and indicate if the agent’s permissions or its ‘memory’ changed. The second concerns decisions: they trace the agent’s reasoning and its confidence level in the actions it took.
The authors give the example of agents tasked with processing a bank’s mortgage applications. The system reviews a file and recommends rejection, then a human advisor validates the decision. During a subsequent audit, or even in real time, the bank or a regulator must be able to reconstruct which agent made the decision, which advisor approved it, and how long it took. The bank must also analyze how the agent arrived at its rejection, to ensure it did not use prohibited information, such as data on the applicant’s neighborhood of origin.
According to the article, recording ‘what happened’ and ‘how the agent decided’ requires different approaches, but both are feasible. The current absence of traces of the first type is largely a matter of software plumbing. It can be addressed using OpenTelemetry, an open observability standard maintained by engineers across the industry and gradually being adopted by agent frameworks.
A Standard Still Under Construction
OpenTelemetry has launched a working group tasked with drafting common formats for agent tracing, but these formats are not yet finalized. According to the authors, this group should urgently incorporate standards for persistent agent identity, as well as for logging human interventions and changes to permissions or memory. This work should be completed before these systems are deployed at scale. It must also consider the privacy implications of an agent identifier linked to real people.
