ServiceNow CoreAI has introduced AutoSynthData, a tool that creates synthetic training data to improve AI agents in enterprise environments. According to the publisher, the tool starts from the failures of a target model and the successes of a more powerful ‘teacher’ model to decide what the model should learn next. Tests conducted on the EnterpriseOps Gym benchmark (Malay et al., 2026) show results in two domains.
An AI agent operates according to the tools, rules, and data of the organization using it. A model can be broadly capable yet still struggle in a given environment, for example, on a poorly managed sequence of steps, a misused combination of tools, or an unmet constraint. The challenge is to turn these weaknesses into training data, as a single failure is not enough: many new tasks that challenge the same capability in different situations are needed.
A task must be feasible, realistic, and difficult
In the method described by ServiceNow, a task consists of a system specification, a user instruction, and a verifier, responsible for saying whether the agent succeeded. The task must first be feasible: there must be at least one sequence of actions that meets the demand in the environment without breaking the rules. It must then be realistic, meaning it should resemble a request a user might make. Finally, it must be difficult: a task already reliably solved provides little information for training.
The verifier must also meet conditions. It must be consistent with the instruction and the state of the environment, reject paths that fail, and accept valid solutions rather than just the single reference trajectory. A verifier that is too lenient can reward bad behaviors, and a verifier that is too strict can penalize good responses.
Before being retained, each task goes through a series of checks. A positive test verifies that the reference solution indeed succeeds in the environment. A negative test modifies part of the expected result to ensure incorrect states fail. Rejected candidates are examined by a critique module, then repaired within a fixed number of attempts. At the batch level, a meta-review identifies overrepresented task families and missing capabilities.
Results in two domains
The tests focus on the Hybrid and ITSM domains of EnterpriseOps Gym. In the Hybrid domain, the target model is Gemma-4-26B-A4B-it and the teacher is Qwen3.8-27B. AutoSynthData produced 2,000 training samples in about 18 hours, then the model was fine-tuned on this data, a process called ‘fine-tuning’. The best checkpoint is from epoch 5. According to ServiceNow, the average Pass@1 score improves by 7.2 percentage points, a relative improvement of 35%, and the verifier success rate rises from 63.01% to 68.55%. The company indicates the model thus closes 59% of the initial gap with the reference model.
In the ITSM domain, the target model is still Gemma-4-26B-A4B-it, and the teacher is DeepSeek-V4.1-Flash. The generation produced 1,994 samples in 66 hours. According to the company, this longer time is mainly due to the use of a larger teacher model and an earlier version of the pipeline, which has since been optimized. The Pass@1 score then rises from 18.77% to 27.18%.
ServiceNow states that the training tasks were generated from capability specifications, without access to the original evaluation tasks. The company also plans to test this calibrated difficulty logic in reinforcement learning, beyond just supervised fine-tuning.
