Researchers from MIT CSAIL and the Toyota Research Institute have introduced SceneSmith, a system that uses three AI agents to build 3D indoor scenes for robot training. These virtual environments allow machines to practice and try different ways of accomplishing a task before being deployed. The work was presented as a spotlight at the 2026 International Conference on Machine Learning.
The main obstacle to generalist robotics remains data. Like humans, robots learn primarily through experience, but physically teaching a machine, for a large number of actions and locations, requires a lot of work and time. Russ Tedrake, a professor at MIT and principal investigator at CSAIL, notes that simulation is a natural path. The physics engines of simulators have advanced significantly, but it remains difficult to produce content rich and varied enough to reflect the complexity of the real world.
Three Agents for a Room
AI agents, semi-autonomous programs capable of performing well-defined tasks, can precisely help produce these settings. SceneSmith relies on three agents that assemble the objects, walls, and overall look of a 3D scene. Each is based on a vision-language model (VLM, a system capable of processing both text and images), in this case OpenAI’s GPT-5.2, described in the source as one of the most advanced models in the field.
The operation follows a three-step chain. A ‘designer’ agent generates the scene elements, a ‘critic’ agent judges whether the result looks realistic, and an ‘orchestrator’ agent manages the exchanges between the two and decides when the design is finished. Once this collaboration is complete, the scene can be loaded directly into physics simulation software. The agents proceed step by step: furniture, objects placed on walls then on the ceiling, and finally objects the robot can manipulate, such as cabinets it can open and close.
Nicholas Pfaff, a PhD student in electrical engineering and computer science at MIT and lead author of the paper with Russ Tedrake, describes a result that surprises him: ‘We found that the system can build 3D scenes in the manner of a human designer.’ According to him, the team generated over 1,300 scenes, with ‘insanely creative’ and varied layouts, which the system had not been explicitly instructed to produce.
According to the researchers, these scenes contain up to six times more objects per room than previous methods, making them useful for teaching a robot to place a cup in the sink, arrange fruit on plates, or move a can from a shelf to a table.
Realism Put to the Test
To verify the realism of these virtual worlds, the researchers tested a pre-trained policy, meaning an AI controller trained largely on real-world data, which had never seen a SceneSmith scene. When asked to ‘take the apple from the bowl and place it on the cutting board,’ the simulated robot accomplished the task. The teams also remotely piloted robots in these spaces to open cabinets, put away bottles, and move from one room to another.
In another test, 100 unique spaces were generated and a VLM agent evaluated the robot’s action plans, which were judged faulty in many cases. Humans agreed with these verdicts in over 99% of cases, according to the researchers, which could allow faulty approaches to be filtered out in simulation before deployment. Furthermore, over 200 users judged the visuals more realistic than those of competing systems in over 90% of cases.
However, the system has a time cost. It can take several hours to produce a single scene, as the agents create and examine each object closely. Increased computing power could improve efficiency, and the team is considering extending the system to deformable objects, such as sponges, if large 3D libraries become available.
