Raphael Memmesheimer, a postdoctoral researcher in the Autonomous Intelligent Systems group at the University of Bonn, received the best paper award at the RoboCup 2026 conference with his colleagues for work on image annotation for competition robots. A five-time world champion in RoboCup@Home, he discusses his research, his journey, and the challenges of the competition.
- Teams ont seulement deux jours pour préparer leurs robots à un appartement et des objets inconnus.
- L’équipe de Bonn annote automatiquement environ 60 % des données grâce à la segmentation semi-automatique.
- L’équipe NimbRo termine troisième en 2026 et réussit le mieux la tâche de lessive.
- L’équipe prévoit d’améliorer la manipulation d’objets et l’usage de deux bras.
Two Days to Learn an Apartment
The starting point is the competition schedule: each team only has two days of on-site preparation. The robots must be assembled, get used to a model apartment they have never seen, and learn to recognize the objects they will interact with, whether it’s a spoon or a specific drink. The number of these objects can reach 50, many of which are unknown before the event.
Some of these objects can be anticipated, like drinks, snacks, or cutlery, but their specific features cannot. Organizers often buy local products from a shopping center, and part of the set comes from the public dataset ‘Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation’. For this work, Raphael Memmesheimer described a multi-step approach based on segmentation, i.e., locating areas in an image that correspond to an object. A single model, trained on objects encountered in previous editions, identifies what could be an object, then reference images allow for semi-automatic labeling of the objects present.
According to him, about 60% of the data is annotated automatically this way, a task that would take many hours if done through human annotation. So-called ‘open-vocabulary’ approaches, capable of recognizing objects described by words, however had difficulties distinguishing objects within the same category, like drinks. Query-guided detection methods work, he says, with 5 to 10 classes, but have not yet been scaled to the competition’s 50 classes.
Competition Results
The approach was already used at RoboCup 2025 in Brazil and at the RoboCup German Open. For the 2026 edition in Korea, the team used an evolved version, which also leverages videos to generate more data. The time savings allow, according to Raphael Memmesheimer, focusing on testing and on the tasks themselves rather than on vision.
Team NimbRo finished third in the competition. A new task, ‘Doing Laundry’, was on the program: the robot had to carry a laundry basket from a washing machine, open the door, grab t-shirts, and fold them. The robot cannot see inside the machine and must therefore feel that it is holding a garment before pulling it out. Raphael Memmesheimer indicates that his team achieved the best results on this task, thanks to semi-automatic labeling.
Other tasks include interacting with guests, which involves opening the door, greeting visitors, and introducing them to others; a table-clearing task after breakfast; a general service task based on a natural language order; and a restaurant task in an unknown location, where teams only know the menu. Before all these tasks, the robots must pass an initial safety and interaction capabilities check.
A Rapidly Evolving Competition
Raphael Memmesheimer, who participated in the Eindhoven edition, recalls that the tasks were then split across three leagues, including an open platform league, which his team won. They have now merged: 24 teams from around the world are participating this year in a single league. He describes successive progress, from the early mapping and navigation to human perception and speech recognition, while task planning remains a challenge.
For the next edition, the team is working on object manipulation, so the robot can grasp them faster, and on using two arms, an area they wish to improve.
