Fine manipulation remains one of the main obstacles preventing robots from carrying out a wide range of everyday tasks. The sense of touch could be the key, but a lack of quality data has held back progress. University labs and startups are now multiplying tactile datasets and techniques to exploit them.
In recent years, vision-language-action (VLA) models have significantly improved robots’ ability to perform complex tasks involving unknown objects and environments. Pre-trained on large amounts of images, videos, and text, then fine-tuned on a smaller number of teleoperated demonstrations-meaning remotely piloted by a human-these models guide robots in tasks like folding laundry, tidying a living room, or using kitchen appliances, based on a simple video feed and natural language instructions.
However, robots struggle with tasks requiring great finesse, such as handling deformable materials or small objects, plugging in a USB cable, or turning a key in a lock. This is partly because VLA models ignore one of the main sources of information humans have in these situations: tactile feedback.
A Touch Difficult to Integrate
“Most dexterous manipulations can be performed by humans with their eyes closed, ” says Trevor Darrell, a computer science professor at the University of California, Berkeley. “Understanding force, slip, and precise grasping is not something you can do properly with traditional vision sensors.”
Yet leveraging tactile sensors remains tricky. Their data has very different characteristics from the images VLA models are usually trained on, and tactile datasets are far behind the internet-scale that many visual and linguistic datasets benefit from. Darrell’s team therefore first pre-trained a model on existing data, before giving it a sense of touch via a specialized sub-model, trained on 100 hours of high-quality tactile data, specifically collected. It includes demonstrations of common actions like wiping, grasping, twisting, or pouring, performed with over 200 different household objects.
The challenge is also one of speed. The goal is for the robot to correct its grip in real time using the tactile signal, but this requires reaction times shorter than those at which most vision-language models operate. The team therefore split the system into sub-models called “experts”: an action expert produces motion plans, while the tactile expert, which runs four times faster, adjusts this plan in real time. After fine-tuning on about 100 teleoperated demonstrations of relatively complex tasks, such as screwing in a light bulb, applying toothpaste to a toothbrush, or transferring an egg from one tray to another, the model achieves an average success rate of 65% on 12 tasks, nearly double that of the best VLA model.
A Question of Diversity
But, Trevor Darrell acknowledges, his data comes from a single robotic hardware. Robotic hands range from fully articulated five-fingered models to simple grippers, and tactile sensors can rely on very different physical principles, from measuring resistance variations to recording images of a soft gel pad deforming. According to Chengbo Yuan, a master’s student at Tsinghua University in Beijing, this makes most tactile AI research dependent on a specific sensor and difficult to share between teams.
Yuan gathered over 3,000 hours of robotic tactile data from public datasets, covering 21 sensor types and various robotic embodiments. He cites the Open X-Embodiment collaboration, which pooled data from many robots, as inspiration. His team designed a hardware-agnostic model: it converts each sensor’s output into a common format and maps it to labeled positions on a human hand model. This model significantly outperformed a baseline model, even on hardware it had never encountered. Yuan attributes this result to acquiring “a kind of common sense of tactile knowledge, ” thanks to training on such varied devices.
Scaling Up
Despite these results, Yuan believes more tactile data is needed. His group is leading a collaboration of 80 institutions aiming to assemble a larger set of teleoperated demonstrations, using a standardized method of collection and processing. Meanwhile, Fudan University in Shanghai and its spin-off NeoteAI have already produced a tactile dataset an order of magnitude larger than previous efforts. Using a proprietary sensor mounted on different robotic arms and a manual gripper operated by humans, they collected over 30,000 hours of demonstrations, with synchronized visual and tactile data.
This data was used to train a model that not only reacts to touch: it also predicts what the robot should feel, in order to guide and evaluate its actions, which significantly improves performance. Shunlin Lu, a postdoctoral researcher at Fudan University and CTO of NeoteAI, sees this as clear evidence that access to large-scale, diverse tactile data leads to significant performance gains.
Another approach is to leverage the vast volumes of robotic visual data already collected. Researchers at the University of Southern California in Los Angeles published a model capable of inferring tactile information from images, trained on over 2,700 demonstrations of everyday manipulation performed with a manual gripper that records both tactile data and images. The model matches images of the contact between the gripper and objects to the pressure felt at that moment, giving even robots without tactile sensors a rudimentary sense of touch.
It remains uncertain how much tactile data will be needed for breakthroughs in the most delicate tasks. So far, observes Chengbo Yuan, the main contribution of touch has mostly been to make robots more efficient at tasks already within reach, like picking and placing objects, and he suspects new algorithms will be needed to tackle problems truly impossible without touch. Long Cheng, from the Chinese Academy of Sciences in Beijing, summarizes the difficulty: “Data is good. But how to use it properly is another question.”
For his part, Shunlin Lu suggests a figure close to 100,000 hours of data, collected in real and varied environments rather than in the lab, to unlock new capabilities. “I think tactile intelligence is actually the next step for physical AI, ” he says.
