MIT engineers have developed an ultrasound bracelet capable of tracking the movements of the wearer’s hand in real time. Paired with an artificial intelligence algorithm, it translates images of the wrist into positions of the five fingers and palm, and can control a robot or a virtual object. The work has been published in the journal Nature Electronics.
The hand remains one of the most complex parts of the body. According to the text, it involves 34 muscles, 27 joints, and more than 100 tendons and ligaments, making it difficult to reproduce with robots or in virtual reality environments.
Existing approaches each have their limitations. Camera systems are impractical to deploy and sensitive to visual obstacles, while sensor-equipped gloves can hinder natural movements. A third approach involves using the electrical signals from forearm muscles, but these signals are easily disrupted by ambient noise and cannot distinguish subtle movements.
The team led by Xuanhe Zhao, a professor of mechanical engineering at MIT, therefore sought to determine if ultrasound imaging could capture finer, continuous movements. The device relies on ultrasound “stickers,” miniaturized versions of the transducers used in medical practice, combined with a hydrogel material that can adhere to the skin. In this study, these sensors were integrated into a bracelet that continuously observes the muscles and tendons of the wrist.
The bracelet is the size of a smartwatch, and its onboard electronics are about the size of a mobile phone. According to the researchers, it produces clear, continuous images of the wrist when the volunteer moves their fingers.
Linking images to gestures
The challenge then was to link these black-and-white images to precise hand positions. The fingers and thumb have 22 degrees of freedom, meaning 22 different ways to extend or tilt them. The researchers identified, within the wrist images, areas corresponding to each of these degrees: one region, for example, provides information on thumb extension, another on index finger movements.
To establish these correspondences, a volunteer wearing the bracelet performed different positions while multiple cameras recorded their gestures. Since real-time translation is impossible to do manually, the team used an artificial intelligence algorithm, trained to recognize patterns in the images and associate them with labels. The images were carefully annotated, then the algorithm was tested on a new set of images, and it successfully predicted the corresponding gestures.
Eight volunteers with different hand and wrist sizes then wore the bracelet. They performed various postures and grasps, including the signs for the 26 letters of the American Sign Language alphabet, and held a tennis ball, a plastic bottle, a pair of scissors, or a pencil. In each case, the bracelet accurately tracked and predicted the hand position.
Demonstration applications
In demonstrations, the team wirelessly connected the bracelet to a simple computer program. Pinching and grasping gestures allowed them to enlarge, shrink, or move an object on screen. The bracelet also served as a wireless remote for a simple commercial robotic hand: the robot reproduced in real time the movements of a volunteer playing on a keyboard, thus playing a simple tune on the piano. The same robot also reproduced finger taps to play a table basketball game.
The researchers have already collected movement data from numerous users, with the aim of building a large dataset. According to the text, this could be used to train humanoid robots for dexterity tasks, such as certain surgical procedures. Zhao plans to further miniaturize the hardware and train the algorithm on a greater number of gestures and morphologies. “We believe this is the most advanced method for tracking fine hand movements, thanks to wrist imaging,” he says.
This work was supported in part by MIT, the National Institutes of Health, the National Science Foundation, the U.S. Department of Defense, and the Singapore National Research Foundation, through the Singapore-MIT Alliance for Research and Technology.
