A simple finger tap may soon replace taps, clicks, and even typing on devices. Researchers from Cornell University and KAIST have developed a new smartwatch-based system that tracks hand movements using built-in sonar and AI.
The system, called WatchHand, turns regular smartwatches into powerful hand-tracking tools. What makes it unique is that it requires no additional hardware. It works using the smartwatch’s built-in speaker and microphone, combined with artificial intelligence.
This approach opens up a new way of interacting with technology, one that feels natural, subtle, and hands-free.
How WatchHand Works
WatchHand uses a method called micro sonar. The smartwatch sends out sound waves that humans cannot hear. These waves bounce off the user’s hand and return to the watch’s microphone. This creates an ‘echo profile.’
READ ALSO: When AI Chooses Who Suffers: MIT’s SEED-SET Flags Ethical Blind Spots Inside AI Decisions
An AI-powered system then reads this echo profile and converts it into a 3D understanding of the hand’s position and movement. This happens in real time. In simple terms, the smartwatch listens to how sound reflects off your fingers and figures out what your hand is doing.
The entire process runs directly on the smartwatch. This means no data needs to be sent to external servers, keeping user information private and secure.
Researchers say this technology can change how people interact with computers, phones, and other devices. Instead of touching screens or using accessories like a mouse, users can rely on small finger movements.
Chi-Jung Lee clearly explained the potential. She said, “With this kind of hand-tracking technology, we might track typing using just a smartwatch. Our hands can act as an input device for computers.”
This means everyday actions like typing, scrolling, or selecting items could be done entirely through gestures.
WATCH ALSO: International Space Station crew wishes New Year 2026
Beyond Convenience: Real-World Applications
WatchHand is not just about convenience. It can also support people who have limited mobility or speech difficulties. Gesture-based control can offer them a new way to communicate and interact with devices.
The system can also be used in augmented reality (AR) and virtual reality (VR). Users could control digital environments using only their hands, without holding controllers.
Jiwan Kim highlighted its importance. He said, “WatchHand lowers the barriers to hand-pose tracking. If a device has a speaker and microphone, our method can work.” This makes the technology highly scalable. Millions of existing devices could gain new features through a simple software update.
To ensure reliability, the researchers tested WatchHand with 40 participants. The testing included four different studies and covered around 36 hours of gesture data.
The system was evaluated on different smartwatch models. It worked on both right and left hands. It also performed well in noisy environments.
The results showed that WatchHand can accurately track finger movements and wrist rotations. However, it is not perfect yet. The system struggles when the user is walking. Movement affects how sound waves bounce back, making it harder for the AI to accurately read hand positions.
Another limitation is compatibility. WatchHand currently works only on Android-based smartwatches and not on Apple’s iOS devices.
READ ALSO: University of Tartu Builds Light-Based Wearable to Track Hidden Plastic in Body
A Shift in Wearable Technology
The development of WatchHand reflects a larger shift in wearable technology. Researchers are moving toward smarter systems that can understand human behavior through everyday devices.
Cheng Zhang, who leads the Smart Computer Interfaces for Future Interactions (SciFi) Lab, described this vision. He said, “WatchHand reflects our goal of turning everyday wearables into intelligent sensing platforms.”
His lab has worked on various sensing technologies in the past. These include tracking movements of the wrist, face, tongue, and even teeth. Their sensors have been placed in rings, glasses, necklaces, headphones, and even clothing.
In recent years, the focus has shifted to acoustic sensing. This method offers high accuracy while consuming less power than other technologies.
Zhang added, “With just a software update, we can unlock new capabilities on millions of devices.”
WATCH ALSO: China’s new humanoid robot can perform combat moves
WatchHand will be presented at the upcoming ACM CHI Conference on Human Factors in Computing Systems, which starts on April 13 in Barcelona. The event is one of the most important global platforms for human-computer interaction research.
The project has received support from the National Science Foundation and South Korea’s Ministry of Science and Information and Communication Technology.
While improvements are still needed, WatchHand signals an important step forward. It shows how simple devices like smartwatches can evolve into powerful tools for interaction. The idea of controlling technology with just a flick of your fingers is no longer just a fantasy. It is slowly becoming part of everyday life, quietly and quite literally, at your fingertips.
