Phantom Hands
Gesture controls trained on the movements you can make.
I was born with a hand difference, and I wanted to try gesture controls that could learn how I move. That became Phantom Hands.
The project maps a person’s available movements to digital actions. It can also use those movements to control a virtual hand, including fingers or gestures the person can’t produce physically.
How it works
Phantom Hands uses Google’s MediaPipe for hand tracking. Through guided demonstrations, users teach it how they express gestures and what those gestures should do.
A wrist rotation might trigger a selection. A finger movement might control a grip. Movement in one part of the hand can also control virtual fingers elsewhere.
The model is personalized, so the same action can be mapped to different movements for different people—or for someone’s left and right hands.
What changed during development
My first approach recognized the hand as a single overall gesture. I thought tracking individual fingers would add little when the system only needed to recognize one intended action at a time.
That approach had trouble when different gestures looked similar. A fist and a pinch could be difficult to distinguish with some hand shapes or ranges of motion. The same gesture could also look different between someone’s left and right hands.
I changed the system to evaluate points associated with individual fingers. Where fingers are absent, representative points are anchored to the person’s available anatomy. Some of those points cannot move independently; the model evaluates their combined pattern for each hand.
In my own use and small-community testing, this helped distinguish gestures such as fist and pinch and supported separate mappings for each hand. It also helped with subtler gestures and changes in available movement.
Updating the controls
Users can teach the system new mappings or revise existing ones as their movement or preferences change.
The system refines the personal movement model over time, filters incidental motion, and asks for clarification when a gesture is uncertain.
Learning happens on the device. Users can revise, export, or rebuild their movement profiles.
Using the project
Phantom Hands is open source. Its movement mappings can be used to explore controls for games, spatial interfaces, creative software, and assistive tools.
I’m sharing it so other people can try it with their own movements and developers can build on the work. I’d be happy to hear from anyone interested in testing it or working on it together.