The first time a virtual hand moved fingers I don’t have, I watched my intention become visible.

That is why I built Phantom Hands.

As computers move beyond keyboards and touchscreens, our bodies are becoming the interface. We point to select. Pinch to open. Reach to grab. But those interactions assume everyone has the same anatomy and range of motion.

I know what it feels like to fall outside that assumption.

Phantom Hands gives software a way to learn the person using it.

A personal model of movement

I engineered a learning system around Google’s MediaPipe that builds an individual model of how someone expresses gestures.

Through guided demonstrations, it learns the relationship between available movement and intended action. A subtle wrist rotation can become a selection. Movement in a remaining finger can become a grip. Signals from one part of the hand can drive virtual fingers elsewhere.

The system reconstructs a responsive virtual hand, allowing someone to express gestures their physical anatomy cannot produce.

The user provides the intention. Phantom Hands supplies the translation.

Built for the complexity of real movement

Human movement changes. Fatigue alters precision. Recovery introduces new capabilities. A progressive condition can make yesterday’s controls difficult today.

Phantom Hands continuously refines its personal movement model, distinguishes deliberate gestures from incidental motion, and asks for clarification when intent is uncertain. Users can teach it new mappings as their needs change.

Learning happens on the device. Each person owns their movement profile and can revise, export, or rebuild it.

One personal breakthrough. A much larger possibility.

What began as an experiment with my own hand became an open-source foundation for accessible gesture interaction.

Phantom Hands connects personalized movement to games, spatial interfaces, creative software, and assistive tools. Developers can support different bodies through a shared translation layer. Researchers can study new approaches. Users can shape controls around their own abilities.

The engineering spans computer vision, personalized machine learning, motion reconstruction, and real-time interaction. Its purpose is something much more immediate.

Picking up an object in a virtual world. Making art independently. Using an interface that finally understands what you meant.

The person defines the gesture. The software adapts.