Gesture Synth is a web-based gesture synthesizer that turns an ordinary webcam into a personal instrument. MediaPipe tracks your hands in real time while a Web Audio engine produces the sound — everything runs locally in the browser, with no uploads and no software to install.
The playing model pairs two complementary mappings. In Gesture mode, raised fingers on the left hand select the chord degree across the full I–vii scale (including VI via index + pinky and VII with the thumb added), while wrist tilt shifts between major and minor. The right hand handles expressiveness: vertical movement controls volume, wrist tilt sweeps the filter for tonal color, raised fingers change chord quality (triads, inversions, sevenths), and a lowered thumb drops the octave. Theremin mode strips it back to pure intuition: the right hand's vertical position continuously controls pitch (~65 Hz to 1200 Hz) and the left hand's height controls volume, like playing a real theremin.
Beyond the core instrument, the project completes the loop from "easy to learn" to "easy to share": a real-time scale/chord visual guide across 12 keys, a step-by-step tutorial, persistent gesture preferences, and a 3-second-countdown recorder that merges the camera feed with the synthesized audio into a downloadable 1080p 16:9 video. On the product side, it ships full localization in 9 languages, multilingual SEO (hreflang, JSON-LD, sitemap), and GA4 analytics — all built on Next.js 16 App Router + TypeScript and deployed on Vercel, free and ready to play.
