Barehands Gestures & Control
Relevant source files
The following files were used as context for generating this wiki page:
Barehands is a gesture-controlled interface module within the Jarvis desktop environment that enables users to interact with "floating panes of glass" using physical hand movements. It utilizes a camera-based hand-tracking system (MediaPipe) to translate real-world gestures into digital actions such as moving, scaling, and interacting with markdown notes, 3D models, and media files.
The system acts as a "body" for the AI assistant, allowing the AI to stage information on a virtual board that the user can then manipulate. Integration is handled through a local server that manages state, command channels, and orbital bloom effects for folder navigation.
Sources: backend/src/barehands/sample-notes/Getting Started/Welcome.md, backend/src/barehands/sample-notes/Getting Started/Wire In Your AI.md
System Architecture
The Barehands system operates via a split-role architecture consisting of a Tracker and a Render stage. The Tracker owns the camera input and physics engine, streaming scene state to a local server, while the Render stage (typically an OBS browser source) mirrors this state with true transparency for broadcasting.
The interaction flow relies on MediaPipe's HandLandmarker to detect 21 hand points, which are processed into specific gestures based on distance ratios and segment alignment.
Sources: backend/src/barehands/stage.html:1134-1188, backend/src/barehands/server.ts:161-224
Core Interaction Vocabulary
Barehands uses a specific set of gestures designed for high reliability in varying camera conditions. These gestures are processed in the "Palm Engine" and "Force Pull" logic.
Primary Gestures
| Gesture | Physical Action | Resulting Action |
|---|---|---|
| Pinch | Index and thumb meet | Grab/Move/Scroll item |
| Pinch + Hold | Pinch still for ~1s | Latch into 3D Rotate mode |
| Two Hands | Two hands pinch one item | Scale / Stretch item |
| Tap | Quick pinch and release | Open note / Close panel / Toggle Ring |
| Clap | Palms together, fingers up | Reset Board / Center Assistant Ring |
| Claw | Fingers hooked, thumb out | Aim at distant items (Force Pull arming) |
| Snap | Close Claw into a fist | Pull aimed item to hand (Force Pull) |
Sources: backend/src/barehands/stage.html:125-132, backend/src/barehands/stage.html:2254-2450
Gesture Recognition Logic
The system implements a "Contrast Law" for pinch detection, comparing the curl of the index finger against the arch of the middle, ring, and pinky fingers.
The Assistant Ring & Orbital Bloom
The Assistant Ring is the central hub of the Barehands interface. It reflects the live state of the Jarvis agent and serves as the entry point for file navigation.
- State Visualization: The ring changes appearance based on the agent's current task (
idle,listening,thinking,speaking). - Mood & Wave: The ring supports mood tints (green, amber, red) and live voice waveforms that drive ring thickness and glow.
- Orbital Bloom: Tapping the ring causes configured folder "Orbs" to bloom around it. Tapping an Orb expands it into a browser pane for selecting notes or media.
Sources: backend/src/barehands/stage.html:150-250, backend/src/barehands/server.ts:16-35
AI Integration & Command Channel
The AI assistant controls the board by POSTing JSON commands to the /cmd endpoint. This allows the AI to "stage" information for the user to manipulate manually.
Common Commands
{ "a": "add_card", "title": "IDEA", "body": "Markdown text..." }
{ "a": "add_img", "src": "misc/chart.png" }
{ "a": "hand", "src": "models/engine.glb" }
{ "a": "yank", "title": "IDEA" }
{ "a": "explode" }
Sources: backend/src/barehands/sample-notes/Getting Started/Wire In Your AI.md, backend/src/barehands/server.ts:69-80
3D Holograms & Media Airlock
Barehands supports 3D model rendering (GLB/GLTF) using Three.js. Models can be rendered in two modes:
- Holo: A blue wireframe fresnel rim shader used for sketches.
- Solid: Real PBR textures with environment lighting.
All media must reside in the media/ folder, which acts as a "Media Airlock." The server enforces a "jail check" to ensure files are only served from allowed paths.
Sources: backend/src/barehands/stage.html:430-580, backend/src/barehands/server.ts:114-142
Configuration (barehands.json)
The system's behavior and content are driven by a single configuration file.
| Option | Type | Description |
|---|---|---|
name | string | The name displayed inside the Assistant Ring. |
port | number | The server listening port (default: 8794). |
orbs | array | List of folder mappings (kind: "notes" or "media"). |
Sources: backend/src/barehands/server.ts:4-14, backend/src/barehands/sample-notes/Field Guide/Make It Yours.md
Summary
Barehands provides a futuristic, spatial computing layer for the Jarvis desktop. By offloading heavy processing like hand tracking to the client browser (via MediaPipe) and utilizing a simple JSON-over-HTTP command channel, it allows an AI assistant to interact with the physical world through gestures. This system transforms static markdown notes and 3D assets into a dynamic, "Iron Man"-style interactive workspace.