Customizing Barehands UI
Relevant source files
The following files were used as context for generating this wiki page:
The Barehands UI serves as a highly customizable "stage" for the J.A.R.V.I.S. assistant, utilizing hand-tracking technology to manipulate virtual objects on a transparent digital workspace. The system is designed to be minimal and build-less, consisting of a single HTML page, a small Python server, and a JSON configuration file. It functions as a "body" waiting for an AI "brain" to drive its actions and state.
Customization revolves around three primary vectors: modifying the core configuration to define available media and notes, styling the glass-inspired interface through CSS and specialized shaders, and wiring external AI assistants to manipulate the "board" via a command channel.
Sources: backend/src/barehands/sample-notes/Field Guide/Make It Yours.md, backend/src/barehands/sample-notes/Getting Started/Wire In Your AI.md
Core Configuration (barehands.json)
The primary method for customizing the Barehands interface is through the barehands.json file. This configuration defines the identity of the assistant and the structure of the digital workspace (the "glass").
Configuration Parameters
| Parameter | Description |
|---|---|
name | The name of the assistant displayed within the central ring. |
port | The network port the Python server listens on (default: 8794). |
orbs | A list of folder entries that manifest as interactable "orbs" on the glass. |
Orb Definitions
Orbs are the primary navigation elements. When the central ring is tapped, these orbs "bloom" around it as circular tap targets.
- Notes Orbs: Defined with
"kind": "notes". These point to directories of Markdown files (e.g., an Obsidian vault). They bloom into navigable markdown trees. - Media Orbs: Defined with
"kind": "kind": "media". These serve as "props airlocks" for images and 3D models.
Sources: backend/src/barehands/sample-notes/Field Guide/Make It Yours.md, backend/src/barehands/stage.html:768-790
Visual Customization and Shaders
The Barehands UI adheres to a "Glass Era" design aesthetic characterized by translucent fills, luminous edges, and high readability provided by text-shadow halos rather than solid backgrounds.
Glass Styling Principles
- Flatness: No card-in-card or divider borders inside panels.
- Elevation: Panels use
shadow-nousand hairline borders rather than thick frames. - Transparency: High-alpha "smoked" fills ensure the background (often live video) remains visible.
Shader Customization
3D models utilize specialized shaders that can be customized to change the "material" look:
- Ghost Glass (Hologram Mode): A Fresnel rim shader that provides a translucent teal volume with X-ray interiors. It includes a "scan beam" effect that cyclically wraps around the model height.
- Solid Mode: Uses PBR (Physically Based Rendering) textures with environment lighting and tone mapping to ensure realistic appearances.
Adaptive Resolution
The interface dynamically adjusts the resolution of canvases (the Ring and 3D Models) based on the current scale. This re-resolution occurs when the object "settles" to avoid performance hits during active hand manipulation.
Sources: instructions/DESIGN.md:20-50, backend/src/barehands/stage.html:105-135, backend/src/barehands/stage.html:314-340, backend/src/barehands/stage.html:455-500
The Command Channel (/cmd)
External assistants or scripts customize the "board" state in real-time by POSTing JSON payloads to the /cmd endpoint. This allows an AI to stage objects, remove them, or trigger animations.
Common Board Commands
| Command | Action | Key Parameters |
|---|---|---|
add_card | Materializes a glass card with text. | title, body, file |
add_img | Displays an image from the media airlock. | src |
hand | Delivers a 3D model or image to the user's reach. | src |
give | Delivers an item directly into the user's active hand. | src |
yank | Forcefully removes an item from the board. | title |
explode | Parts a 3D model into its constituent components. | title |
assemble | Rebuilds an exploded 3D model. | title |
Command Architecture
The following diagram illustrates how an AI assistant interacts with the Barehands UI components:
The AI assistant acts as the "brain," sending high-level instructions to the command channel, which are then translated into specific object lifecycle events and animations on the stage.
Sources: backend/src/barehands/sample-notes/Getting Started/Wire In Your AI.md, backend/src/barehands/stage.html:648-735
State Synchronization
The system maintains synchronization between the Tracker (owning camera and physics) and the Render (truly transparent OBS browser source) roles.
State Files
The central "Ring" widget reflects the system's internal state through three polled files:
state/state: Single word indicatingidle,listening,thinking, orspeaking.state/mood.json: A JSON object (e.g.,{"mood": "green"}) that tints the UI colors.state/wave.json: Contains live voice waveform samples used to drive the ring's glow and thickness during thespeakingstate.
Cross-Role Synchronization
State is pushed from the Tracker to the server at approximately 45Hz. Coordinates are transmitted as window fractions (0.0 to 1.0) to ensure items scale correctly regardless of differing window or canvas sizes between the tracker and the render source.
This flow ensures that while the Tracker handles heavy processing like computer vision, the Render role remains lightweight and perfectly synchronized for broadcast.
Sources: backend/src/barehands/sample-notes/Getting Started/Wire In Your AI.md, backend/src/barehands/stage.html:1085-1150, backend/src/barehands/stage.html:1175-1200
Gesture Customization
While gesture logic is primarily handled in stage.html, developers can tune thresholds to adapt to specific hand shapes or camera environments.
Gesture Tuning Protocol
The project utilizes a "Tuning Clinic" protocol:
- Sampling: Use the "P" key to record the "Correct" vs. "Impostor" pose.
- Metric Identification: Identify the separating metric (e.g., finger arch ratio vs. palm-flat distance).
- Thresholding: Set thresholds "mid-canyon"—halfway between the two sampled clusters.
Primary Gesture Logic
- Pinch: Used for grabbing, moving, and flicking items.
- Hold-Still (~1s): Promotes a simple move into a 3D rotation/spin mode.
- Two Hands: Specifically reserved for scaling items.
- Clap (Prayer vertical): Triggers a board reset, bringing the Ring back to center stage.
- Claw (The Force Pull): A complex sequence involving an open-hand flash, claw pose aiming, 2-second "strain" hold, and a snap release to pull items.
Sources: backend/src/barehands/stage.html:2-20, backend/src/barehands/stage.html:860-960, backend/src/barehands/stage.html:1000-1065
Conclusion
Customizing Barehands UI is a multi-layered process ranging from simple JSON configuration changes to deep shader and gesture tuning. By separating the "body" (UI/Physics) from the "brain" (AI Assistant), the system allows developers to build immersive, sci-fi interfaces that are both highly reactive and easily portable across different AI backends.