3D Canvas & Gaussian Splats
Relevant source files
The following files were used as context for generating this wiki page:
The 3D Canvas and Gaussian Splat system provides real-time visualization of WiFi sensing data. It renders a 3D signal field and detected human presence using Three.js and custom shaders. This system transforms raw Channel State Information (CSI) and signal features into a spatial representation, allowing users to observe motion, presence, and signal disruption within a virtual room environment.
The module operates across both web and mobile platforms. On mobile, the system utilizes a WebView to execute the Three.js renderer, while the web version integrates directly into the browser DOM. The visualization consists of a coordinate grid, sensor node markers, a signal field composed of "splats," and a dynamic "body blob" representing detected human subjects.
Sources: ui/components/gaussian-splats.js, ui/mobile/src/assets/webview/gaussian-splats.html, ui/components/SensingTab.js
System Architecture
The visualization architecture relies on a specialized renderer that consumes SensingFrame data. A central manager initializes the Three.js scene, camera, and lighting before creating the specific geometric components of the sensing environment.
Core Components
| Component | Description |
|---|---|
| GaussianSplatRenderer | Manages the Three.js lifecycle, scene setup, and data-driven geometry updates. |
| SensingTab | Integrates the renderer into the UI, handles service connections, and updates the HUD. |
| Signal Field | A 20x20 grid of splats representing localized signal intensity/disruption. |
| Body Blob | A cluster of dynamic splats that appear and pulse based on presence and breathing data. |
| Node Markers | 3D spheres representing the physical locations of ESP32 sensing nodes and routers. |
Sources: ui/components/gaussian-splats.js:52-110, ui/components/SensingTab.js:9-50, ui/mobile/src/assets/webview/gaussian-splats.html:129-195
Data Flow Logic
The following diagram illustrates how sensing data flows from the WebSocket service to the 3D renderer and HUD elements.
Gaussian Splat Rendering
The system uses a custom ShaderMaterial to render "splats" as screen-space discs. Unlike standard particles, the size, color, and opacity of these splats are driven directly by WiFi signal features such as variance and classification confidence.
Splat Attributes and Mapping
The renderer maps scalar values from the sensing data to visual properties using a blue-to-red color gradient.
- Size: Driven by signal variance or disruption magnitude.
- Color: Blue represents a "quiet" environment, green represents presence, and red indicates active motion.
- Opacity: Driven by the classification confidence.
Sources: ui/components/gaussian-splats.js:1-12, ui/components/gaussian-splats.js:41-57
Shader Implementation
The vertex shader calculates point size based on the splatSize attribute and distance from the camera. The fragment shader discards pixels outside a circular radius to create soft-edge discs.
// Fragment Shader Logic
const SPLAT_FRAGMENT = `
varying vec3 vColor;
varying float vOpacity;
void main() {
float dist = length(gl_PointCoord - vec2(0.5));
if (dist > 0.5) discard;
float alpha = smoothstep(0.5, 0.2, dist) * vOpacity;
gl_FragColor = vec4(vColor, alpha);
}
`;
Sources: ui/components/gaussian-splats.js:30-46, ui/mobile/src/assets/webview/gaussian-splats.html:77-87
Visualization Environments
The renderer constructs a virtual room to provide spatial context for the sensing data.
Room Geometry
The system adds a GridHelper at y = 0 to serve as a floor reference. A wireframe box (20x6x20 units) defines the room boundaries.
Node and Router Tracking
- Router: A green sphere placed at the center
(0, 0.5, 0)with a gentle glow pulse. - Sensing Nodes: Cyan spheres representing ESP32 devices. The renderer supports multiple nodes, mapping
node_idto a specific color palette and updating their positions based on the incomingnodesarray in the data frame.
Sources: ui/components/gaussian-splats.js:115-132, ui/components/gaussian-splats.js:160-178, ui/components/gaussian-splats.js:338-360
Body Disruption Blob
When the system detects presence, it activates a "Body Blob" consisting of 64 splats distributed in a sphere. The blob pulses at a frequency of 0.004 based on the breathing_band_power feature. If the motion level is "active," the blob shifts toward red; otherwise, it remains green.
Sources: ui/components/gaussian-splats.js:180-216, ui/components/gaussian-splats.js:295-320
User Interaction and Controls
The 3D view supports interactive navigation to allow inspection of the signal field from different angles.
Interaction Flow
The renderer implements mouse and touch controls for rotation, zooming, and panning.
Visualization Modes
The system supports different operational states that affect the visualization:
- LIVE: Connected to hardware, rendering real-time CSI data.
- SIMULATED: Rendering synthetic data for demonstration or testing.
- RSSI: Rendering coarse presence detection based on signal strength only.
Sources: ui/components/SensingTab.js:157-178, ui/viz.html:238-255
Integration and Lifecycle
The SensingTab class manages the initialization and disposal of the 3D components to prevent memory leaks and redundant processing.
- Initialization:
SensingTab.init()builds the DOM, loads Three.js via CDN if necessary, and instantiates theGaussianSplatRenderer. - Connection: The tab starts the
sensingServiceand subscribes to data updates. - Resize Handling: A
ResizeObservermonitors the viewport dimensions and triggersrenderer.resize()to update the camera aspect ratio and renderer size. - Disposal: The
dispose()method unsubscribes from the data stream, disconnects observers, and calls the renderer's own disposal logic to clean up WebGL resources.
Sources: ui/components/SensingTab.js:17-30, ui/components/SensingTab.js:296-318, ui/components/gaussian-splats.js:377-385
Summary
The 3D Canvas and Gaussian Splat module serves as the primary visual interface for interpreting WiFi-based human sensing. By utilizing custom shaders and a dynamic point-cloud architecture, it provides a low-latency, spatial representation of signal disruptions and human vitals. The system integrates tightly with the sensing service to ensure that visualization accurately reflects the current hardware or simulation state.