Replay Systems and CNS View
Relevant source files
The following files were used as context for generating this wiki page:
Replay systems in Flyhard provide a mechanism to reconstruct and visualize simulated trials by synchronizing recorded data from multiple sources, including CARLA vehicle cameras, MuJoCo body physics, and neural activity traces. The Central Nervous System (CNS) view specifically visualizes neural activity using anatomical geometries derived from the MaleCNS dataset, projecting recorded model states onto neuron skeletons and neuropil meshes.
The system ensures temporal alignment across different simulation layers—vision, body mechanics, and neural dynamics—to produce multi-panel videos that reflect the causal relationship between neural commands and physical actuation.
Sources: scripts/render_carla_cockpit.py:2-4, scripts/record_learned_wheel.py:6-9, scripts/acquire_cns_geometry.py:2-4
Replay Architecture and Data Flow
The replay process consumes artifacts generated during capture runs, primarily compressed NumPy archives (.npz) and JSON manifests. The primary script, render_carla_cockpit.py, acts as an orchestrator that aligns three distinct visual layers into a single 16:9 frame.
Synchronized Data Components
| Component | Source Data | Description |
|---|---|---|
| CARLA View | carla-camera.mp4 | Recorded video from the CARLA simulator cockpit or exterior cameras. |
| CNS View | neural-trace.npz | Neural activity data mapped to geometry.npz anatomical structures. |
| Body View | body-trace.npz | Recorded MuJoCo states (qpos, qvel, ctrl) replayed through the WheelRig. |
Sources: scripts/render_carla_cockpit.py:34-45, scripts/record_learned_wheel.py:46-50
Replay Data Pipeline
The following diagram illustrates how recorded simulation artifacts are processed to generate the final synchronized replay video.
The system uses frames.json to index specific body and neural states to video timestamps.
Sources: scripts/render_carla_cockpit.py:65-75, scripts/body_baseline.py:78-85
CNS Visualization System
The CNS View visualizes a subset of the fruit fly connectome. It uses actual MaleCNS v1.0 neuron skeletons and neuropil meshes rather than simplified somata points.
Geometry Acquisition and Selection
The script acquire_cns_geometry.py fetches anatomical data from the MaleCNS collaboration. It selects a class-stratified subset of 512 neurons based on their maximum absolute states recorded during "calm" simulation runs.
- Skeletons: Derived from precomputed MaleCNS data, downsampled (stride 4) while preserving branch endpoints.
- Neuropil Meshes: Includes 114 measured compartments from the brain and Ventral Nerve Cord (VNC).
- Coordinate System: Converts published nanometers to micrometers.
Sources: scripts/acquire_cns_geometry.py:35-55, apps/website/public/media/attribution.md:14-23
Activity Rendering
The render_cns.py script colors neuron skeletons based on recorded signed model states.
- Color Mapping: Positive states are colored orange; negative states are colored blue. Zero states appear gray.
- Scaling: Each neuron has a fixed color range derived from its sampled absolute states over a specific episode, with a brightness floor of 1e-5.
- Implementation: Utilizes VTK and EGL for offscreen GPU rendering of anatomical structures.
Sources: scripts/render_carla_cockpit.py:125-130, apps/website/public/media/attribution.md:19-25
Body Physics Replay
Physical motion is reconstructed by feeding recorded generalized coordinates (qpos) into a MuJoCo MjData object.
The WheelRig Replay
The WheelRig class in flyhard.cockpit provides the physical environment for the fly body. During replay, the system:
- Loads the
WheelRigmodel. - Sets
replay.qpos,replay.qvel, andreplay.ctrlfrombody-trace.npzfor each frame. - Executes
mj_forwardto update derived kinematic quantities. - Renders the scene using a fixed camera configuration.
Sources: scripts/render_carla_cockpit.py:53-65, scripts/record_learned_wheel.py:78-85
Precision and Validation
The replay system validates accuracy by comparing the current simulation state against the recorded state.
- Clock Error: The script asserts that the difference between the CARLA camera time and MuJoCo body time is less than $10^{-4}$ seconds.
- State Error: The system tracks
replay_max_qpos_errorto ensure the physical reconstruction does not deviate from the original simulation.
Sources: scripts/render_carla_cockpit.py:77-78, scripts/render_carla_cockpit.py:116
Interchange and Asset Preparation
For high-fidelity rendering or integration with CARLA's Unreal-based environment, the system provides asset export utilities.
Interior Asset Export
The export_interior_assets.py script prepares fly meshes and recorded poses for external tools.
- Mesh Tesselation: Uses
trimeshto convert MuJoCo primitives (spheres, capsules) into triangle meshes. - Coordinate Transformation: Maps MuJoCo coordinates to a right-handed "X forward, Y left, Z up" system in meters, anchored for CARLA vehicle interiors.
- Pose Recording: Exports a
poses.jsonfile containing body part transforms for every frame, enabling playback in Unreal Engine without physics simulation.
Sources: scripts/export_interior_assets.py:35-55, scripts/export_interior_assets.py:72-80
Summary of Replay Metrics
Replay runs generate a render-metrics.json file summarizing the session performance and hardware details.
| Field | Description |
|---|---|
max_camera_body_clock_error_s | Maximum temporal drift between vision and physics layers. |
replay_max_qpos_error | Maximum discrepancy in physical joint positions during replay. |
model_rate_color_scale | The method used for mapping neural states to colors. |
gpu_capabilities | Details of the GPU and OpenGL version used for rendering. |
Sources: scripts/render_carla_cockpit.py:121-135, reports/2026-09-09-clean-videos/carla-new-yorker-v1/render-metrics.json
The Flyhard replay system effectively bridges the gap between low-level neural/physical simulation and high-level visual presentation, providing a verifiable record of the experiment's findings.