Simulation Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Simulation Pipeline manages the closed-loop execution environment where a neural connectome model controls a simulated biological body to operate a vehicle. It integrates high-fidelity physical simulations of a fruit fly (NeuroMechFly) within a cockpit environment, capturing sensory-motor data and translating limb movements into vehicle control inputs for the CARLA simulator.
The pipeline spans multiple stages: physics-based body movement in MuJoCo, coordinate transformation for visual assets, and integration with the CARLA vehicle physics engine. This enables the experiment's core loop where visual input drives the connectome, which actuates limbs to physically contact the steering wheel and pedals, resulting in vehicle motion.
Sources: README.md:12-25, scripts/body_baseline.py:11-13
Core Simulation Architecture
The pipeline synchronizes three distinct simulation layers: the neural policy, the MuJoCo-based physical body/cockpit, and the CARLA vehicle environment. The synchronization ensures that all captured frames share a consistent neural, body, and CARLA clock.
The experiment enforces causal operation: the body must physically move the controls to drive the car, rather than using direct programmatic vehicle commands.
Sources: README.md:16-25, scripts/body_baseline.py:35-50
Physical Body and Cockpit Simulation
The NeuroMechFly body simulation uses MuJoCo to handle joint physics and contact dynamics. The pipeline constructs a specialized WheelRig to facilitate the interaction between the fly and the vehicle controls.
WheelRig Configuration
The WheelRig defines the physical environment surrounding the fly. It includes:
- Seat: Provides physical support for the fly body.
- Steering Wheel: A passive, rotatable mesh that the fly's forelegs grip.
- Pedals: Spring-loaded components with engineered contact surfaces for foot interaction.
- Grip Constraints: Disclosed forefoot grip constraints that link the fly's legs to the controls.
Sources: src/flyhard/cockpit.py:10-50, scripts/export_interior_assets.py:46-48, scripts/body_baseline.py:23-32
Body Data Capture
During simulation, the pipeline records a "trace" containing the full physical state. This includes:
- qpos: Joint positions and world coordinates.
- qvel: Joint and object velocities.
- ctrl: Actuator control signals.
- contacts: Number and location of physical contact points.
Sources: scripts/body_baseline.py:61-70, scripts/export_interior_assets.py:39-42
Asset Export and Interchange
To render the simulation in external engines like CARLA or Unreal Engine, the pipeline converts MuJoCo-native geometry and recorded poses into interchange formats (JSON and FBX).
Geometry Transformation
The pipeline scales and mirrors coordinates to match the target simulator's coordinate system. For CARLA/Unreal, this involves transforming from MuJoCo's coordinate system to a right-handed system (X forward, Y left, Z up) measured in meters.
| Parameter | Description | File Context |
|---|---|---|
scale | Scaling factor (default 0.16) from rig units to meters | export_interior_assets.py |
anchor | Offset coordinates [0.27, -0.41, 1.03] for car placement | export_interior_assets.py |
mirror | Diagonal matrix [1, -1, 1] for axis flipping | export_interior_assets.py |
Sources: scripts/export_interior_assets.py:32-37, scripts/export_interior_assets.py:73-85
FBX Export Pipeline
For visual fidelity in high-end renderers, rigid meshes are exported per MuJoCo body part.
Sources: scripts/export_interior_fbx.py:18-65
CARLA Integration
The pipeline uses the CARLA simulator to provide the driving environment and visual feedback. A "smoke test" verifies the connection between the Python client and the CARLA server.
- Vehicle Proxy: Current simulations use a stock Mini Cooper (blueprint
vehicle.mini.cooper_s_2021). - Camera Integration: CARLA 0.9.16 offscreen cameras capture visual input for the neural model.
- Physics Interaction: Steering commands derived from the
WheelRigrotation are applied to the CARLA vehicle actors.
Sources: scripts/carla_smoke.py:10-25, README.md:12-19, scripts/export_interior_assets.py:90-95
Summary of Data Flow
The Simulation Pipeline operates as a highly structured data conversion and synchronization engine. It moves from high-frequency physics steps (typically 1e-4 seconds) in MuJoCo to recorded frames used for visual rendering and model inference. By strictly citing source hashes and maintaining source-manifests, the pipeline ensures that all neural, body, and vehicle recordings remain reproducible and synchronized across the research lifecycle.
Sources: scripts/body_baseline.py:70-85, scripts/export_interior_assets.py:100-115