CARLA Simulator Integration
Relevant source files
The following files were used as context for generating this wiki page:
CARLA Simulator Integration provides the virtual environment where a connectome-based neural model operates a simulated vehicle. The project uses CARLA version 0.9.16 to simulate vehicle motion, provide visual input, and handle environmental physics. The integration bridges the biological simulation of a fruit fly (run in MuJoCo) with the autonomous driving simulator through a synchronized control loop.
Sources: README.md:10, README.md:58, THIRD_PARTY.md:55
System Architecture
The integration connects visual input to a trained connectome model, which actuates limbs that physically interact with steering and pedal controls. These control positions translate into CARLA vehicle motion commands.
The system maintains a synchronized clock between the neural model, the body simulation, and the CARLA environment.
Sources: README.md:46-51, reports/2026-09-09-interior/interior-replay-v1/action-source.json:17
Simulator Configuration and Synchronization
The integration utilizes specific simulation parameters to ensure physical accuracy and frame-level synchronization.
| Parameter | Value | Description |
|---|---|---|
| CARLA Version | 0.9.16 | The specific simulator release used for research. |
| Time Step (dt) | 0.04s | The fixed time interval for CARLA simulation ticks. |
| Camera FPS | 25 | The frame rate for video capture and visual processing. |
| Physics dt | 5e-05s | The MuJoCo physics time step for high-frequency contact. |
| Map | Town10HD_Opt | The default environment used for steering experiments. |
Sources: reports/2026-09-09-interior/interior-replay-v1/action-source.json:23-28, reports/2026-09-09-interior/interior-replay-v1/action-source.json:44
Coupling Logic
The system uses an interval-based coupling mechanism. At the start of each interval, the system reads the wheel position from the physical simulation. It holds the derived steering command through one CARLA tick and records both simulators at the end of the interval.
Sources: reports/2026-09-09-interior/interior-replay-v1/action-source.json:17
Control and Validation
The project validates the causal link between the fly's physical actions and the vehicle's response.
Steering Control
A learned foreleg controls a passive steering wheel in the MuJoCo rig. The steering wheel's rotation is mapped to CARLA steering commands using a steer_gain (default 0.1). If the foot grip is disconnected, the steering response drops by over 99%.
Sources: README.md:21, scripts/validate_carla_cockpit.py:46, reports/2026-09-09-interior/interior-replay-v1/action-source.json:11
Validation Process
The validation script scripts/validate_carla_cockpit.py reconciles saved physics traces with CARLA readbacks.
- Replay Physics: It drives the body model using only saved neural actions and compares the resulting
qposto the original trace. - Verify Steering: It calculates the expected steering command from the recorded wheel angle and matches it against the applied CARLA steering.
- Clock Check: It ensures the camera time matches the body simulation time within a 1e-4 second tolerance.
Sources: scripts/validate_carla_cockpit.py:24-30, scripts/validate_carla_cockpit.py:41-43
Video Rendering and Presentation
Visualization combines CARLA footage with neural and physical data to provide a comprehensive view of the experiment.
The renderer scripts/render_carla_cockpit.py produces 1080p video by:
- Positioning CARLA camera footage in the main view.
- Adding a neural activity display showing actual MaleCNS skeletons.
- Rendering the MuJoCo physical fly and cockpit rig in a side panel.
- Synchronizing all elements to the same 25 FPS clock.
Sources: scripts/render_carla_cockpit.py:20-25, scripts/render_carla_cockpit.py:84-88
Livery and Asset Integration
The simulated vehicle uses a Mini Cooper S 2021 proxy. Sponsorship artwork is integrated through a custom livery system.
- Asset Origin: Vehicle geometry and textures originate from CARLA's
vehicle.mini.cooper_s_2021blueprint. - Surface Mapping: Sponsors are applied as separate UV-mapped surfaces (decals) rather than a single paint texture.
- Native Integration: For native CARLA rendering, these surfaces must be attached as mesh components to the Unreal vehicle asset or baked into CARLA material UV maps.
Sources: THIRD_PARTY.md:58, apps/mini-livery/README.md:46-51, apps/website/LIVERY-EXPORT.md:46-52
Summary
CARLA Simulator Integration serves as the physical world for the Flyhard experiment. It provides the environmental physics and visual feedback necessary to test whether a connectome-derived model can control a simulated vehicle through a physically realistic cockpit interface. The integration emphasizes rigorous synchronization and validation to ensure that vehicle motion is a direct causal result of the simulated fly's limb movements.