Connectome-Based Driving Simulation
Relevant source files
The following files were used as context for generating this wiki page:
The Connectome-Based Driving Simulation is a research experiment that uses a neural model derived from a fruit fly's measured connectome to operate a simulated vehicle. The system integrates biological connectivity data with physical limb actuation to control a steering wheel and pedals within the CARLA simulator.
Sources: README.md:3-8, README.md:46-50
The simulation requires the neural model to causally operate physical controls. A simulated fly body receives camera input, processes it through the connectome-based model, and activates limbs that make physical contact with the steering wheel. This contact generates measured control positions that dictate vehicle motion in CARLA.
Sources: README.md:46-53
Core Architecture and Data Flow
The simulation architecture establishes a causal loop between neural activity, physical simulation, and the CARLA environment.
The diagram shows the transformation of sensory data into vehicle motion through a connectome-constrained neural controller.
Sources: README.md:46-50, scripts/render_carla_cockpit.py:90-105
Component Synchronization
All simulation components, including the neural model, MuJoCo body physics, and CARLA environment, share a synchronized clock. In recorded experiments, 600 video frames share a unified timeline across all subsystems.
Sources: README.md:19-21, scripts/render_carla_cockpit.py:65-75
Connectome Model Implementation
The model uses the MaleCNS v1.0 dataset, specifically neurons with a "Traced" status. This produces a graph containing 165,122 neurons and 25,563,197 measured connections.
Sources: README.md:12-14, scripts/prepare_graph.py:53-60
Graph Processing
The system processes the connectome into a format suitable for PyTorch-based inference.
| Feature | Implementation |
|---|---|
| Matrix Format | Compressed Sparse Row (CSR) |
| Edge Weights | Derived from measured synapse counts |
| Selection Criteria | status == 'Traced' |
| Dynamics | Integrated rate dynamics and trained gains |
| Sources: scripts/prepare_graph.py:90-100, src/flyhard/connectome.py:10-25 |
Neural Dynamics
The Connectome class implements the neuronal dynamics. It uses a sparse weight matrix to compute interactions between presynaptic and postsynaptic states.
Sources: src/flyhard/connectome.py:30-45
# src/flyhard/connectome.py
def forward(self, state, external_input=None):
# Incoming matrix multiplied by presynaptic state
synaptic_input = torch.sparse.mm(self.weight_matrix, state)
if external_input is not None:
synaptic_input += external_input
# Apply rate dynamics
new_state = state + (self.dt / self.tau) * (-state + synaptic_input)
return new_state
Sources: src/flyhard/connectome.py:50-65, scripts/prepare_graph.py:105-110
Motor Policy and Control
The WheelPolicy class maps observations to joint commands for the simulated fly. Observations include requested target angles, current physical wheel angles, and the limb's joint positions (qpos).
Sources: src/flyhard/motor_policy.py:15-30, scripts/record_learned_wheel.py:35-40
Control Loop
- Observation: The system samples the environment at specific intervals (e.g., 20Hz for neural decisions).
- Inference: The
WheelPolicyprocesses observations through the connectome graph. - Actuation: The model outputs joint commands for fly leg servos.
- Physical Interaction: A disclosed forefoot grip transmits force to the passive steering wheel in MuJoCo.
- Measurement: The resulting wheel angle is measured and normalized for CARLA's steering input.
Sources: scripts/record_learned_wheel.py:40-55, scripts/render_carla_cockpit.py:95-105
Simulation Visualization
Visualization integrates CARLA camera footage with anatomical neural activity and 3D body simulation.
Anatomical Mapping
Neural activity is projected onto actual MaleCNS neuron skeletons and neuropil surfaces. The display typically shows a class-stratified subset (e.g., 512 neurons) within measured neuropil compartments.
Sources: scripts/render_carla_cockpit.py:115-120, scripts/record_learned_wheel.py:100-110
Color Representation
- Positive states: Represented in orange.
- Negative states: Represented in blue.
- Neutral states: Represented in gray.
Brightness is normalized per neuron based on absolute states sampled over an episode.
Sources: scripts/record_learned_wheel.py:115-125, scripts/render_carla_cockpit.py:125-130
Summary of Pilot Results
The pilot experiment (2026-09-09) validated the steering skill using this architecture.
- Accuracy: 100/100 held-out targets passed after 186 seconds of training.
- Error Rate: Maximum final hold error of 4.85 degrees.
- Causality: Disconnecting the foot grip removes over 99% of the steering response, confirming the physical operation of controls.
Sources: README.md:15-23