Experiment Plan and Roadmap
Relevant source files
The following files were used as context for generating this wiki page:
The Flyhard project executes a research experiment to train a neural model based on a fruit fly's connectome to drive a simulated vehicle. This roadmap outlines the progression from baseline physical simulations to integrated closed-loop driving within the CARLA environment. The experiment uses a measured connectome of 165,122 neurons and over 25 million connections to actuate a simulated fly body that physically operates a steering wheel and pedals.
Sources: README.md:3-5, README.md:43-46, THIRD_PARTY.md:10-15
Core Architecture and Data Flow
The experiment follows a causal chain where neural activity results in physical vehicle control. The system avoids direct scripting of vehicle movement in favor of physical limb actuation.
The diagram shows the closed-loop system where camera input feeds the connectome model to produce physical movement.
System Components
| Component | Description |
|---|---|
| Camera Input | Visual data from CARLA 0.9.16 offscreen cameras. |
| Connectome Model | Neural network based on MaleCNS v1.0 measured connectivity. |
| Limb Actuation | Joint commands for the NeuroMechFly simulated body. |
| Physical Controls | A passive steering wheel and spring-loaded pedals. |
| CARLA Vehicle | A simulated vehicle (currently a Mini Cooper) receiving physical inputs. |
Sources: README.md:43-46, README.md:14-15, THIRD_PARTY.md:41-45
Experimental Milestones
The project progresses through specific stages labeled E00 through E03, each addressing a narrow research question with distinct evidence requirements.
Phase 1: Physical Baseline (E00 - E02)
- E00: Body Dependency: Replays upstream NeuroMechFly motion targets to verify physical simulation in MuJoCo. No neural training occurs in this stage.
- E01: Pilot Steering: Establishes initial steering capability. Results from 2026-09-09 show 100/100 held-out targets passed for stationary steering.
- E02: Diagnostic IK: Uses inverse kinematics for diagnostic purposes to verify joint reach and contact mechanics.
Sources: scripts/body_baseline.py:1-6, README.md:10-12, THIRD_PARTY.md:46-48
Phase 2: Learned Control (E03)
- E03: Connectome Policy: Applies the trained connectome policy to joint commands.
- Skill Integration: The model currently operates the left foreleg to turn the wheel. Future work involves integrating pedal control (accelerator/brake) into the coordinated loop.
Sources: THIRD_PARTY.md:48-50, README.md:52-55
Technical Roadmap
The roadmap prioritizes replication and complexity scaling to achieve autonomous visual driving.
Current Status (Validated)
- Steering Skill: 600 optimizer updates over 186 seconds of training achieved precise steering control on a stationary wheel.
- CARLA Integration: Physical steering drives the CARLA vehicle on a shared neural/body/CARLA clock.
- Hardware Validation: Validated on Runpod A6000 and A40 instances using a packaged GPU runtime.
Sources: README.md:10-22, README.md:52-53
Next Objectives
- Three-Seed Replication: Replicate the current steering skill across two additional training seeds to ensure statistical robustness.
- Pedal Skill Training: Refine and teach the fly to operate the spring-loaded pedal.
- Coordinated Loop: Integrate learned pedal control with steering in the CARLA environment.
- Visual Driving: Transition from scripted turn requests to steering based on camera input.
- Panda Shell Construction: Build the "Flyat" (Fiat Panda) vehicle shell to replace the current stock Mini Cooper proxy.
Sources: README.md:52-57, README.md:6-8
Infrastructure and Presentation
The experiment requires strict synchronization between neural activity and physical simulation.
The sequence diagram illustrates how all components share the same recorded clock to ensure causal consistency.
Sources: README.md:17-18, README.md:65-67
Funding and Sustainability
The project uses a continuous live auction model to fund compute resources and future experiments. Supporters purchase ad placements on the vehicle livery.
- Entry Bid: Minimum $1 to outbid current owners.
- Target Goal: $100,000 for the full experiment.
- Ad Rotation: Placements are open-ended; artwork remains until outbid by $1 or more.
Sources: apps/website/src/components/auction-site.tsx:43-52, apps/website/src/app/terms/page.tsx:13-18
The experiment remains a research-focused project. It does not guarantee autonomous driving success but documents each step through verifiable metrics and open-source artifacts.
Sources: README.md:48-50, apps/website/src/app/terms/page.tsx:37-41