Experiment Lifecycle and Checkpointing
Relevant source files
The following files were used as context for generating this wiki page:
Experiment Lifecycle and Checkpointing manages the execution stages, data persistence, and state verification for training neural models within the Flyhard project. The system ensures that training runs—such as teaching a simulated fly connectome to drive in CARLA—are reproducible by saving checkpoints, metrics, and physical traces. This lifecycle tracks progress from initial physical baselines through integrated visual driving pilots.
The framework captures approximately 3 GB of peak training allocation and produces artifact manifests for every run to maintain data provenance. It allows the system to recover from failures, verify physical contact interventions, and replay simulation states with high precision.
Sources: README.md:1-24, README.md:46-52
Experiment Stages and Execution Flow
The project follows a verifiable experiment plan where each stage addresses a narrow research question. The lifecycle starts with dependency testing (E00) and progresses to connectome policy application (E03).
Execution Lifecycle
- Configuration: The script initializes with a
predeclaredconfiguration containing experiment IDs, claims, and thresholds. - Simulation Setup: MuJoCo or CARLA environments are instantiated with specific physical parameters (e.g.,
timestep=1e-4). - Physical Step Loop: The actor (fly model) performs actions, and the system records joint angles, contacts, and physical states.
- Trace Storage: The system compresses the resulting data into
.npzfiles for later replay or analysis. - Metric Reporting: A final
metrics.jsonrecords wall time, simulation performance, and pass/fail status.
Sources: scripts/body_baseline.py:38-66, README.md:38-44, THIRD_PARTY.md
Experiment Flow Diagram
The following diagram illustrates the transition from raw configuration to validated experiment artifacts.
The diagram shows the sequential flow of data from configuration to final metric recording. Sources: scripts/body_baseline.py:53-100
Checkpointing and Data Persistence
Checkpointing ensures that model training can be resumed and results can be audited. The system saves the model weights and the entire state of the neural/body/simulation clock.
Primary Artifacts
| Artifact Type | File Format | Description |
|---|---|---|
| Model Checkpoint | .pt | PyTorch weights and optimizer state (e.g., checkpoint.pt). |
| Physical Trace | .npz | Compressed arrays of time, qpos, ctrl, and contacts. |
| Manifest | .json | SHA-256 hashes of all generated files to ensure integrity. |
| Metrics | .json | Execution statistics, performance, and version metadata. |
Sources: README.md:46-52, reports/2026-09-09-runtime/artifact-manifest.json, reports/2026-09-09-clean-videos/artifact-manifest.json:115-118
State Validation
The lifecycle includes an explicit validation step. The system reloads saved .npz traces and compares them against the internal memory state to prevent silent corruption. For example, body_baseline.py asserts that reloaded qpos and time arrays are identical to those generated during the physics loop.
np.savez_compressed(out / "trace.npz", time=times, qpos=qpos, ctrl=controls, contacts=contacts)
loaded = np.load(out / "trace.npz")
assert np.array_equal(loaded["qpos"], qpos)
assert np.array_equal(loaded["time"], times)
Sources: scripts/body_baseline.py:68-71
Runtime Environment and Provenance
To ensure experiments run identically across different hardware (e.g., Runpod A6000 vs. A40), the project utilizes a packaged GPU runtime.
Environment Tracking
The system tracks all package versions and source revisions to maintain the "Build Receipt." This receipt links specific neural activity results to the code version that produced them.
- Version Pinning: Packages like
torch 2.8.0+cu128,mujoco 3.9.0, andflygym 2.1.0are explicitly recorded. - Source Hashing: Every script and source file in the
src/flyhard/directory is hashed. - Runtime Graphics: The system verifies offscreen cameras and MuJoCo rendering capabilities upon launch.
Sources: reports/2026-09-09-runtime/build-receipt.json:1-60, README.md:26-30
Build and Deployment Flow
This sequence shows how code is packaged into a verified image before deployment for experiment execution. Sources: reports/2026-09-09-runtime/build.json, README.md:26-30
Result Verification and Replay
The project implements exact saved state replay. Instead of running a separate animation loop, the replay mechanism loads physical states (qpos) from the checkpointed trace directly into the MuJoCo renderer.
Replay Logic
- Clock Synchronization: The replay uses the saved
timearray from the simulation. - Error Checking: The system calculates the
max_qpos_errorbetween the recorded state and the reloaded physical model. - Visual Evidence: Videos are generated at a fixed 30 FPS, overlaying simulation time and project metadata using the Geist typeface.
Sources: scripts/body_baseline.py:73-95, README.md:58-60, THIRD_PARTY.md
Conclusion
The Experiment Lifecycle and Checkpointing system provides a rigorous framework for conducting and documenting neuro-mechanical simulations. By enforcing strict artifact manifests, environment receipts, and physical state validation, the Flyhard project ensures that progress toward autonomous visual driving is verifiable and reproducible across various computing environments.