Artifact Validation and QC
Relevant source files
The following files were used as context for generating this wiki page:
Artifact Validation and Quality Control (QC) ensures the integrity, consistency, and accuracy of generated data, simulation recordings, and exported assets within the Flyhard project. This system verifies that simulation runs, neural traces, and 3D assets meet specific technical requirements before they are used for research reporting or public presentation.
The scope of QC covers automated checks for simulation health, visual artifact verification in CARLA and interior replays, and the integrity of build receipts and manifests. These processes validate that the trained connectome-based model causally operates the controls as expected during connected steering experiments.
Sources: README.md, scripts/verify_artifacts.py, scripts/qc_carla_video.py
Artifact Integrity Verification
The system validates the presence and integrity of files listed in manifests using SHA-256 hashing. This process ensures that artifacts such as body-trace.npz, neural-trace.npz, and various configuration files match the state recorded during the simulation run.
Manifest and Build Verification
Build receipts and manifests serve as the authoritative record for a specific experiment or runtime image. The verification logic compares current file hashes against these records to detect corruption or unauthorized modifications.
The diagram shows the logic for verifying artifact integrity against a JSON manifest.
Sources: reports/2026-09-09-runtime/build-receipt.json, reports/2026-09-09-clean-videos/source-manifest.json, scripts/export_interior_assets.py:20-22
Quality Control for Video and Simulation
QC scripts perform automated checks on simulation output to ensure synchronization and mechanical validity. These checks are applied to both the external CARLA environment and the internal cockpit simulation.
CARLA Video QC
The CARLA QC process validates that the simulated clock, neural model, and vehicle actions are synchronized. It specifically checks for:
- Frame Synchronization: Ensures every video frame shares a consistent neural/body/CARLA clock.
- Response Validation: Verifies that steering responses are caused by the neural model. For example, it checks that disconnecting the foot grip removes over 99% of the steering response.
- Visual Correctness: Confirms that the stock Mini Cooper proxy or eventual Panda shell renders correctly at 600 frames for a 24-second sequence.
Sources: README.md, scripts/qc_carla_video.py
Interior and Cockpit Validation
Validation of the interior replay focuses on the physical interaction between the fly body and the vehicle controls (wheel and pedals).
| Validation Check | Description | Target File |
|---|---|---|
| Grip Intervention | Verifies mechanical response when forefoot grip is applied/removed. | body-trace.npz |
| Contact Sole Physics | Validates fine timestep sensitivity for the spring pedal sole. | body-trace.npz |
| Mesh Coverage | Ensures projected surfaces (e.g. ad panels) have >90% coverage on bodywork. | layout-checks.json |
| Joint Servo Status | Confirms servos are disabled for passive elements like the steering wheel. | config.json |
Sources: README.md, scripts/validate_carla_cockpit.py, apps/mini-livery/rebuild-layout.py:86-88
Asset Export and Interchange QC
When preparing fly meshes and recorded poses for Unreal Engine or CARLA import, the system calculates and enforces error thresholds for coordinate transformations.
Transformation Roundtrip Error
The export_interior_assets.py script calculates the conversion error during coordinate transformation between MuJoCo world space and local body space. The script asserts that this error must remain below a strict threshold.
# scripts/export_interior_assets.py:68
assert conversion_error < 1e-10
Export Manifest Metrics
A manifest is generated for every asset export to track QC metrics:
- Vertex/Triangle Counts: Total geometry complexity for visibility and performance checks.
- Max Roundtrip Error: The highest recorded discrepancy in rig-to-car unit conversion.
- SHA-256 Trace Mapping: Linking the exported geometry to the specific
body-trace.npzsource.
Sources: scripts/export_interior_assets.py:68-100
Runtime Environment Health
Artifact validation extends to the packaged GPU runtime, ensuring the Python environment and graphics libraries are correctly installed and functional.
The sequence shows the validation steps performed when launching the packaged GPU runtime.
Sources: reports/2026-09-09-runtime/build.json, README.md
Key Validation Artifacts
| Artifact Name | Description | Source |
|---|---|---|
build-receipt.json | Records pinned package versions (e.g., torch 2.8.0, carla 0.9.16) and environment hashes. | reports/2026-09-09-runtime/ |
metrics.json | Contains numerical scores from hold-out targets (e.g., hold error limits). | runs/carla-calm-v2/ |
source-manifest.json | Maps script filenames to their specific SHA-256 hashes to ensure code reproducibility. | reports/2026-09-09-clean-videos/ |
Sources: reports/2026-09-09-runtime/build-receipt.json, reports/2026-09-09-clean-videos/source-manifest.json, README.md
Summary
Artifact Validation and QC in Flyhard provide a rigorous framework for maintaining the scientific validity of the experiment. By combining automated hash verification, mechanical physics validation, and strict coordinate transformation checks, the project ensures that the 165,122 neurons of the connectome-based model are accurately represented and causally responsible for the simulated vehicle control.