CI/CD Pipelines & Automations
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The CI/CD (Continuous Integration/Continuous Deployment) system in NeuralDBG acts as an automated "gatekeeper" designed to enforce code quality, security standards, and architectural integrity. By moving validation from local environments to the cloud, the project ensures that human error does not lead to the merging of broken or insecure code. This system is critical for maintaining the "Hub & Spokes" architecture, where the core logic (Hub) must remain stable against external dependency churn.
The automation suite encompasses multi-platform testing, security scanning, rule synchronization across branches, and specialized MLOps workflows such as experiment tracking and automated documentation publishing. These pipelines are governed by strict enforcement rules, including the "CI/CD Debugging First" protocol, which prioritizes pipeline health over feature development.
Sources: infrastructure_planning/milestone_0_tasks.md:1-26, copilot-instructions.md:46-52, GAD.md:273-286
CI/CD Enforcement Gates
NeuralDBG implements mandatory validation gates within GitHub Actions to ensure every contribution meets the project's technical and security baselines. These gates function as "unbreakable digital bouncers" that reject pushes if criteria are not met.
Core Validation Requirements
The automated pipelines enforce the following strict requirements:
- Test Coverage: A minimum of 60% code coverage is required for all PRs.
- Security Scanning: Mandatory execution of
banditandsafetyto detect vulnerabilities in code and dependencies. - Static Analysis: Integration of CodeQL, SonarQube, and Codacy for deep structural analysis.
- Cross-Platform Support: Workflows must execute successfully on both Linux and Windows runners.
Sources: copilot-instructions.md:65-71, infrastructure_planning/milestone_0_tasks.md:44-48, infrastructure_planning/milestone_25_tasks.md:10-14, CHANGELOG.md:21-23
The CI Gatekeeper Flow
The following diagram illustrates how the CI/CD pipeline prevents unstable code from reaching the main branch:
The diagram shows the logic where both coverage and security checks must pass to allow a merge.
Sources: infrastructure_planning/milestone_0_tasks.md:16-36, copilot-instructions.md:65-71
Rule Synchronization Automation
A specialized automation layer manages the synchronization of AI instructions and project rules across different branches and repositories. This ensures global parity for the project's "Physical Laws."
Multi-Layer Sync Defense
The synchronization procedure utilizes four distinct layers to maintain consistency:
- Automated Sync (GitHub Actions): Pushes to
ceo/branches trigger a workflow (rule-sync.yml) that propagates changes tomainandinfra/branches. - Mandatory PR Templates: Checklists in PR templates force verification of rule parity.
- Manual Sync Script: A Python utility (
scripts/sync_rules.py) serves as a fallback for emergency synchronization. - AI Session Verification: AI agents are programmed to check rule currency at the start of every session.
Sources: infrastructure_planning/RULE_SYNC_PROCEDURE.md:13-35, copilot-instructions.md:165-171
Rule Authority Matrix
Automations enforce different permissions based on branch scope:
| Branch Scope | Rule Modification | Code Modification |
|---|---|---|
ceo/** | Allowed | Allowed |
infra/** | Merge Only | Allowed |
feat/** | Merge Only | Allowed |
main | Merge Only | Protected |
Sources: infrastructure_planning/RULE_SYNC_PROCEDURE.md:37-46, GAD.md:243-255
MLOps & Documentation Automations
Beyond standard CI, the project automates MLOps workflows and administrative reporting tasks to reduce developer cognitive load.
Experiment and Artifact Tracking
The automation roadmap includes the integration of tools to manage the machine learning lifecycle:
- Experiment Tracking: Automated logging of loss curves and gradients using MLflow or Weights & Biases, replacing manual log parsing.
- Data Versioning (DVC): Automations to handle large artifacts like synthetic data and model weights, keeping the Git repository lean.
Sources: infrastructure_planning/milestone_0_tasks.md:50-59, infrastructure_planning/milestone_50_tasks.md:10-18
Session Summary Syncing
To maintain traceability across different platforms, a specialized automation syncs session documentation:
- Google Docs Integration: A Python script (
scripts/publish_session_summary_to_gdocs.py) convertsSESSION_SUMMARY.mdinto formatted Google Docs content. - Scheduled Sync: A GitHub Action (
publish-summary-to-google-docs.yml) runs on a schedule (weekdays at 22:30 UTC) to ensure external stakeholders have access to the latest summaries.
Sources: GOOGLE_DOCS_SYNC.md:1-26, CHANGELOG.md:12-14
Operational Protocols
The CI/CD system is governed by a set of operational rules that dictate developer behavior when pipelines fail.
Rule 39: CI/CD Debugging First
This mandatory protocol requires that if any CI/CD check fails, all feature development must stop immediately. The AI agent and developer must prioritize fixing the pipeline.
The sequence diagram illustrates the prioritization of pipeline health over feature implementation.
Sources: copilot-instructions.md:188-203, GAD.md:273-286, CHANGELOG.md:16-16
Summary
The CI/CD Pipelines & Automations in NeuralDBG serve as the foundation for project reliability and scalability. By enforcing 60% test coverage, mandatory security scans, and automated rule synchronization, the system minimizes technical debt and security risks. Specialized automations for MLOps and documentation further streamline the development process, allowing the core team to focus on causal inference research while maintaining professional-grade delivery standards.
Sources: infrastructure_planning/milestone_0_tasks.md:7-14, copilot-instructions.md:92-113