Compliance & Auditing
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Compliance & Auditing system within the Engineering Platform provides a robust framework for validating engineering standards, code protection levels, and test authenticity. Its primary purpose is to ensure that all project contributions adhere to established organizational standards, maintain structural integrity, and avoid "hallucination" in automated testing environments.
The scope of this system covers automated shell-based compliance checks, deep static analysis of test suites, and hierarchical code protection audits. These mechanisms work together to provide engineering health assessments and enforce safety guardrails before code is merged into production branches.
Compliance Architecture
Compliance is enforced through a combination of automated scripts and specific AI skills designed to scan, verify, and report on the state of the repository.
Engineering Health Assessment
The project uses a structured audit methodology (e.g., 5-Phase Engineering Platform Audit Kit) to assess the health of the codebase, including code quality, CI/CD status, and git patterns.
Sources: audits/legalai-system/phase1-audit.json:118-125
Automated Compliance Checking
The check-compliance.sh utility serves as the primary gateway for validating a project against platform standards. It performs checks across several categories:
| Category | Description | Severity |
|---|---|---|
| Essential Files | Validates presence of package.json, README.md, and CLAUDE.md. | ERROR/WARN |
| Linting | Checks for .eslintrc.js and .prettierrc. | WARN |
| Testing | Verifies vitest.config.ts and existence of a test script. | WARN |
| Git Hooks | Checks for .husky/pre-commit implementation. | WARN |
| Security | Scans for hardcoded API_KEY, SECRET, or PASSWORD. | ERROR |
Sources: scripts/check-compliance.sh:23-86
System Workflow
The following diagram illustrates how the compliance and audit tools interact with the repository to produce health reports and enforcement actions.
The diagram shows the parallel execution paths of auditing, compliance checking, and test validation.
Sources: scripts/check-compliance.sh:88-106, scripts/test-determinism-gate.js:415-430, audits/legalai-system/phase1-audit.json:96-116
Code Protection Auditing
The protection system categorizes code into stability tiers to prevent regressions in critical paths.
Protection Levels
| Level | Enforcement Logic | Usage Context |
|---|---|---|
@immutable | Never change | External APIs, core data structures, published interfaces. |
@protected | Refactor only | Working features passing manual tests; behavior must stay same. |
@maintainable | Bug fixes allowed | Features with known edge case issues but working core logic. |
Sources: skills/protect/SKILL.md:11-25
The Audit Workflow
The protection-audit skill involves scanning the codebase for JSDoc-style annotations and verifying them against a central CODE_PROTECTION_POLICY.md.
- Scan: Uses
scripts/audit-protection.jsto identify@protectedand@immutabletags. - Verify: Compares findings against the policy document to identify discrepancies.
- Report: Generates coverage metrics (e.g., "30.6% Coverage").
- Identify Gaps: Cross-references manual test results with protection levels to find unprotected passing features.
Sources: skills/protection-audit/SKILL.md:31-180
Test Determinism & Hallucination Gate
A critical component of the auditing system is the Determinism Gate, which identifies "hallucinating tests"—regression tests that pass even when the backend is inaccessible.
Anti-Pattern Detection
The system performs static analysis to find specific code patterns that lead to false positives:
- Catch Swallow: Swallowing errors via
.catch(() => false). - Multi-Status: Accepting 200, 404, and 503 status codes in the same assertion.
- If-Visible Guard: Using
if (await locator.isVisible())which causes tests to pass silently if an element is missing. - OR Truthiness: Using
expect(A || B || C).toBeTruthy().
Sources: scripts/test-determinism-gate.js:28-111
Runtime Determinism Logic
The audit can be run in --runtime mode, which executes Playwright tests while blocking all API routes (returning 503 Service Unavailable). If a test still passes in this state, it is flagged as "hallucinating" and failing the determinism audit.
Sources: scripts/test-determinism-gate.js:335-360
The sequence shows how the runtime gate forces backend failures to validate test sensitivity.
Sources: scripts/test-determinism-gate.js:338-380
Audit Reporting
The system produces structured outputs for architectural review.
Audit Findings Structure
Audits are summarized in JSON format to track engineering metrics over time:
{
"project": "LegalAI_System",
"audit_type": "Engineering Health Assessment",
"findings": {
"strengths": ["Strong testing culture", "Modern stack"],
"weaknesses": ["Excessive console.log usage", "No CI enforcement"]
},
"anti_patterns": [
{
"pattern": "console.log proliferation",
"severity": "medium",
"recommendation": "Migrate to winston/pino"
}
]
}
Sources: audits/legalai-system/phase1-audit.json:1-116
Conclusion
Compliance & Auditing in the Engineering Platform is a multi-layered defense strategy. It moves from basic file-system compliance via shell scripts to complex behavioral auditing of the test suite. By combining engineering health assessments with strict code protection levels and determinism gates, the system ensures that software remains stable, maintainable, and verifiable across different development cycles. This centralized approach allows the platform to maintain a "Health Score" and provide actionable recommendations for technical debt reduction.