Testing Framework
Relevant source files
The following files were used as context for generating this wiki page:
- crates/tools/architecture-tests/src/tests.rs
- architecture/19-architecture-conformance-and-regression-policy.md
- AGENTS.md
- CONTRIBUTING.md
- architecture/22.6-standard-library-implementation-plan.md
- crates/compiler/documentation/tests/documentation.rs
- architecture/07-implementation-roadmap.md
- crates/libraries/README.md
The Pop Lang testing framework is a multi-layered system designed to enforce architectural conformance, prevent regressions (specifically "Lua regressions"), and ensure cross-backend semantic equivalence. It follows a strict "Architecture → Tests → Implementation" workflow, where deterministic tests must be authored and observed failing before any feature implementation is permitted. Sources: AGENTS.md:36-39, CONTRIBUTING.md:27-31, architecture/19-architecture-conformance-and-regression-policy.md:126-130
The framework encompasses several specialized testing domains, including architecture-conformance tests, documentation tests, foundation library tests, and runtime/GC stress tests. All testing activities are unified under the pop test command. Sources: AGENTS.md:434-436, architecture/19-architecture-conformance-and-regression-policy.md:15-18
Test-Driven Architecture (TDA) Workflow
Every feature and behavior in the project follows a mandatory sequence. Architecture acts as the binding contract, and tests serve as the executable representation of that contract. Sources: AGENTS.md:214-220, CONTRIBUTING.md:27-31
The flow ensures that implementation never precedes its tests and that the smallest possible change is made to satisfy the accepted contract. Sources: AGENTS.md:214-225, CONTRIBUTING.md:27-33
Architecture Conformance Testing
Architecture tests are used to enforce structural invariants, dependency directions, and project-wide policies. These tests verify that components remain within their authorized boundaries. Sources: architecture/19-architecture-conformance-and-regression-policy.md:220-272, crates/tools/architecture-tests/src/tests.rs:1-12
Key Conformance Domains
| Domain | Purpose | Enforcement Logic |
|---|---|---|
| Dependency Direction | Ensures Pop.Standard does not depend on Pop.Internal. | Scans Cargo.toml for prohibited workspace inheritance. |
| Crate Inventory | Validates that all crates match the accepted ADR 0018 list. | Compares MEMBERS constant against root Cargo.toml. |
| Lua Regression | Prevents the language from sliding back into dynamic/table-based semantics. | Permanent negative test corpus for forbidden features. |
| Runtime Ownership | Enforces separation between PLRI, Collector, and Native ABI. | Verifies focused modules and restricted imports. |
Sources: crates/tools/architecture-tests/src/tests.rs:14-150, architecture/19-architecture-conformance-and-regression-policy.md:78-100, architecture/19-architecture-conformance-and-regression-policy.md:220-272
Lua Regression Suite
A critical component of the conformance policy is the prevention of "Lua regressions." These tests ensure that static typing and native object models are not bypassed by table-centered or dynamic mechanisms. Sources: architecture/19-architecture-conformance-and-regression-policy.md:78-83
The regression suite blocks releases if any change introduces these patterns. Sources: architecture/19-architecture-conformance-and-regression-policy.md:84-100, architecture/19-architecture-conformance-and-regression-policy.md:269
Documentation Testing
The framework includes specialized support for testing code examples embedded in XML documentation. These tests ensure that public library examples are not only syntactically correct but also semantically valid. Sources: architecture/20-xml-documentation-comments.md:143-146, crates/compiler/documentation/tests/documentation.rs:1-10
- Syntax: Use
<code language="pop" test="true">to mark testable examples. Sources: architecture/20-xml-documentation-comments.md:143 - Validation: Examples are parsed and type-checked in an isolated documentation test module. Sources: architecture/20-xml-documentation-comments.md:143-145
- Checklist: Public
Pop.StandardAPIs require at least one compiled example for non-trivial functionality. Sources: architecture/20-xml-documentation-comments.md:165-171
Runtime and Backend Testing
Runtime testing focuses on the Pop Lang Runtime Interface (PLRI) and Garbage Collection (GC) invariants. Sources: architecture/19-architecture-conformance-and-regression-policy.md:225-272, architecture/07-implementation-roadmap.md:104-125
Differential Testing
Differential tests are used to compare the observable behavior of different backends (LLVM vs. MIR Interpreter vs. VM). Sources: architecture/19-architecture-conformance-and-regression-policy.md:52-54, architecture/07-implementation-roadmap.md:126-128
Backend semantic disagreement is treated as a bug unless caused by documented target-specific capabilities. Sources: architecture/19-architecture-conformance-and-regression-policy.md:52-57, architecture/07-implementation-roadmap.md:127-128
GC and Stress Testing
GC testing includes specific stress tests for:
- Moving Nursery: Validates pointer relocation during nursery collections. Sources: architecture/07-implementation-roadmap.md:105-107
- SATB Marking: Ensures snapshot-at-the-beginning consistency during concurrent marking. Sources: architecture/07-implementation-roadmap.md:144-148
- Selective Evacuation: Tests failure-atomic relocation of objects from fragmented regions. Sources: architecture/07-implementation-roadmap.md:154-159
Foundation Library Testing
Contributions to Pop.Standard require focused tests by public API family. These tests are categorized as follows:
- Positive Behavior: Verification of expected functionality.
- Rejection Boundaries: Ensuring invalid inputs (e.g., wrong types, inaccessible declarations) are rejected.
- Convention/Consistency: Enforcing naming and documentation standards.
- Regression Coverage: Ensuring fixed bugs do not recur.
Sources: crates/libraries/README.md:61-66, architecture/22.6-standard-library-implementation-plan.md:18-24
Test Discovery
The pop-test-runner utilizes conventional source discovery to run tests against Pop.Standard Modules. For example, crates/tools/test-runner/tests/foundation_sources.rs provides coverage for the semantic pipeline of standard library sources. Sources: crates/libraries/README.md:67-70
Continuous Integration (CI) Gates
CI serves as the final arbiter of architectural conformance. Release gates include:
- HIR/MIR Verification: Rejection of forbidden dynamic or LLVM-leaking operations in IR.
- API Baselines: Comparison of
Pop.Standardagainst an exact prelude snapshot to prevent unauthorized changes. - Artifact Reproducibility: Ensuring
.poplibartifacts are bit-for-bit reproducible across independent builds.
Sources: architecture/19-architecture-conformance-and-regression-policy.md:220-272, architecture/22.6-standard-library-implementation-plan.md:11-16
Summary
The testing framework is the primary enforcement mechanism for Pop Lang's architecture-first philosophy. By integrating structural conformance, cross-backend differential checks, and strict Lua-regression prevention, it ensures the project maintains its native, statically typed identity while evolving through the ADR process. Sources: architecture/19-architecture-conformance-and-regression-policy.md:273-276, AGENTS.md:36-39