Testing Strategies & CI

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/test_ci_benchmark_regression.py
• scripts/check-udeps.sh
• CONTRIBUTING.md
• oxidize-core/tests/architecture_docs.rs
• oxidize-c/CONTRIBUTING.md
• PRD.md
• CLAUDE.md

Testing and Continuous Integration (CI) within the oxidize project ensure performance stability, numerical accuracy, and code quality across multiple language ports (Rust, C, C++, Go, Python). The project enforces a strict quality gate where all pull requests must pass workspace tests, linting, formatting, and benchmark regression checks before review.

This system relies on localized testing conventions—such as co-locating tests in Rust or paired files in C—and centralized CI scripts for performance gating and license auditing.

Workspace Quality Gates

The project utilizes a centralized Makefile to manage quality checks. Developers must run these checks locally to ensure a pull request is ready for CI validation.

Command	Action
make test	Runs all workspace tests (cargo test --workspace).
make lint	Runs clippy to identify and fix common Rust errors.
make fmt	Checks code formatting against project standards.
make udeps	Identifies unused Cargo dependencies (requires nightly).
make audit	Audits licenses and security vulnerabilities via cargo-deny.
make check	Runs formatting, linting, and tests in sequence.
make ci	Runs the full local quality gate, including builds.

Sources: CONTRIBUTING.md:20-30, README.md:120-130, CLAUDE.md:130-140

Multi-Language Testing Strategies

The oxidize workspace implements specific testing conventions for each of its supported language ports to maintain parity with the core Rust engine.

Rust Core Testing (oxidize-core)

Rust tests use the standard #[cfg(test)] convention, where test modules reside at the bottom of the source files they validate. The project also includes dedicated architecture documentation tests that verify the presence of required sections in global documentation files.

C Port Testing (oxidize-c)

The C11 port uses the Criterion 2.4.3 framework. Every source file in src/ must have a corresponding test file in tests/ using the test_<module>.c naming convention.

• Isolation: Criterion forks each test into its own process to prevent global state leaks.
• Sanitizers: Tests compile with -fsanitize=address,undefined (ASan/UBSan) to detect memory leaks and undefined behavior.
• Parity Tests: The C port prioritizes bit-exact parity tests, comparing C output against captured Rust reference fixtures stored in oxidize-core/tests/fixtures/.

Sources: oxidize-c/CONTRIBUTING.md:28-80

Performance & Benchmark Regression

Performance-sensitive changes, such as new kernels or quantization schemes, require benchmark evidence. The project uses criterion.rs for local benchmarks and custom Python scripts for regression detection.

Continuous Integration Pipeline

The CI system, implemented via GitHub Actions, validates the project across Linux, macOS, and Windows. It targets multiple architectures including ARM64 and WASM32.

CI Tasks and Objectives

• Build Validation: Verifies cross-platform compilation support.
• Dependency Auditing: Uses scripts/check-udeps.sh and cargo deny to ensure workspace cleanliness and security.
• Numerical Accuracy: Validates quantization output against the llama.cpp reference implementation.
• Remote Benchmarking: Supports benchmarking on specific hardware (e.g., dual-socket Xeon NUMA nodes) using remote scripts to ensure performance on commodity hardware.

Sources: PRD.md:1.2-1.8, PRD.md:16.10, scripts/check-udeps.sh

Summary of Testing Tools

Tool	Purpose	Source File Reference
criterion.rs	Rust benchmark harness.	PRD.md:1.4
Criterion (C)	Unit testing framework for C11 port.	oxidize-c/CONTRIBUTING.md:28
cargo deny	License and security auditing.	PRD.md:1.7
memmap2	Memory-mapped file loading validation.	PRD.md:2.3
ASan / UBSan	Memory and behavior sanitizers for C code.	oxidize-c/CONTRIBUTING.md:72

The testing strategy emphasizes that the Rust oxidize-core serves as the source of truth for behavior, numerics, and naming across all other ports and interfaces. Sources: oxidize-c/CONTRIBUTING.md:107