Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• package.json
• CONTRIBUTING.md
• README.md
• src/lib/tests/oliveRecipeSchema.test.ts
• src/lib/tests/recipePipeline.test.ts
• src/lib/oliveRecipeSchema.ts

The testing strategy for Olive Studio is designed to ensure the integrity of Microsoft Olive optimization recipes as they transition from UI state to executable JSON configurations. Given the project's role as a visual builder and runner, the strategy focuses heavily on structural validation, pipeline compatibility, and automated smoke tests to prevent invalid configurations from reaching the execution engine.

The strategy encompasses multiple layers of verification:

• Unit Testing: Validating core logic for recipe generation, schema compliance, and state sanitization.
• Validation Layer: Enforcing structural rules and cross-pass compatibility before execution.
• Smoke Testing: Verifying the end-to-end recipe builder logic via specialized scripts.
• Continuous Integration: Automating linting, type-checking, and test execution on every pull request.

Sources: README.md:121-127, CONTRIBUTING.md:15-19

Core Testing Layers

The project utilizes a multi-tiered approach to ensure code quality and functional accuracy. The following table summarizes the primary testing commands and their purposes:

Command	Tool	Purpose
npm run test	Vitest	Runs the suite of unit tests for library logic and schemas.
npm run lint	TypeScript (TSC)	Performs static type-checking to ensure API consistency.
npm run validate:recipe	TSX / Custom Script	Executes smoke tests for the recipe builder and schema.
npm run test:coverage	Vitest / V8	Generates code coverage reports for the test suite.

Sources: package.json:28-40, CONTRIBUTING.md:65-68

Unit Testing and Schema Validation

Unit tests are primarily located in src/lib/__tests__ and utilize the Vitest framework. These tests focus on the logic that transforms user input into valid Olive JSON. A critical component is the oliveRecipeSchema.ts, which defines the structural requirements for a valid recipe, including input_model, systems, passes, and engine definitions.

The diagram above illustrates the data flow from UI state to a validated JSON artifact, where schema checks act as a gatekeeper.
Sources: src/lib/oliveRecipeSchema.ts:25-95, src/lib/tests/recipePipeline.test.ts:39-85

Recipe Integrity and Compatibility

Validation architecture is structured to fail fast, preventing the long error traces typically associated with invalid Olive YAML/JSON.

Multi-Layer Validation Architecture

1. Pipeline Compatibility: Rules defined in pipelineValidation.ts manage pass ↔ execution provider (EP) relationships and perform auto-sanitization (e.g., ensuring AWQ passes only run on supported hardware).
2. Structural Schema: The validateOliveRecipeStructure function checks for mandatory fields and correct data types within the generated object.
3. Server Preflight: Before a process is spawned, the server performs a final JSON schema check and may run olive run --list_required_packages to verify environment readiness.

Sources: README.md:121-127, src/lib/oliveRecipeSchema.ts:25-50

Schema Requirement Specifications

The schema validation ensures that every recipe adheres to the following structural constraints:

Component	Required Fields	Constraints
input_model	type, config	type must be a non-empty string; config must be an object.
systems	local_system (or equivalent)	Must contain an accelerators array with at least one device.
passes	type	Each pass must use a known Olive 0.12.1 pass name.
engine	host, target	Must reference valid keys defined in the systems object.

Sources: src/lib/oliveRecipeSchema.ts:30-85, src/lib/tests/oliveRecipeSchema.test.ts:40-60

Continuous Integration (CI) and Deployment Checks

The project enforces a strict CI pipeline via GitHub Actions that runs on every push or pull request to the main or master branches.

The sequence diagram demonstrates the automated verification steps required for code to be accepted into the codebase.
Sources: README.md:170-174, CONTRIBUTING.md:65-68

Commit and Contribution Guidelines

Contributors are required to follow specific conventions to maintain testability:

• Conventional Commits: Use prefixes like feat:, fix:, or test: to categorize changes.
• Focused PRs: Smaller, reviewable PRs are preferred to maintain code clarity.
• Manual Smoke Testing: For UI or server changes, developers are encouraged to run "Execute Live" with a small CPU recipe to verify log streaming and process management.

Sources: CONTRIBUTING.md:33-40, CONTRIBUTING.md:78-85

Summary of Test Utilities

The testing strategy is supported by several internal utilities and configurations:

• vitest.config.ts: Configures the unit testing environment, including alias resolution and coverage thresholds.
• validate-recipe-builder.ts: A specialized script used for deep smoke testing the mapping between UI state and Olive JSON.
• oliveRecipeSchema.test.ts: Contains over 20 test cases covering edge cases such as empty accelerators, missing engine keys, and invalid pass configurations.

Sources: package.json:28-40, src/lib/tests/oliveRecipeSchema.test.ts:1-200

Testing in Olive Studio ensures that the complexity of Microsoft Olive's configuration requirements is abstracted safely behind a validated UI, providing developers with confidence that their generated recipes are runnable and structurally sound before high-resource optimization jobs begin.