Testing & Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
The marginalia project uses a multi-layered testing strategy to verify the correctness of AI-generated code reviews, ensure the stability of the studio web application, and validate the contract between the polling bot and Claude Code commands. The system relies heavily on fixtures—static files representing expected inputs and outputs—to simulate real-world review scenarios without requiring live GitLab or Anthropic API connections during test execution.
Core Testing Architecture
The project organizes tests into three main categories: unit/integration tests for the Node.js poller, visual and component tests for the React studio, and contract tests for AI skills.
The diagram above shows the relationship between different test commands and the fixture-based validation system for AI skills.
Sources: package.json:11-20, README.md:57-60
Test Commands
| Command | Component | Description |
|---|---|---|
npm test | Core Bot | Runs Node.js native tests against .test.mjs files. |
npm run doctor | Environment | Performs read-only checks of the setup and configuration. |
npm run studio:storybook | Studio UI | Launches Storybook for component-level visual testing. |
npm run doctor -- --dry-run <iid> | Skill Logic | Simulates a real review chain on a specific Merge Request (MR). |
Sources: package.json:15-20, README.md:21-25
Skill & Report Fixtures
The bot verifies that Claude Code commands follow the "Output Contract" by comparing command output against fixtures stored in test/fixtures/skills/. This ensures that any new way of writing reviews remains compatible with the studio's parsing logic.
Fixture Structure
Each fixture folder contains the necessary files to simulate a complete review cycle:
report.md: The raw Markdown output expected from a review command.stdout.txt: Simulated standard output, often containing<<<REVIEW_FILE>>>blocks.expect.json: The "normalized shape" the bot must produce (verdict, summary, and findings).findings.json: (Optional) The raw JSON a command might write directly.
Sources: README.md:57-60, test/fixtures/skills/marker-block/stdout.txt:1-6
Contract Validation Flow
The validation process ensures that information from the AI is correctly transformed into the ReviewShape data structure used by the Studio API.
This sequence illustrates how the npm test command uses fixtures to verify the parsing logic within the project.
Sources: README.md:57-60, studio/src/lib/api.ts:86-96
Studio Component Testing (Storybook)
The Marginalia studio uses Storybook to document and test its design system, known as Scholia. This allows developers to verify presentational components in isolation against the project's design principles.
- Visual Consistency: Verifies colors, typography, and spacing defined in
DESIGN.md. - Theme Testing: Supports testing components in both light and dark modes, which follow the operating system settings.
- Reduced Motion: Components are tested to ensure they respect the
prefers-reduced-motionCSS media feature.
Sources: README.md:154-159, DESIGN.md:181-185
Diagnostic & Environment Testing
The doctor script serves as a pre-flight testing tool for the local environment. It identifies missing requirements and misconfigurations before the bot attempts to run.
- Setup Verification: Checks GitLab URL, project ID, and token validity.
- Command Validation: Verifies that the
REVIEW_COMMANDandTRIAGE_COMMANDare accessible. - Sandbox Testing: Checks if tools are allowed under the
CLAUDE_EXTRA_ALLOWED_TOOLSconfiguration. - Dry Runs: Executing
npm run doctor -- --dry-run <iid>tests the entire execution chain by fetching a real MR and writing the resulting report to a temporary file without posting to GitLab.
Sources: README.md:21-25, README.md:46-48, README.md:195-200
Manual Verification Guidelines
The SKILL.md file defines a verification protocol that the AI agents themselves must follow during a review. This constitutes a form of "runtime testing" performed by the AI:
- Verification over Guessing: Agents are instructed to run tests, type checks, and linters defined in the target repository.
- Evidence Collection: Failures must be reported with the actual output as evidence in the finding.
- Throwaway Checks: For behavioral changes, agents write and run temporary checks to confirm suspicions.
Sources: skills/marginalia-review/SKILL.md:32-41
Testing and fixtures in Marginalia ensure that the local studio can reliably display reviews from diverse AI commands. By using a contract-based approach validated by file-system fixtures, the project maintains a strict separation between the AI's "taste" and the technical requirements of the review platform.