Testing Strategies
Relevant source files
The following files were used as context for generating this wiki page:
Scoutline employs a multi-tiered testing strategy to ensure the reliability, security, and accuracy of its research CLI. The system validates everything from local file parsing and output budgeting to live Model Context Protocol (MCP) integrations and package distribution.
The project distinguishes between offline unit/integration tests and opt-in live tests. Offline tests use hermetic environments with injected dependencies, such as fake file systems and IO doubles, to ensure deterministic results. Live tests exercise real API transports and provider endpoints to guard against regression in third-party integrations.
Sources: packages/scoutline/tests/output-budget-t4.test.js:20-25, packages/scoutline/tests/mcp-live.test.js:25-30
Tiered Testing Architecture
The testing architecture is divided into specialized domains, each targeting a specific layer of the application.
Unit and Logic Testing
Unit tests verify the internal logic of modules like the output budgeter, section differ, and local context parser. These tests use node:test and node:assert/strict. To maintain hermeticity, tests often use withTempDir for isolated file system operations and hermeticMainDeps to prevent access to ambient user configurations.
Sources: packages/scoutline/tests/output-budget-t4.test.js:20-27, packages/scoutline/tests/package.test.js:33-35
Regression Guarding of Documentation
Scoutline treats documentation as a testable surface. Tests verify that README files, architecture diagrams, and help text accurately reflect the current state of the CLI. This includes byte-identical pins for critical rule texts and schema version checks in SKILL.md.
Sources: packages/scoutline/tests/science-docs.test.js:10-25, packages/scoutline/tests/agent-registration-docs.test.js:10-20
Mermaid Diagram: Test Execution Flow
The following diagram illustrates how tests are integrated into the development and release lifecycle.
The flow ensures that the CLI is built before testing, and offline gates are cleared before any live or release-related activities occur.
Sources: CONTRIBUTING.md:7-12, RELEASING.md:14-20
Specialized Testing Domains
| Domain | Focus Area | Key Tools/Methods |
|---|---|---|
| Output Budgeting | Trimming and compaction of large payloads | READ_LADDER, CRAWL_LADDER, applyBudget |
| MCP Integration | Discovery and execution of Model Context Protocol tools | ZaiMcpClient, listTools, callToolRaw |
| Package Safety | Tarball contents and install-time side effects | npm pack --dry-run, installTarballOffline |
| Provider Parity | Comparison between direct transport and legacy SDKs | assertNoHeaderEcho, fetchMiniMaxSearch |
Sources: packages/scoutline/tests/output-budget-t4.test.js:30-40, packages/scoutline/tests/mcp-live.test.js:55-65, packages/scoutline/tests/package.test.js:15-25
Live Integration Testing
Live tests are opt-in and require environment variables like ZAI_LIVE_TESTS=1 and specific API keys. They perform real network requests to verify:
- Tool Discovery: Ensuring that the MCP client correctly discovers core tools like
web_search_primeandwebReader. - Transport Parity: Confirming that direct HTTP transport yields response envelopes equivalent to legacy SDKs without echoing sensitive request headers into the body.
- Vision Conformance: Validating specialized vision operations (e.g.,
analyze_image,ui_to_artifact) against live endpoints.
Sources: packages/scoutline/tests/mcp-live.test.js:10-20, packages/scoutline/tests/mcp-live.test.js:280-300
Sequence Diagram: MCP Live Test Sequence
The following sequence shows the interaction between the test suite and the Z.AI MCP infrastructure.
This sequence verifies tool discovery and the public dotted-name resolution to internal identities.
Sources: packages/scoutline/tests/mcp-live.test.js:88-120
Package and Distribution Testing
The project implements an "offline gate" for package publishing. The prepublishOnly script triggers npm run test:offline, which performs:
- Pack Dry Run: Checks that
package.jsonallowlists onlybin,dist, andskillsfolders while excluding tests and credentials. - Tarball Install Round-trip: Installs a locally generated
.tgzinto a fresh directory to verify that the CLI binary is executable and the library exports are load-safe without side effects. - Dependency Pinning: Ensures that
mmx-cliand other sensitive dependencies are locked to exact versions.
Sources: packages/scoutline/tests/package.test.js:10-25, packages/scoutline/package.json:28-35
Conclusion
The Scoutline testing strategy emphasizes deterministic offline validation and rigorous documentation guarding. By combining these with opt-in live integration tests and automated package distribution checks, the project maintains high standards for investigative-research software while allowing for rapid iteration on research capabilities.
Sources: packages/scoutline/tests/output-budget-t4.test.js:20-30, RELEASING.md:1-10