Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:The Zen MCP Server project employs a comprehensive, multi-layered testing strategy to ensure code quality, functional correctness, and API compatibility. The approach is designed to provide fast feedback during development while also validating complex, real-world scenarios through end-to-end testing. The strategy encompasses unit tests for isolated components, integration tests for provider interactions, and full-system simulator tests for workflow validation.
This tiered approach balances speed and coverage, enabling developers to run quick, local checks before committing, and leveraging more extensive tests in CI/CD pipelines to guarantee stability and prevent regressions.
Testing Layers
The project's testing is structured into three distinct layers, each with a specific scope and purpose.
This diagram illustrates the hierarchical testing strategy, from isolated unit tests to full end-to-end simulator tests.
Sources: tests/CLAUDE.md, simulator_tests/CLAUDE.md, AGENTS.md
Unit Tests
Unit tests form the foundation of the testing pyramid. They are designed to be fast, isolated, and free of external dependencies like network calls or API keys. These tests focus on individual modules and functions, using mocks and fixtures to simulate external services. The primary goal is to verify the correctness of core business logic.
Sources: tests/CLAUDE.md, AGENTS.md
Integration Tests
Integration tests validate the interaction between the server and external AI providers. They are marked with the integration pytest marker and typically run against a local model server like Ollama to avoid costs and rate limits associated with production APIs. These tests require environment variables to be set for the API endpoint.
Sources: run_integration_tests.sh, tests/CLAUDE.md, AGENTS.md
Simulator Tests
Simulator tests are end-to-end workflow validations that mimic real user interactions with the Zen MCP Server. They use actual API keys and models to test complex scenarios, such as cross-tool conversation continuity and multi-model consensus. These are the most comprehensive tests and are crucial for ensuring the system works as a whole.
Sources: simulator_tests/CLAUDE.md, simulator_tests/base_test.py
Test Execution
Tests are executed using pytest. Different test layers can be targeted using command-line arguments and pytest markers.
| Test Type | Command | Description |
|---|---|---|
| All Unit Tests | python -m pytest tests/ -v -m "not integration" | Runs all tests that are not marked as integration. This is the default pre-commit check. |
| Integration Tests | ./run_integration_tests.sh | A script that sets up the environment and runs tests marked with integration. |
| Simulator Tests (Quick) | python communication_simulator_test.py --quick | Runs a small, curated set of 6 core end-to-end tests for rapid validation. |
| Simulator Tests (Full) | python communication_simulator_test.py | Runs the complete suite of simulator tests. |
| Individual Test | pytest tests/test_file.py::TestClass::test_method | Runs a specific test function. |
Sources: AGENTS.md, run_integration_tests.sh, simulator_tests/CLAUDE.md
Pytest Markers
The project uses pytest markers to categorize tests, allowing for selective execution.
| Marker | Description |
|---|---|
unit | A unit test that is fast and has no external dependencies. |
integration | An integration test that requires API keys or network access. |
slow | A test that is known to be time-consuming. |
network | A test that requires a network connection. |
Source: tests/CLAUDE.md
Framework and Dependencies
The testing framework is built on pytest and several key plugins.
| Dependency | Version | Purpose |
|---|---|---|
pytest | >=8.0.0 | The core testing framework. |
pytest-asyncio | >=0.23.0 | Provides support for testing asyncio code. |
pytest-cov | >=5.0.0 | Generates code coverage reports. |
pytest-mock | >=3.0.0 | Provides a mocker fixture for patching and mocking. |
vcrpy | >=6.0.0 | Records and replays HTTP interactions to cassettes. |
responses | >=0.25.0 | A utility for mocking out requests library calls. |
Source: pyproject.toml
Unit Testing and Fixtures
Unit tests are located in the tests/ directory. Global fixtures and configurations are defined in tests/conftest.py. A key fixture is mock_provider, which simulates an AI provider for tests that do not require real API calls.
# tests/conftest.py:13-16
@pytest.fixture
def mock_provider():
"""模拟AI提供商用于单元测试"""
This fixture allows tests to run in isolation without needing to connect to an actual AI service.
Source: tests/conftest.py:13-16, tests/CLAUDE.md
HTTP Request Recording (Cassettes)
To ensure test stability and repeatability for tests that do involve network requests, the project uses vcrpy. This tool records HTTP requests and their corresponding responses into YAML or JSON files called "cassettes". On subsequent test runs, vcrpy intercepts the HTTP calls and replays the recorded responses from the cassette files, eliminating the need for a live network connection and ensuring deterministic test outcomes. These cassettes are stored in directories like tests/openai_cassettes/ and tests/gemini_cassettes/.
Sources: tests/CLAUDE.md, tests/openai_cassettes/consensus_step1_gpt5_for.json, tests/gemini_cassettes/consensus/step2_gemini25_flash_against/mldev.json
Integration Testing Workflow
Integration tests are managed by the run_integration_tests.sh script. This script sets the necessary environment variables and invokes pytest to run only the tests marked as integration.
The following diagram shows the flow for running integration tests.
Source: run_integration_tests.sh
The script's core logic is as follows:
# run_integration_tests.sh:10-14
# Default to local Ollama if no custom URL is provided
export CUSTOM_API_URL="${CUSTOM_API_URL:-http://localhost:11434}"
# Run integration tests
.zen_venv/bin/activate && pytest tests/ -v -m "integration"
This setup ensures that integration tests are consistently run against a configurable, and typically local, AI model endpoint.
Source: run_integration_tests.sh:10-14
Simulator Testing Architecture
The simulator framework is the most advanced part of the testing strategy, designed for end-to-end validation.
Core Components
communication_simulator_test.py: The main test runner that discovers and executes all simulator test cases.simulator_tests/base_test.py: A base class (BaseTest) that provides the foundational infrastructure for all simulator tests. It includes methods for sending messages, managing conversation threads, and asserting outcomes.simulator_tests/log_utils.py: A utility for monitoring server logs in real-time during test execution to catch errors or unexpected behavior.
Sources: simulator_tests/CLAUDE.md, simulator_tests/base_test.py
Simulator Test Execution Flow
The following sequence diagram illustrates a typical simulator test run.
This diagram shows how the test runner invokes a test case, which uses the BaseTest infrastructure to communicate with the live server, which in turn calls a real AI provider.
Sources: simulator_tests/base_test.py, simulator_tests/CLAUDE.md
Quick Mode
To provide rapid feedback, the simulator has a "quick mode" that runs a subset of 6 high-priority tests covering the most critical workflows.
| Quick Mode Test Case | Area of Focus |
|---|---|
cross_tool_continuation | Validates that conversation context is maintained across different tools. |
conversation_chain_validation | Ensures the integrity of the conversation history and memory. |
consensus_workflow_accurate | Tests the multi-model consensus tool for accuracy. |
codereview_validation | Verifies the functionality of the code review tool. |
planner_validation | Checks the task planning tool's ability to break down requests. |
token_allocation_validation | Confirms that token budgets are managed correctly. |
Source: simulator_tests/CLAUDE.md
Summary
The testing strategy for Zen MCP Server is robust and multi-faceted, ensuring high confidence in code changes. It combines fast, isolated unit tests for core logic, targeted integration tests for provider compatibility, and comprehensive end-to-end simulator tests for real-world workflow validation. This structured approach is critical for maintaining the quality and reliability of a complex system that interacts with multiple external AI services.
Sources: AGENTS.md, tests/CLAUDE.md, simulator_tests/CLAUDE.md