Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The pi-mono repository employs a multi-layered testing strategy that combines end-to-end (E2E) tests against live LLM providers, integration tests for agent capabilities, and unit tests for individual components and tools. The primary testing frameworks are Vitest and the native Node.js test runner.
A key aspect of the strategy is the conditional execution of tests that require API keys. These tests are designed to be skipped in CI environments to prevent key exposure, but can be run locally by developers who have the necessary credentials configured. This pragmatic approach ensures both security in the CI/CD pipeline and comprehensive validation during local development.
Sources: README.md:21-31
Frameworks and CI
The project uses a combination of test runners and a structured CI process to maintain code quality.
Test Runners
- Vitest: Used for testing in packages like
ai. - Node.js Test Runner: Used for testing in packages like
tui.
Sources: packages/ai/test/stream.test.ts:4, packages/tui/test/markdown.test.ts:2
Continuous Integration (CI)
The CI process, run via GitHub Actions, executes npm run check (lint, format, type-check) and npm run test for each package in parallel. A critical policy is that tests requiring LLM API keys are skipped during CI runs to prevent accidental exposure of secrets. This is achieved using describe.skipIf() blocks that check for the presence of environment variables.
Sources: README.md:21-31, packages/ai/test/stream.test.ts:383
This diagram illustrates the CI testing workflow.
Sources: README.md:21-31
End-to-End LLM Testing
The packages/ai directory contains an extensive E2E test suite that validates the unified API against multiple LLM providers. These tests cover a wide range of functionalities to ensure consistent behavior across different models.
Sources: packages/ai/test/stream.test.ts
Test Scenarios
The test suite includes standardized functions to test common LLM capabilities, which are then run against each configured provider (e.g., Google Gemini, OpenAI GPT, Anthropic Claude).
| Function | Description |
|---|---|
basicTextGeneration | Verifies simple request-response cycles and conversation history. |
handleToolCall | Tests the model's ability to correctly identify and use a provided tool (calculator). |
handleStreaming | Ensures that text generation events (text_start, text_delta, text_end) are correctly emitted. |
handleThinking | Validates the "thinking" or "reasoning" phase for models that support it. |
handleImage | Checks multimodal capabilities by providing an image and asking the model to describe it. |
multiTurn | Simulates a longer conversation involving multiple tool calls and responses to verify state management. |
Sources: packages/ai/test/stream.test.ts:35-379
E2E Test Flow
The following diagram shows the sequence of a typical multi-turn E2E test, which involves tool usage and state management.
Ollama Integration Testing
The E2E suite includes specific tests for Ollama. Before running these tests, the setup process checks if the gpt-oss:20b model is available locally and pulls it if necessary. It then spawns an ollama serve process and waits for the server to become ready before executing the tests. The Ollama server process is terminated after the tests complete.
Sources: packages/ai/test/stream.test.ts:655-715
Agent and Tool Testing
Specialized tests exist to validate the agent's logic and the functionality of its tools.
Agent Loop Testing
The agentLoop function, which orchestrates the interaction between the user, the LLM, and the tools, is tested in packages/ai/test/agent.test.ts. These tests verify complex, multi-turn scenarios involving parallel tool calls and stateful calculations. For example, the calculateTest function prompts the agent to perform two multiplications in parallel, sum the results, and return the final answer.
The test asserts that the correct number of turns and tool calls were made and that the final numerical answer matches the expected sum. It also includes a test for aborting an agent operation mid-flow.
Sources: packages/ai/test/agent.test.ts:5-152
Coding Agent Tool Testing
The coding-agent tools (read, write, edit, bash, etc.) are tested individually in packages/coding-agent/test/tools.test.ts. These tests use a temporary directory created before each test and cleaned up afterward to provide an isolated file system environment.
This diagram shows the test lifecycle for a single tool test.
Key test cases for tools include:
readTool: Reading files, handling non-existent files, and correctly applying truncation logic for line and byte limits.writeTool: Writing new files and creating parent directories as needed.editTool: Successfully replacing text, and failing correctly when the target text is not found or found multiple times.bashTool: Executing simple commands, handling errors, and respecting timeouts.grepTool: Correctly searching files and respecting limits and context line parameters.findTool: Finding hidden files and respecting.gitignorerules.
Sources: packages/coding-agent/test/tools.test.ts:35-231
UI and RPC Testing
The project includes tests for both the Terminal UI (TUI) components and the headless Remote Procedure Call (RPC) mode.
TUI Component Testing
UI components in the packages/tui library are tested to ensure they render correctly. For example, packages/tui/test/markdown.test.ts tests the Markdown component's ability to render various elements like nested lists, tables, and pre-styled text. The tests work by rendering the component to a string and asserting that the output (including ANSI escape codes for color and style) matches the expected structure.
Sources: packages/tui/test/markdown.test.ts:11-265
RPC Mode Testing
The RPC mode of the coding-agent is tested by spawning the agent as a child process and communicating with it via stdin and stdout using JSON-RPC. The tests in packages/coding-agent/test/rpc.test.ts cover:
- Sending a
promptcommand and verifying that the conversation is correctly saved to a session file. - Sending a
compactcommand and verifying that a compaction entry is written to the session log. - Sending a
bashcommand and ensuring thebash_endevent is emitted and the execution result is logged. - Verifying that output from a
bashcommand is correctly included in the LLM's context for subsequent prompts.
The following diagram illustrates the RPC test interaction.
Test Data
For UI testing, especially in packages/web-ui, predefined session objects are used as test data. These objects, found in files like packages/web-ui/src/utils/test-sessions.ts, contain a complete snapshot of a conversation, including the system prompt, model details, and a series of user, assistant, and tool messages. This allows for deterministic testing of how the UI renders complex conversation histories.