Tool Snapshot Testing
Relevant source files
The following files were used as context for generating this wiki page:
Tool Snapshot Testing is a mechanism within the GitHub MCP Server project designed to ensure that the JSON schemas defined for various tools remain stable and do not change unexpectedly. Since the Model Context Protocol (MCP) relies on precise tool definitions for LLM interaction, any accidental modification to a tool's name, description, or parameter schema could break integrations or lead to unpredictable AI behavior.
The system works by capturing a "snapshot" of a tool's serialized JSON structure and storing it in a dedicated directory. During unit testing, the current tool definition is marshaled and compared against the stored snapshot. If a difference is detected, the test fails, providing a diff of the changes and requiring the developer to either revert the change or intentionally update the snapshot.
Sources: internal/toolsnaps/toolsnaps.go:1-10, CONTRIBUTING.md:55
Core Architecture and Implementation
The snapshot testing logic is encapsulated in the internal/toolsnaps package. The primary entry point is the Test function, which accepts a tool name and the tool object itself (typically an mcp.Tool).
The Testing Workflow
When toolsnaps.Test(toolName, tool) is invoked, the following logic is executed:
- Serialization: The tool object is marshaled into an indented JSON string.
- Path Resolution: The system looks for a snapshot file at
__toolsnaps__/<toolName>.snap. - Mode Detection:
- If the environment variable
UPDATE_TOOLSNAPSis set to"true", the system overwrites the existing snapshot with the current definition. - If the snapshot does not exist and the code is not running in a Continuous Integration (CI) environment, a new snapshot is created.
- If running in CI (
GITHUB_ACTIONS="true") and the snapshot is missing, the test fails immediately.
- If the environment variable
- Comparison: If a snapshot exists and update mode is off, the system performs a semantic JSON comparison using the
jd(JSON Diff) library.
Sources: internal/toolsnaps/toolsnaps.go:20-55, internal/toolsnaps/toolsnaps_test.go:25-45
Logical Flow Diagram
The following diagram illustrates the decision tree used by the snapshot testing utility:
The diagram shows how environment variables and file presence determine whether a test passes, fails, or updates the baseline metadata.
Sources: internal/toolsnaps/toolsnaps.go:20-60
Integration in Tool Definitions
Snapshot testing is integrated into the unit tests of every major GitHub toolset, including Repositories, Pull Requests, and Issues. This ensures that every tool exposed by the server has a locked schema.
Example Integration Patterns
In the test suites, the toolsnaps.Test function is typically called at the beginning of a tool's specific test function to verify the definition before testing functionality.
| Toolset | Example Tool Tested | Relevant Test File |
|---|---|---|
| Repositories | get_file_contents | pkg/github/repositories_test.go |
| Pull Requests | get_pull_request | pkg/github/pullrequests_test.go |
| Issues | get_issue | pkg/github/issues_test.go |
Sources: pkg/github/repositories_test.go:33, pkg/github/pullrequests_test.go:25, pkg/github/issues_test.go:25
Configuration and Environment Variables
The behavior of the snapshot system is controlled primarily through environment variables. This allows for different behaviors on local developer machines versus automated build environments.
System Configuration Options
| Variable | Type | Purpose |
|---|---|---|
UPDATE_TOOLSNAPS | Boolean | When true, forces the system to update the .snap files rather than comparing them. |
GITHUB_ACTIONS | Boolean | Used to detect CI environments. Prevents accidental creation of new snapshots that haven't been reviewed. |
Sources: internal/toolsnaps/toolsnaps.go:28-40, CONTRIBUTING.md:55
Manual Update Command
To intentionally update snapshots after making schema changes, developers use the following command:
UPDATE_TOOLSNAPS=true go test ./...
Sources: CONTRIBUTING.md:55
Semantic Comparison Logic
The system utilizes the jd library for JSON comparison. A critical feature of this implementation is the use of jd.SET during diffing. This allows arrays within the JSON schema (such as the list of required parameters) to be compared without sensitivity to the order of elements. This prevents false positives where the functionality remains identical but the internal ordering of a slice has shifted.
// jd.Set allows arrays to be compared without order sensitivity
diff := toolNode.Diff(snapNode, jd.SET).Render()
if diff != "" {
return fmt.Errorf("tool schema for %s has changed unexpectedly:\n%s", toolName, diff)
}
Sources: internal/toolsnaps/toolsnaps.go:52-60
Implementation Details: Snapshot Storage
Snapshots are stored in a directory named __toolsnaps__ relative to the package being tested. The utility handles directory creation automatically using 0700 permissions for directories and 0600 for the .snap files.
Snapshot File Lifecycle Sequence
The sequence demonstrates the interaction between the test runner and the file system to validate or initialize snapshot data.
Sources: internal/toolsnaps/toolsnaps.go:20-75, internal/toolsnaps/toolsnaps_test.go:15-30
Conclusion
Tool Snapshot Testing serves as a regression suite for the public-facing API of the GitHub MCP Server. By enforcing that all tool schemas are committed to version control and validated during every test run, the project maintains a high level of interface stability. This prevents breaking changes from reaching users and ensures that AI models interacting with the server receive consistent tool definitions.