Testing Infrastructure
Relevant source files
The following files were used as context for generating this wiki page:
The testing infrastructure in SCIP is designed to ensure the correctness of indexers, the CLI, and the underlying protocol. It relies heavily on a combination of snapshot testing, a specialized testing language called "reprolang," and automated linting to validate SCIP index files. The primary goal is to provide a robust way to verify that source code is accurately transformed into the SCIP format, supporting navigation features like "Go to Definition" and "Find References."
This infrastructure allows developers to design new code navigation behaviors and investigate the desugaring process of SCIP. It is particularly focused on monorepo scalability, parallelism, and limiting the impact of indexer bugs through strict validation and human-readable snapshot formats.
Sources: Development.md:43-58, DESIGN.md:25-40
Snapshot Testing Framework
Snapshot testing is the core mechanism for verifying the output of SCIP producers. The framework compares the actual SCIP index generated from source files against "golden" snapshots stored in the repository.
Snapshot Generation and Comparison
The framework uses SnapshotTest to iterate through input directories, generate SCIP indexes using an indexFunction, and compare the resulting snapshots. If the -update-snapshots flag is provided, the framework overwrites existing snapshots with new results.
Pretty-Printed Format
To make snapshots human-readable, SCIP indexes are rendered into a text format where occurrences (definitions, references, etc.) are displayed as comments directly below the relevant code lines.
| Component | Description |
|---|---|
⌄ enclosing_range_start | Indicates the start of a logical scope (e.g., class or function). |
^ reference | Pointed markers identifying a symbol reference at a specific character range. |
^ definition | Pointed markers identifying a symbol definition. |
diagnostic | Displays severity (Error, Warning, etc.) and associated messages. |
relationship | Lists connections like implementations or type definitions. |
Sources: bindings/go/scip/testutil/format.go:78-154
Reprolang: The Test Language
Reprolang is a small, verbose language specifically created to test the SCIP CLI and design new navigation behaviors. It includes declarations, references, and imports defined via a tree-sitter grammar.
Integration in CLI Tests
Reprolang is used to simulate complex indexing scenarios without the overhead of a full production language. Tests in cmd/scip/main_test.go use reprolang to verify that different SCIP roles (definitions vs. references) and diagnostics are correctly indexed.
Automated Linting
The lint command performing well-formedness checks on SCIP indexes. It is primarily used by indexer authors to ensure their output adheres to the protocol specifications.
Validation Rules
The linter checks for several categories of errors and warnings:
- Canonicalization: Verifies symbols follow the standard SCIP formatting.
- Completeness: Ensures symbols in relationships or occurrences actually exist in the
SymbolInformationtable. - Consistency: Flags occurrences marked as both
DefinitionandForwardDefinition. - Redundancy: Warns about duplicate
SymbolInformationor multiple identical relationships.
| Error Type | Logic | Source |
|---|---|---|
bothLocalAndExternalSymbolError | Symbol present in both external list and a document. | cmd/scip/lint.go:174 |
missingRelationshipFlagError | Relationship exists but has no type (ref, def, etc.). | cmd/scip/lint.go:203 |
missingSymbolForOccurrenceError | Occurrence found for a symbol never defined. | cmd/scip/lint.go:264 |
nonCanonicalSymbolError | Symbol string does not match VerboseSymbolFormatter output. | cmd/scip/lint.go:287 |
Sources: cmd/scip/lint.go:37-120, cmd/scip/lint_test.go:111-155
Regression Testing for CLI
The testing infrastructure includes specific checks to ensure the CLI documentation remains in sync with the actual implementation. TestCLIReferenceInSync runs every subcommand with the --help flag and validates that the output is contained within docs/CLI.md.
Sources: cmd/scip/main_test.go:34-66
Summary
The SCIP testing infrastructure provides a multi-layered validation strategy. By combining high-level snapshot tests that simulate real-world code navigation with low-level protocol linting and a dedicated test language (reprolang), the project ensures that indexers produce reliable, efficient, and protocol-compliant data. This enables rapid development and debugging of code intelligence features while maintaining monorepo-scale performance.