Targeted Testing (scip test)
Relevant source files
The following files were used as context for generating this wiki page:
Targeted testing in the SCIP ecosystem is facilitated primarily through the scip test command and associated snapshot testing utilities. This system allows developers to verify the correctness of SCIP indexers by comparing generated indices against expected "snapshots" or by running specific assertions against code navigation data such as occurrences, symbols, and diagnostics. Sources: CHANGELOG.md:41-43, Development.md:48-51
The testing framework is designed to validate language-specific semantics, ensure proper symbol resolution, and check the accuracy of range information. It is integrated into the SCIP CLI and supported by Go bindings that provide helper functions for rendering indices into human-readable text formats suitable for version control. Sources: bindings/go/scip/testutil/format.go:20-25, cmd/scip/main.go:22-29
Testing Architecture and Workflow
The testing system follows a "golden file" or snapshot testing pattern. An indexer generates a SCIP index from source code, which is then converted into a formatted text representation. This representation is compared against a reference file (snapshot). If the generated output matches the reference, the test passes. Sources: bindings/go/scip/testutil/snapshot_testing.go:37-45, Development.md:48-51
Execution Flow
The following diagram illustrates the high-level flow of a targeted test execution:
The flow begins with source files being processed by an indexer (or the internal reprolang for internal testing) to produce a scip.Index. This index is then pretty-printed and compared against stored expected values. Sources: cmd/scip/main_test.go:76-95, bindings/go/scip/testutil/snapshot_testing.go:37-45
Core Components
The test Command
The scip test command is a subcommand of the SCIP CLI used to perform targeted assertions. It supports checking documents within an index for specific occurrences and roles. Sources: cmd/scip/main.go:26, CHANGELOG.md:15
| Flag | Type | Description |
|---|---|---|
--check-documents | Boolean | Enables validation of documents within the SCIP index. |
--comment-syntax | String | Sets the prefix for inline comments (default: //). |
--project-root | String | Overrides the project root path in the index. |
Sources: CHANGELOG.md:15, cmd/scip/main.go:68-84
Snapshot Formatting
The testutil package provides FormatSnapshots, which transforms the binary SCIP data into a text-based format. This format includes source code lines interleaved with comments describing the SCIP entities present at those positions. Sources: bindings/go/scip/testutil/format.go:20-56
Key attributes rendered in snapshots include:
- Occurrences: Marked with caret (
^) symbols and identified asdefinition,reference, orforward_definition. - Enclosing Ranges: Marked with start (
⌄) and end (⌃) indicators. - Symbols: Formatted using a
SymbolFormatter. - Metadata: Includes
kind,documentation,relationships, anddiagnostics.
Sources: bindings/go/scip/testutil/format.go:104-160, CHANGELOG.md:11-12
Implementation Details
Occurrence Assertion Logic
During testing, specific test cases (often written in reprolang) verify that symbols are correctly resolved to their expected roles and locations. Failure occurs if the actual role or symbol string does not match the expected assertion in the test file. Sources: cmd/scip/main_test.go:120-145
// Example of assertion failure handling in cmd/scip/main_test.go
// Actual: 'definition reprolang ... fails-wrong-symbol.repro/hello().'
// Expected: 'definition reprolang ... fails-wrong-role.repro/hello2().'
Sources: cmd/scip/main_test.go:132-135
Snapshot Comparison
The comparison logic uses the myers diff algorithm to provide clear feedback when a test fails. Developers can update snapshots by running tests with the -update-snapshots flag. Sources: bindings/go/scip/testutil/snapshot_testing.go:19-21, bindings/go/scip/testutil/snapshot_testing.go:71-85
The sequence shows how testutil coordinates between the index generation and the file system to validate or update test state. Sources: bindings/go/scip/testutil/snapshot_testing.go:23-88
Internal Testing with Reprolang
For internal development of SCIP itself, a small, verbose language called reprolang is used. It allows the team to design and investigate SCIP semantics (like Go to definition or Find references) without requiring a full-scale industrial indexer. Reprolang consists of declarations, references, and imports. Sources: Development.md:16-21, cmd/scip/tests/reprolang/bindings/go/repro/indexer.go:12-20
Summary
Targeted testing provides the necessary infrastructure to ensure SCIP indexers maintain high fidelity. By combining automated snapshot comparisons with granular occurrence assertions, the project ensures that changes to the protocol or indexers do not regress code navigation features. Sources: DESIGN.md:28-35, Development.md:48-52