Reprolang Internal Test Language
Relevant source files
The following files were used as context for generating this wiki page:
Reprolang is a verbose, small internal language specifically designed for testing the SCIP (Sourcegraph Code Intelligence Protocol) CLI and indexer functionality. It allows developers to check existing code navigation behavior, design new behaviors, and investigate the effects of SCIP to LSIF desugaring in a controlled environment. The language consists of simple declarations for definitions, references, imports, and relationships, making the resulting LSIF index much smaller and easier to debug than those generated from real-world languages.
The language is defined using a Tree-sitter grammar and is not intended for use outside of the SCIP repository. It serves as the primary driver for snapshot testing and semantic verification within the cmd/scip test suite.
Sources: Development.md:16-20, Development.md:71-76
Grammar and Syntax
Reprolang syntax is built around explicit statements that map directly to semantic concepts in SCIP. A source file consists of a series of statements, which can be definitions, references, relationship declarations, or comments.
Statement Types
| Statement Type | Syntax Keyword | Description |
|---|---|---|
| Definition | definition | Declares a new symbol. Can include docstrings and roles. |
| Reference | reference | References an existing symbol. Can be marked as a forward_definition. |
| Relationships | relationships | Defines relationships for symbols that lack a local definition. |
| Comment | # | General comments or special # docstring: markers. |
Sources: cmd/scip/tests/reprolang/grammar.js:8-18, cmd/scip/tests/reprolang/grammar.js:52-53
Identifier Resolution
Identifiers in Reprolang can be either global or workspace scoped. Global identifiers are explicitly prefixed with the global keyword followed by a project name and descriptors. Workspace identifiers are standard strings. Local symbols are identified by the local prefix in their value.
Sources: cmd/scip/tests/reprolang/grammar.js:54-65, cmd/scip/tests/reprolang/bindings/go/repro/ast.go:68-70
Architecture and Data Flow
The Reprolang indexer follows a multi-phase process to transform .repro source files into a standard SCIP Index.
Indexing Pipeline
The indexing process is orchestrated by the Index function in the repro package. It uses reproContext to maintain a global scope and package information throughout the phases.
- Parse Phase: Uses
go-tree-sitterto parse source text into a syntax tree. Statements are extracted into internal structures likedefinitionStatementandreferenceStatement. - Definition Resolution: Iterates through dependencies and local files to populate the
reproScopewith symbol names. - Reference Resolution: Updates reference statements with the correct symbol strings by looking them up in the established scopes.
- Emission: Converts the resolved internal structures into
scip.Document,scip.Occurrence, andscip.Symbolobjects.
Sources: cmd/scip/tests/reprolang/bindings/go/repro/indexer.go:13-88, cmd/scip/tests/reprolang/bindings/go/repro/parser.go:12-20
Data Structures
The internal representation of Reprolang components is managed via several Go structs that map to the Tree-sitter grammar nodes.
AST Components
reproSourceFile: Holds the Tree-sitter node, slices of statements, and alocalScopefor the specific file.identifier: Represents a name in the source, containing its raw value, resolved SCIP symbol, and source range.relationships: A container for various SCIP relationship types:implements,references,type_defines, anddefined_by.
Sources: cmd/scip/tests/reprolang/bindings/go/repro/source_file.go:9-16, cmd/scip/tests/reprolang/bindings/go/repro/ast.go:13-33
Symbol Roles and Relationships
Reprolang supports the following relations within definition and relationship statements:
| Relation | Logic Mapping |
|---|---|
implements | Sets is_implementation in SCIP. |
type_defines | Sets is_type_definition in SCIP. |
references | Sets is_reference in SCIP. |
defined_by | Used to associate a relationship with a specific defining symbol. |
Sources: cmd/scip/tests/reprolang/grammar.js:31-38, cmd/scip/tests/reprolang/bindings/go/repro/parser.go:34-45
Testing and Snapshots
Reprolang is the foundation for the SCIP project's snapshot testing. The TestSCIPSnapshots function in main_test.go uses the Reprolang indexer to generate SCIP indexes for various test scenarios (found in cmd/scip/tests/).
Snapshot Execution Flow
Test files are typically named passes*.repro or fails*.repro. The test runner validates that the actual SCIP output (roles, ranges, diagnostics) matches the expected assertions defined in the test files.
Sources: cmd/scip/main_test.go:73-108, cmd/scip/main_test.go:133-176
Summary
Reprolang serves as a critical internal tool for the SCIP project, providing a human-readable and highly controllable way to define complex code intelligence scenarios. By abstracting away the complexities of real-world languages, it allows for precise testing of the SCIP protocol's core features, including cross-package symbol resolution, relationship mapping, and diagnostic reporting.
Sources: Development.md:71-76, cmd/scip/tests/reprolang/bindings/go/repro/indexer.go:15-18