Writing a SCIP Indexer

Relevant source files

The following files were used as context for generating this wiki page:

• DESIGN.md
• README.md
• scip.proto
• cmd/scip/tests/reprolang/bindings/go/repro/indexer.go
• cmd/scip/lint.go
• Development.md

SCIP (Source Code Intelligence Protocol) is a language-agnostic protocol designed to index source code to power advanced code navigation features such as "Go to definition," "Find references," and "Find implementations." Unlike previous protocols like LSIF, SCIP is optimized for the ease of writing indexers (producers), supporting parallelism, incremental indexing, and robustness against indexer-specific bugs.

The primary goal of a SCIP indexer is to translate the semantic structure of a codebase into a standardized Protobuf format. This index acts as a transmission format between the language-specific analysis tools (producers) and code intelligence consumers, such as Sourcegraph.

Sources: README.md:3-12, DESIGN.md:14-38

Indexer Architecture and Data Model

A SCIP indexer must output data conforming to the scip.proto schema. The high-level structure of a SCIP index is organized around a root Index message containing metadata and a collection of documents.

The Index Hierarchy

The following diagram illustrates the relationship between the core SCIP components:

The SCIP data model uses a document-centric approach to support streaming and parallelism.
Sources: scip.proto:26-135, DESIGN.md:99-110

Core Components Summary

Component	Description	Key Fields
Metadata	Global information about the indexing run.	project_root, tool_info
Document	Represents a single source file.	relative_path, occurrences, symbols
Occurrence	Links a source range to a symbol.	range, symbol, symbol_roles
SymbolInformation	Semantic metadata for a symbol.	documentation, relationships, kind

Sources: scip.proto:46-724

The Indexing Process

Writing a SCIP indexer typically involves four distinct phases, ranging from initial parsing to final Protobuf serialization.

Indexing Lifecycle

This flowchart represents the logical flow implemented in standard reference indexers.
Sources: cmd/scip/tests/reprolang/bindings/go/repro/indexer.go:27-89

Implementation Phases

1. Parse Sources: Read source files and generate an Abstract Syntax Tree (AST) or similar representation using tools like Tree-sitter.
2. Resolve Definitions: Traverse the AST to identify all declared entities (classes, methods, variables). Assign a unique SCIP symbol to each definition and store it in a SymbolInformation object.
3. Resolve References: Identify usages of symbols. For every reference, determine which definition it points to (even if defined in another file or package).
4. Emit SCIP: Construct the Index message, populate the Document and Occurrence fields, and serialize the result to a file (usually index.scip).

Sources: cmd/scip/tests/reprolang/bindings/go/repro/indexer.go:48-89

Symbol Naming and Ranges

Standardized Symbol Strings

SCIP uses a specific string format for IDs to ensure uniqueness across repositories and packages. Symbols are either "local" or "global." Local symbols (prefixed with local ) are restricted to a single document, while global symbols follow a structured URI-like grammar.

<symbol>    ::= <scheme> ' ' <package> ' ' (<descriptor>)+ | 'local ' <local-id>
<package>   ::= <manager> ' ' <package-name> ' ' <version>

Sources: scip.proto:168-195, DESIGN.md:126-140

Source Ranges

Ranges are encoded as a repeated int32 to optimize payload size. They are 0-based and follow a half-open interval [start, end).

• Four elements: [startLine, startCharacter, endLine, endCharacter]
• Three elements: [startLine, startCharacter, endCharacter] (implies single-line)

Sources: scip.proto:635-655

Validation and Tooling

Indexers should be verified using the scip CLI to ensure protocol compliance and data quality.

Useful Commands for Indexer Developers

Command	Purpose
scip lint	Performs well-formedness checks, identifying non-canonical symbols or missing definitions.
scip print	Renders the binary Protobuf index into a human-readable format (text or JSON).
scip stats	Provides statistics on index size, occurrence counts, and lines of code covered.
scip test	Runs targeted assertions against specific occurrences in the index.

Sources: Development.md:52-65, cmd/scip/lint.go:17-30, cmd/scip/stats.go:27-35

Verification Logic

The scip lint tool checks for several critical errors:

• Duplicate Symbols: Ensuring SymbolInformation is not repeated for the same ID within a document.
• Missing Symbols: Ensuring every Occurrence with a symbol ID has a corresponding definition or external symbol entry.
• Canonicalization: Verifying that symbol strings are formatted exactly according to the SCIP grammar.

Sources: cmd/scip/lint.go:43-100

Conclusion

Writing a SCIP indexer requires a deep understanding of the target language's semantics and the mapping of those semantics to the SCIP Protobuf schema. By leveraging the standardized symbol grammar and the document-centric architecture, indexers can provide high-fidelity code navigation data that scales to large monorepos and complex package ecosystems. The included scip CLI tools serve as the primary environment for debugging and validating new indexer implementations.