Consuming SCIP Data
Relevant source files
The following files were used as context for generating this wiki page:
Consuming SCIP (Semantic Code Intelligence Protocol) data involves ingesting language-agnostic indexes to power code navigation features like "Go to definition," "Find references," and "Find implementations." SCIP is designed primarily as a transmission format rather than a storage format, optimizing for producers (indexers) while providing robust, Protobuf-based structures for consumers to process.
Consumers can interact with SCIP data through provided language bindings (Go, Rust, TypeScript, Haskell) or by generating their own code from the scip.proto schema. The protocol emphasizes streaming capabilities, allowing consumers to process large indexes by reading documents one at a time to manage memory efficiency.
Sources: README.md:3-10, DESIGN.md:12-16, scip.proto:19-25
Data Ingestion and Parsing
SCIP indexes are typically stored as binary Protobuf payloads. Due to the potentially large memory footprint of a complete index, it is recommended to consume the Index payload one field at a time in a streaming fashion.
Streaming Architecture
The Index message is structured to support streaming. Consumers should look for the Metadata field at the start of the stream, followed by repeated Document and SymbolInformation fields.
The diagram shows the logical flow of data from a raw binary stream into constituent SCIP components.
Implementation Patterns
Consumers typically implement a "Visitor" pattern or an event-driven parser to handle incoming data fragments. For example, in Go, an IndexVisitor is used to trigger callbacks for metadata, documents, and external symbols during a ParseStreaming operation.
Sources: scip.proto:19-27, bindings/go/scip/parse_test.go:94-110
The SCIP Data Model
Consumers must navigate several key entities defined in the schema to reconstruct the code's semantic graph.
Core Entities
| Entity | Description | Key Fields |
|---|---|---|
| Index | The root container for a workspace index. | metadata, documents, external_symbols |
| Document | Metadata and semantic data for a specific source file. | relative_path, occurrences, symbols |
| Occurrence | Associates a source range with a symbol or syntax kind. | range, symbol, symbol_roles, diagnostics |
| SymbolInformation | Rich metadata about a specific symbol. | symbol, documentation, relationships, kind |
| Relationship | Defines links between symbols (e.g., implementation). | symbol, is_reference, is_implementation |
Sources: scip.proto:29-373
Source Range Interpretation
A critical task for consumers is interpreting the range field in an Occurrence. To save space, ranges are encoded as a repeated int32 array:
- Four elements:
[startLine, startCharacter, endLine, endCharacter] - Three elements:
[startLine, startCharacter, endCharacter](inferred same line)
Consumers must also respect the PositionEncoding specified in the Document to correctly calculate character offsets (UTF-8 bytes, UTF-16 code units, or UTF-32 code units).
Sources: scip.proto:447-466, scip.proto:114-138
Storage and Conversion
While SCIP is a transmission format, consumers often need to convert it into a queryable format like LSIF or a relational database for efficient lookup.
SQLite Conversion Pattern
The project includes an experimental converter that transforms SCIP data into a SQLite database. This process illustrates how consumers can map the hierarchical Protobuf data into a relational schema for querying.
This diagram represents the SQLite schema used when converting SCIP data for local inspection.
Sources: cmd/scip/convert.go:137-185
Chunking for Performance
When converting to a database, consumers may "chunk" occurrences (e.g., 200 occurrences per chunk) to prevent the database size from exploding while keeping blob sizes manageable for retrieval.
Sources: cmd/scip/convert.go:23-25, cmd/scip/convert.go:343-356
Debugging and Validation
Consumers can use the scip CLI to inspect and validate indexes before processing.
scip print: Renders the binary Protobuf into a human-readable, colorful textual format or JSON.scip lint: Performs well-formedness checks, identifying potential issues like non-canonical symbols, duplicate occurrences, or missing symbol information.scip stats: Provides high-level metrics such as document counts, total occurrences, and mean document size.
Sources: Development.md:46-56, cmd/scip/lint.go:19-25, cmd/scip/stats.go:28-35
Conclusion
Consuming SCIP data requires a robust understanding of its Protobuf schema and an implementation strategy that prioritizes streaming for memory efficiency. By leveraging the provided language bindings and the core data model of Documents, Occurrences, and Symbols, consumers can build powerful code intelligence tools that are resilient to indexer bugs and capable of scaling to large codebases.