Core Entities: Documents, Occurrences, Ranges
Relevant source files
The following files were used as context for generating this wiki page:
The SCIP Code Intelligence Protocol utilizes a document-centric architecture to represent semantic information about source code. The core entities—Documents, Occurrences, and Ranges—work together to map source code positions to symbols, syntax highlighting, and diagnostics. This structure avoids the complexities of a wholesale graph-based representation, opting instead for a format that naturally supports parallelism and streaming during both indexing and consumption.
The relationship between these entities is hierarchical: an Index contains multiple Documents, each of which contains a collection of Occurrences defined by specific Ranges. This design facilitates file-level incrementality and limits the "blast radius" of indexer bugs by localizing data within individual document units.
Sources: DESIGN.md:91-115, scip.proto:21-30
Document Structure and Metadata
A Document is the primary unit of indexing in SCIP, representing a single source file. It contains the relative path to the file, the programming language identifier, and the semantic data (Occurrences and SymbolInformation) associated with that file.
The diagram above shows the relationship between the top-level SCIP Index and the individual Document entities.
Sources: scip.proto:75-108
Key Document Attributes
| Field | Type | Description |
|---|---|---|
relative_path | string | Unique path relative to project_root. Must be canonical and use forward slashes. |
language | string | Programming language ID (e.g., "go", "typescript"). |
occurrences | repeated Occurrence | List of all semantic markers within the file. |
symbols | repeated SymbolInformation | Metadata for symbols defined specifically within this document. |
position_encoding | PositionEncoding | Specifies how character offsets are interpreted (UTF-8, UTF-16, or UTF-32). |
Sources: scip.proto:75-130
Occurrences and Ranges
An Occurrence associates a specific span of source code (the Range) with semantic information, such as a symbol reference, a definition, or a diagnostic. To optimize payload size, SCIP uses a compressed repeated int32 format for ranges instead of nested objects.
Range Encoding Logic
Ranges are represented as half-open intervals [start, end). They can be encoded in two formats:
- Four elements:
[startLine, startCharacter, endLine, endCharacter] - Three elements:
[startLine, startCharacter, endCharacter](where the end line is the same as the start line).
Line and character numbers are 0-based. The interpretation of the "character" value depends on the position_encoding defined in the parent Document.
Sources: scip.proto:483-518, bindings/go/scip/canonicalize.go:41-45
Occurrence Components
| Component | Description |
|---|---|
range | The physical location in the file (3 or 4 integers). |
symbol | The unique string identifier for the entity at this location. |
symbol_roles | A bitset defining if this is a definition, reference, import, etc. |
syntax_kind | Highlighting information (e.g., Keyword, Comment, Identifier). |
enclosing_range | The range of the nearest non-trivial enclosing AST node (useful for call hierarchies). |
Sources: scip.proto:520-565
Enclosing Ranges
The enclosing_range field provides context for an occurrence by identifying the bounds of the parent AST node. This is critical for features like "expand selection" or determining the scope of a function body for call hierarchies. For a definition, the enclosing range typically includes the entire definition block, including attached documentation or decorators.
This diagram illustrates how an Enclosing Range encompasses both the specific Occurrence range and its surrounding context.
Sources: scip.proto:566-608, bindings/go/scip/testutil/format.go:167-175
Data Flattening and Canonicalization
To ensure data integrity, SCIP provides logic for "flattening" and "canonicalizing" documents. This process merges duplicate entries and ensures a deterministic order for occurrences and symbols.
- Flattening: Merges documents with the same relative path and occurrences with equivalent ranges. It combines roles, diagnostics, and documentation into a single representative object.
- Canonicalization: Removes illegal ranges (not length 3 or 4), sorts occurrences by position, and sorts symbols by name.
// bindings/go/scip/flatten.go:61-75
func FlattenOccurrences(occurrences []*Occurrence) []*Occurrence {
if len(occurrences) == 0 {
return occurrences
}
_ = SortOccurrences(occurrences)
flattened := make([]*Occurrence, 0, len(occurrences))
flattened = append(flattened, occurrences[0])
// ... logic to merge overlapping ranges and combine roles/diagnostics
}
Sources: bindings/go/scip/flatten.go:1-15, bindings/go/scip/canonicalize.go:1-15
Database Representation
When converted to a transmission-optimized format like SQLite, documents and occurrences are chunked to maintain performance. Occurrences are often stored as compressed blobs within chunks to prevent database size explosion while allowing range-based queries.
| Table | Key Columns | Relation |
|---|---|---|
documents | id, relative_path, text | Base file info |
chunks | document_id, start_line, end_line, occurrences (BLOB) | Grouped occurrences |
defn_enclosing_ranges | document_id, symbol_id, start_line, end_line | Enclosing range index |
Sources: cmd/scip/convert.go:142-175, cmd/scip/convert.go:487-512
Summary
The core entities of SCIP—Documents, Occurrences, and Ranges—provide a robust and scalable foundation for code intelligence. By localizing semantic data within Documents and using a compact, integer-based representation for Ranges, SCIP achieves its primary goals of indexer performance and ease of debugging. The inclusion of metadata like position_encoding and enclosing_range ensures that the protocol remains language-agnostic while supporting advanced IDE features across various ecosystems.