Streaming Index Parsing

Relevant source files

The following files were used as context for generating this wiki page:

• scip.proto
• DESIGN.md
• CHANGELOG.md
• bindings/go/scip/parse_test.go
• Development.md

Streaming Index Parsing is a core capability of the SCIP (Sourcegraph Code Intelligence Protocol) Go bindings that allows for processing large index files with optimized memory consumption. Rather than loading an entire Index object—which can have a large memory footprint—into memory, the streaming API enables consumers to visit individual components like metadata, documents, and external symbols as they are read from the input stream.

This approach is specifically supported by the Protobuf Tag-Length-Value (TLV) format, which allows for merging by concatenation and sequential reading of repeated fields.

Sources: scip.proto:21-26, DESIGN.md:104-107, CHANGELOG.md:129-132

Architecture and Design

The streaming architecture is designed around the IndexVisitor pattern. Because a SCIP Index message contains repeated fields for documents and external_symbols, these can be emitted or consumed one at a time. To ensure validity during a stream, the metadata field must appear first in the stream and occur only once.

Core Components

Component	Description
IndexVisitor	A struct containing callback functions for different SCIP entities (Metadata, Document, ExternalSymbol).
ParseStreaming	The primary entry point that takes an io.Reader and executes visitor callbacks.
Metadata	Required at the start of the stream; contains versioning and project root information.
Document	Represents a single source file, processed as a discrete unit in the stream.

Sources: scip.proto:17-30, CHANGELOG.md:129-132

Data Flow

The following diagram illustrates how the IndexVisitor processes a SCIP byte stream:

The streaming parser iterates through the Protobuf payload, identifying fields by their tags and invoking the corresponding visitor method.

Sources: scip.proto:17-30, bindings/go/scip/parse_test.go:101-120

Implementation Details

The Go implementation uses an IndexVisitor which provides functional hooks for different message types. This design allows for selective parsing; for example, a consumer can choose to only visit documents while ignoring external symbols to save processing time.

Visitor Pattern Logic

In the Go bindings, the IndexVisitor follows these rules:

1. Context Support: Callback functions accept a context.Context for cancellation and timeout management.
2. Error Handling: If a visitor function returns an error, the parser short-circuits and returns that error to the caller.
3. Memory Efficiency: By processing one Document at a time, indexers and consumers can clear memory used for previous iterations instead of holding the entire codebase in RAM.

Sources: CHANGELOG.md:17-19, DESIGN.md:104-107, bindings/go/scip/parse_test.go:101-120

Code Example: Streaming Parser Usage

parsedIndex := Index{}

indexVisitor := IndexVisitor{
    VisitMetadata: func(ctx context.Context, metadata *Metadata) error {
        parsedIndex.Metadata = metadata
        return nil
    },
    VisitDocument: func(ctx context.Context, document *Document) error {
        // Process document immediately or append to a collection
        parsedIndex.Documents = append(parsedIndex.Documents, document)
        return nil
    },
}

err := indexVisitor.ParseStreaming(context.Background(), reader)

Sources: bindings/go/scip/parse_test.go:104-118

Performance and Benchmarking

Streaming is critical for scaling SCIP to large monorepos. The protocol's design avoids integer IDs and global symbol tables where possible to support this. Benchmark results should be shared specifically when making changes to symbol parsing logic within the streaming flow to ensure no regressions in processing speed.

Standard benchmarks can be run using the speedtest utility:

go run ./bindings/go/scip/speedtest

Sources: DESIGN.md:112-114, Development.md:73-77

Summary

Streaming Index Parsing provides the technical foundation for SCIP to handle massive codebases by treating the index as a sequence of discrete units rather than a monolithic object. By enforcing that Metadata appears first and utilizing the IndexVisitor pattern, SCIP ensures that producers can append data to files incrementally and consumers can process that data with constant or near-constant memory overhead per document.

Sources: scip.proto:21-25, DESIGN.md:104-107