SCIP CLI Overview
Relevant source files
The following files were used as context for generating this wiki page:
The SCIP CLI is a language-agnostic command-line tool designed to work with SCIP (Source Code Intelligence Protocol) indexes. It provides a suite of utilities for developers and indexer authors to inspect, validate, analyze, and transform SCIP data, which is used to power code navigation features like Go to definition and Find references.
The CLI acts as a primary tool for debugging and interacting with the SCIP Protobuf schema. It supports various operations including linting for well-formedness, generating statistics, creating snapshots for testing, and converting indexes into queryable formats like SQLite.
Sources: README.md:3-10, cmd/scip/main.go:49-54, Development.md:40-45
Core Command Architecture
The CLI is built using the urfave/cli/v2 package in Go. It follows a subcommand-based structure where the entry point dispatches requests to specialized command handlers.
The diagram above shows the high-level command structure of the SCIP CLI application.
Subcommand Definitions
| Command | Description | Relevant Files |
|---|---|---|
lint | Performs well-formedness checks on a SCIP index to identify correctness and redundancy issues. | cmd/scip/lint.go |
print | Inspects Protobuf output, supporting both colored textual and JSON formats. | Development.md, CHANGELOG.md |
snapshot | Renders a SCIP index into a pretty-printed text format suitable for snapshot testing. | cmd/scip/main.go |
stats | Outputs technical statistics about a SCIP index, such as document sizes and occurrence counts. | cmd/scip/stats.go |
test | Targeted testing of specific occurrences within an index. | CHANGELOG.md |
expt-convert | Experimental command to convert a SCIP index into a compressed SQLite database. | cmd/scip/convert.go |
Sources: cmd/scip/main.go:20-27, Development.md:48-60, CHANGELOG.md:35-45, cmd/scip/stats.go:21-34
Index Validation and Linting
The lint command is primarily intended for indexer authors to verify that the generated SCIP data adheres to protocol expectations and identifies potential bugs.
Validation Logic
The linter populates a internal symbol table and performs multiple passes over the index:
- Pass 1: Registers external symbols and detects duplicates.
- Pass 2: Maps documents and detects path collisions.
- Pass 3: Processes symbol definitions and relationships within documents.
- Pass 4: Validates every occurrence against the established symbol table.
The linter data flow ensures that every reference in an occurrence has a corresponding definition or external symbol entry.
Sources: cmd/scip/lint.go:44-95, cmd/scip/lint.go:138-190
Index Statistics and Analysis
The stats command provides quantitative insights into the composition of a SCIP index. It calculates various metrics and percentiles to help developers understand index density and performance characteristics.
Metrics Calculated
- Document Sizes: Measured in bytes (Sum, Mean, Max, and 50/90/99 percentiles).
- Occurrence Counts: Total and per-document frequency.
- Definition Counts: Frequency of symbols marked with
SymbolRole.Definition. - Lines of Code (LOC): Integrated with
goclocto count actual source lines if the project root is available.
Sources: cmd/scip/stats.go:73-86, cmd/scip/stats.go:106-135
Experimental SQLite Conversion
The CLI includes an experimental feature to transform the transmission-oriented SCIP format into a queryable SQLite database. This is handled by the expt-convert command.
Data Transformation Pipeline
The conversion process involves:
- Canonicalization: Cleaning the document data.
- Chunking: Splitting occurrences into chunks (default ~200 per chunk) to optimize DB access.
- Compression: Blobs are compressed using
zstdbefore storage. - Indexing: Creating SQLite indexes for line ranges, symbol IDs, and roles.
The conversion sequence illustrates how SCIP documents are decomposed into SQLite tables while maintaining compressed occurrence data.
Sources: cmd/scip/convert.go:142-192, cmd/scip/convert.go:344-360, cmd/scip/convert.go:94-110
Global Configuration Flags
Several flags are shared across multiple subcommands to maintain consistency in how the CLI interacts with the filesystem and index files.
| Flag | Default | Description |
|---|---|---|
--from | index.scip | The path to the input SCIP index file. |
--project-root | N/A | Overrides the project root URI found in the SCIP metadata. |
--comment-syntax | // | Customizes the prefix used for inline comments in snapshot files. |
Sources: cmd/scip/main.go:61-84
Summary
The SCIP CLI serves as the central toolkit for managing the lifecycle of SCIP indexes. By providing high-fidelity validation through lint, detailed analysis via stats, and experimental query capabilities through SQLite conversion, it bridges the gap between raw Protobuf transmission data and actionable code intelligence insights. End users can install it by compiling from source using go build ./cmd/scip or downloading pre-compiled binaries.
Sources: README.md:68-75, Development.md:40-45