Analyzing Index Stats (scip stats)

The scip stats command is a diagnostic utility within the SCIP CLI designed to provide quantitative insights into a SCIP index file. It calculates various metrics, including document counts, occurrence density, symbol definitions, and lines of code, helping developers understand the scale and composition of their indexed codebase.

This tool is particularly useful for indexer authors to verify the density of their generated indexes and for performance benchmarking. It operates by parsing the SCIP Protobuf index and traversing its document and occurrence structures to aggregate statistical distributions.

Sources: cmd/scip/stats.go:27-33, Development.md:73-77

Architecture and Data Flow

The stats command follows a linear execution path: reading a SCIP index, processing its contents to aggregate counts and distributions, and outputting the result as a formatted JSON object.

The diagram shows the high-level process from index ingestion to the final JSON reporting.
Sources: cmd/scip/stats.go:48-70

Key Components

• statsMain: The entry point for the command logic. It handles file reading, triggers the counting logic, and manages JSON serialization of the output. Sources: cmd/scip/stats.go:48-67
• countStatistics: The core processing function. It iterates through all scip.Document objects and their scip.Occurrence lists to populate statistical buffers. Sources: cmd/scip/stats.go:107-142
• countLinesOfCode: A helper that uses the gocloc library to analyze the local project root and calculate physical lines of code, correlating the SCIP index with the actual source files. Sources: cmd/scip/stats.go:150-181
• NewStats: A constructor that transforms raw slices of numeric data into a structured Stats object containing mean, standard deviation, and various percentiles. Sources: cmd/scip/stats.go:81-96

Data Models

Configuration Flags

The command supports specific flags to control its behavior, primarily related to input sources and path resolution.

Sources: cmd/scip/stats.go:37-41, cmd/scip/main.go:88-103

Statistics Structure

The output is encapsulated in the indexStatistics and Stats types, which group related metrics.

The relationship between aggregated totals and specific statistical distributions for sizes and counts.
Sources: cmd/scip/stats.go:72-105

Statistical Calculations

Percentage and Distribution

The tool calculates several percentiles (50th, 90th, 95th, 99th, and 99.9th) using the montanaflynn/stats library. These metrics provide a more nuanced view than simple averages, highlighting outliers in document size or occurrence density.

• Document Sizes: Measured in bytes after Protobuf marshaling.
• Occurrence Counts: Number of occurrences per document.
• Definition Counts: Number of occurrences marked with the Definition role per document.

Sources: cmd/scip/stats.go:81-96, cmd/scip/stats.go:124-140

Role-Based Filtering

During the traversal of occurrences, the tool identifies definitions by matching the SymbolRole.

// From cmd/scip/stats.go:133-138
for _, occurrence := range document.Occurrences {
    if scip.SymbolRole_Definition.Matches(occurrence) {
        stats.Definitions += 1
        definitionCounts[len(definitionCounts)-1] += 1
    }
}

Sources: cmd/scip/stats.go:133-138, scip.proto:479-482

Project Root Resolution

A critical part of the analysis (especially for LinesOfCode) is locating the source files. The tool attempts to resolve the project root using a specific priority:

1. The --project-root CLI flag (if provided).
2. The Metadata.project_root URI defined within the SCIP index.
3. The current working directory (as a fallback if the metadata path does not exist).

Sources: cmd/scip/stats.go:151-166, bindings/go/scip/testutil/format.go:27-40

Conclusion

Analyzing index statistics via scip stats provides essential visibility into the density and health of a SCIP index. By combining internal metadata analysis with external line-of-code counts, it allows developers to validate indexer output against the physical reality of the source code.