Benchmarking with Speedtest
Relevant source files
The following files were used as context for generating this wiki page:
Benchmarking within the SCIP project is primarily conducted using a custom tool located in the speedtest package. This tool is designed to provide fine-grained control over performance measurements, specifically targeting the efficiency of symbol parsing and validation logic. Unlike standard Go benchmarking, Speedtest allows for specific iteration counts and comparative analysis between different versions of the parsing implementation.
Sources: bindings/go/scip/speedtest/speedtest_main.go:23-25, Development.md:88-90
Purpose and Scope
The primary objective of the Speedtest utility is to measure the latency and memory allocation overhead of SCIP symbol parsing. This is critical because symbol parsing is a high-frequency operation during index processing. The tool compares the legacy "V1" parser with the "V2" parser and validation logic to ensure that updates do not introduce performance regressions.
Sources: bindings/go/scip/speedtest/speedtest_main.go:48-75, Development.md:98-100
Architecture and Components
The benchmarking system is built around a standalone Go executable that operates independently of the standard go test -bench framework. This architecture ensures that the number of iterations can be strictly controlled to produce deterministic results for comparison.
System Flow
The following diagram illustrates the high-level execution flow of the Speedtest utility:
The Speedtest utility iterates through sample SCIP indexes, extracts all occurrences, and executes three distinct benchmarking functions against the symbol strings.
Sources: bindings/go/scip/speedtest/speedtest_main.go:27-31, bindings/go/scip/speedtest/speedtest_main.go:33-80
Core Data Structures
The internal logic uses several structures to track performance metrics:
| Structure | Purpose | Description |
|---|---|---|
simpleBenchmark | Configuration | Defines the current iteration count (N) and the maximum iterations (MaxN). |
benchmarkResult | Aggregator | Holds the name of the benchmark and slices of timing and allocation data. |
timing | Time Metric | Records total duration for a specific number of iterations. |
allocBytes | Memory Metric | Records total heap allocations (in bytes) for a specific number of iterations. |
Sources: bindings/go/scip/speedtest/speedtest_main.go:157-172
Benchmarking Logic
The utility performs iterative testing across various scales, typically ranging from 1,000 to 100,000 iterations. It measures two primary categories: speed and heap allocations.
Execution Phases
- Warm-up and Execution: The parser is executed for
Noperations. - Timing: Uses
time.Now()andtime.Sub()to calculate total duration and derivestime/op. - Memory Measurement: Uses
runtime.ReadMemStatsbefore and after the test run. To ensure accuracy, the Garbage Collector (GC) is temporarily disabled during allocation testing usingdebug.SetGCPercent(-1).
Sources: bindings/go/scip/speedtest/speedtest_main.go:174-192
Comparison Targets
The tool specifically evaluates three implementation paths:
- Symbol Parse (v1): The legacy parsing logic (
internal.ParsePartialSymbolV1ToBeDeleted). - Symbol Parse (v2): The current optimized parsing logic using
beaut.UTF8Stringandscip.ParseSymbolUTF8With. - Symbol Validate (v2): A lightweight validation check (
scip.ValidateSymbolUTF8) that does not perform full AST construction.
The sequence diagram shows how the tool isolates memory allocation measurements by controlling the Go runtime's garbage collection behavior.
Sources: bindings/go/scip/speedtest/speedtest_main.go:183-191
Performance Requirements
SCIP design prioritizes efficiency to handle large monorepos. Key performance considerations reflected in the benchmarks include:
- Protobuf Efficiency: Using Protobuf as a binary format to reduce I/O and unmarshalling overhead compared to LSIF's JSON-based graph.
- Allocation Minimization: Optimized parsers aim for zero or near-zero allocations when memory is pre-allocated.
- Parallelism: The index structure (Documents and Occurrences) is designed to be easily processed in parallel, though Speedtest focuses on single-threaded parsing speed.
Sources: DESIGN.md:96-103, bindings/go/scip/memtest/low_mem_test.go:68-80
Running Benchmarks
Developers should run benchmarks when modifying symbol parsing or validation logic.
Usage Instructions
- Place sample SCIP indexes in
dev/sample_indexes. - Execute the benchmark runner:
go run ./bindings/go/scip/speedtest - Review the tabulated output, which includes ratios comparing V2 performance against the V1 baseline.
Sources: Development.md:88-96, bindings/go/scip/speedtest/speedtest_main.go:85-115
Output Format
The results are printed in a table formatted with go-pretty. It displays:
- Speed: Nanoseconds per operation (
ns/op). - Allocs: Bytes per operation (
B/op). - Ratio: Multiplier comparison (e.g.,
0.50xfor 2x faster).
Sources: bindings/go/scip/speedtest/speedtest_main.go:85-115
Summary
Benchmarking with Speedtest is a specialized process within the SCIP project that bypasses standard testing tools to provide deep insights into symbol parsing efficiency. By measuring both execution time and memory allocations across multiple scales, the tool ensures that the SCIP protocol remains a high-performance transmission format for large-scale source code indexing.
Sources: bindings/go/scip/speedtest/speedtest_main.go:23-40, DESIGN.md:20-25