Symbol Parsing and Formatting
Relevant source files
The following files were used as context for generating this wiki page:
SCIP symbols serve as unique identifiers for semantic entities such as classes, methods, or local variables across different programming languages and repositories. The protocol defines a standardized string representation for these symbols, allowing tools to exchange and correlate code intelligence data.
Symbol Parsing and Formatting refers to the bidirectional process of converting these string identifiers into structured data models (and vice versa) for programmatic manipulation. This system ensures that symbols remain canonical, searchable, and globally unique while supporting language-specific nuances through a rich descriptor grammar.
Symbol Architecture and Structure
The SCIP data model distinguishes between two primary types of symbols: Global Symbols and Local Symbols.
Global Symbols
Global symbols are designed to be unique across a package ecosystem. They consist of a scheme (the indexer ID), a package (manager, name, and version), and a sequence of descriptors that form the fully qualified name.
Sources: scip.proto:155-177, bindings/go/scip/symbol_test.go:12-45
Local Symbols
Local symbols are restricted to the scope of a single document. They always begin with the prefix local and are followed by a simple identifier. These are used for entities that cannot be accessed from outside the current file.
Sources: scip.proto:178-180, bindings/go/scip/symbol.go:19-21
Symbol Components Table
| Component | Description | Format/Example |
|---|---|---|
| Scheme | Identifies the tool/indexer. | lsif-java, rust-analyzer |
| Package | Coordinates for the distribution unit. | maven package 1.0.0 |
| Descriptors | Sequence of namespaces, types, or terms. | java/io/File#Entry. |
| Disambiguator | Used for methods with same names. | (+1), (455f465bc33b4cdf) |
Sources: scip.proto:181-205, bindings/go/scip/symbol_test.go:46-60
Parsing Logic
The parsing process converts a raw symbol string into a structured Symbol message. This involves state-machine-like logic to handle escaped identifiers and specific suffix characters.
Parsing Flow
The parser identifies the symbol type by checking for the local prefix. For global symbols, it splits the string by spaces to extract the scheme and package information before iterating through descriptors.
This diagram illustrates the decision tree for identifying local vs. global symbols and the sequential extraction of global components.
Sources: bindings/rust/src/symbol.rs:163-195, bindings/go/scip/symbol.go:49-62
Escaping Mechanisms
Parsing must handle two types of escaping:
- Space Escaping: Used in the scheme and package components. Spaces are escaped by repeating them (double space).
- Backtick Escaping: Used in descriptors for names containing special characters. Backticks within names are escaped with a double backtick.
Sources: scip.proto:166-175, bindings/go/scip/symbol_formatter.go:142-156
Formatting and Canonicalization
Formatting is the process of generating a string from a Symbol object. SCIP enforces canonicalization, meaning there is exactly one valid string representation for any given structured symbol.
Descriptor Suffixes
The type of an entity is determined by its suffix in the string representation:
| Suffix | Descriptor Type |
|---|---|
/ | Namespace |
# | Type (Class, Interface, etc.) |
. | Term (Constant, Field) |
! | Macro |
: | Meta |
() | Method |
[] | Type Parameter |
() | Parameter |
Sources: scip.proto:191-205, bindings/go/scip/symbol_formatter.go:94-118
Formatting Process
The formatter iterates through the structured components and applies the appropriate suffixes and escaping. Go and Rust bindings provide a SymbolFormatter that can be configured to include or exclude specific parts of the symbol (e.g., for cleaner snapshot tests).
This sequence shows how the formatter reconstructs the canonical string by applying grammar rules to the structured message.
Sources: bindings/go/scip/symbol_formatter.go:62-92, bindings/rust/src/symbol.rs:80-146
Validation and Linting
The scip lint command uses the parsing and formatting logic to verify that an index is well-formed. A key check is ensuring that every symbol in the index is canonical. If a symbol, once parsed and re-formatted, does not match its original string, it is flagged as an error.
Common Linting Checks
- Non-canonical Symbols: Symbols that don't match the standard string representation.
- Local vs. External: Ensuring that
SymbolInformationfor a symbol isn't defined both as an external symbol and within a local document. - Relationship Integrity: Ensuring relationships point to symbols that actually exist in the index or are correctly marked as external.
Sources: cmd/scip/lint.go:195-215, cmd/scip/lint.go:240-250
Conclusion
Symbol Parsing and Formatting are foundational to SCIP's ability to provide high-fidelity code navigation. By strictly defining a grammar in the Protobuf schema and implementing consistent logic across language bindings (Go, Rust, TypeScript), SCIP ensures that semantic identities are preserved across toolchains and language boundaries. Accurate implementation of these components is critical for indexers to produce valid, interoperable data.