Semantic Code Retrieval Tools
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Semantic Code Retrieval Tools within the Serena project provide AI agents with IDE-like capabilities to explore codebases at the symbol level rather than through primitive text-based searches. These tools exploit the relational structure of code to understand relationships between classes, methods, and functions, allowing agents to find relevant context efficiently without reading entire files. This approach is designed to be token-efficient and more reliable than traditional regex searches, especially in large and complex projects.
Sources: README.md:10-15, README.md:133-135, src/serena/resources/config/prompt_templates/system_prompt.yml:12-16
The system operates using two primary backends: Language Servers implementing the Language Server Protocol (LSP) and the Serena JetBrains Plugin. These backends provide the underlying code analysis required for high-level retrieval tasks such as finding symbol definitions, viewing file outlines, and tracing call hierarchies.
Sources: README.md:85-92, README.md:140-150
Core Retrieval Capabilities
The retrieval system is built around several specialized tools that allow for targeted discovery of code elements. Unlike basic file reading, these tools return structured information about the symbolic structure of the project.
Symbol Identification and Navigation
Symbols are identified by their name_path and relative_path. The system encourages a "step-by-step acquisition" strategy where an agent first gets an overview of a file's symbols before requesting the body of specific elements.
| Capability | Tool Name | Description |
|---|---|---|
| File Outline | get_symbols_overview | Provides a compact, hierarchical view of symbols within a specific file. |
| Find Symbol | find_symbol | Locates symbols by name pattern; can include docstrings, signatures, or the full body. |
| Cross-References | find_referencing_symbols | Identifies all locations where a specific symbol is called or used. |
| Hierarchy Discovery | type_hierarchy | Fetches superclasses and subclasses for a given symbol (JetBrains exclusive). |
| Find Declaration | find_declaration | Locates the specific definition of a symbol. |
| Find Implementation | find_implementations | Locates concrete implementations of interfaces or abstract classes. |
Sources: src/serena/resources/config/prompt_templates/system_prompt.yml:33-40, README.md:140-152, news/20260111.html:10-22
Data Flow for Retrieval Requests
The following sequence diagram illustrates how an AI agent interacts with the Serena semantic tools to retrieve code information using the LSP backend.
Implementation Details
LSP Backend Support
The SolidLSP library provides the bridge to various language servers. For instance, the KotlinLanguageServer handles asynchronous indexing and communicates via standard LSP requests like textDocument/definition and textDocument/references.
Key components in the retrieval logic include:
- Symbol Kind Mapping: Various language-specific symbols (e.g., Kotlin classes, mSL aliases) are mapped to unified types for the agent.
- Response Buffering: The system includes a
symbol_info_budgetto prevent long-running retrieval tasks from blocking the agent by returning partial results after a timeout.
Sources: src/solidlsp/language_servers/kotlin_language_server.py:202-230, src/serena/config/serena_config.py:730-740
mIRC Scripting Language (mSL) Retrieval
The msl_lsp_server.py provides a concrete example of how custom semantic retrieval is implemented for specialized languages. It uses regex patterns to parse top-level constructs and builds a symbol map.
# mSL top-level construct patterns used for symbol discovery
ALIAS_PATTERN = re.compile(
r"^[ \t]*alias\s+(?:-l\s+)?([a-zA-Z_][\w.]*)\s*(?:\{|$)",
re.MULTILINE | re.IGNORECASE,
)
EVENT_PATTERN = re.compile(
r"^[ \t]*on\s+(\*|\d+):(\w+):([^{]*?)(?:\{|$)",
re.MULTILINE | re.IGNORECASE,
)
Sources: src/solidlsp/language_servers/msl_lsp_server.py:24-35
Configuration and Budgeting
Retrieval behavior is governed by the SerenaConfig, which defines the time budget for gathering extended information (docstrings and signatures).
| Configuration Option | Type | Default | Description |
|---|---|---|---|
symbol_info_budget | float | 10.0 | Max seconds for gathering extended info before returning partial results. |
language_backend | Enum | LSP | Determines whether to use LSP servers or the JetBrains plugin. |
ignored_paths | list[str] | [] | Paths excluded from semantic indexing. |
Sources: src/serena/config/serena_config.py:730-745, src/serena/config/serena_config.py:710-725
Conclusion
Semantic Code Retrieval Tools serve as the primary sensory organ for AI agents within the Serena ecosystem. By abstracting away raw file reading in favor of symbol-aware navigation, the system enables more precise analysis and refactoring. The multi-backend support ensures that these capabilities are available across more than 40 programming languages, maintaining a consistent interface for the agent regardless of the underlying language technology.
Sources: README.md:13-18, README.md:105-115