Segment and MatchResult System

Relevant source files

The following files were used as context for generating this wiki page:

• src/sqlfluff/core/parser/segments/base.py
• src/sqlfluff/core/parser/match_result.py
• docsv/development/architecture.md
• src/sqlfluff/AGENTS.md
• AGENTS.md
• src/sqlfluff/dialects/AGENTS.md

The Segment and MatchResult system provides the core infrastructure for representing and constructing the SQL Abstract Syntax Tree (AST). All nodes in the AST inherit from a base segment class, forming a recursive tree structure where segments represent everything from high-level statements to low-level whitespace. The matching system uses grammars to evaluate these segments against raw SQL, producing results that dictate how the flat sequence of initial tokens transforms into a nested hierarchy.

This system operates primarily during the parsing stage of the SQLFluff pipeline. It follows the lexer, which produces a flat sequence of typed segments. The parser then applies dialect-specific grammars to these segments, utilizing the MatchResult to capture instructions for tree construction without immediately modifying the existing structure.

Sources: docsv/development/architecture.md, AGENTS.md

Segment Architecture

The BaseSegment class serves as the foundation for all AST nodes. It defines the interface for matching, traversal, and immutability. Segments in SQLFluff are strictly immutable; you must create a new instance via .copy() or use the LintFix mechanism to propose changes rather than modifying an existing segment in place.

Segment Types and Hierarchy

Segments form a tree-like structure starting from a FileSegment. This root contains StatementSegment nodes, which further contain clauses and expressions.

Segment Category	Description	Implementation Detail
RawSegment	Low-level segments produced by the lexer.	Includes whitespace and individual code tokens.
BaseSegment	The parent class for all AST nodes.	Defines match_grammar and type.
UnparsableSegment	A wrapper for segments that fail to match a grammar.	Often created via GREEDY parse modes.

Sources: docsv/development/architecture.md, src/sqlfluff/AGENTS.md

Recursive Matching Logic

A segment's .match() method delegates work to its match_grammar. The parser performs a single pass through the file, recursively matching segments based on their definitions. This recursion terminates when the parser reaches raw segments that contain a single token and have no children.

MatchResult System

The MatchResult object captures the outcome of a grammar's attempt to match a sequence of segments. It acts as a container for instructions on how to structure the resulting tree.

Components of a MatchResult

A MatchResult includes the matched segments and any remaining segments that did not match. It provides an .apply() method which, when called at the end of the matching process, finalizes the nested structure.

• Matched Segments: The portion of the input that successfully aligned with the grammar rules.
• Unmatched Segments: The remaining input following a successful match.
• Structural Instructions: Data used to turn a flat sequence into a nested tree.

Sources: docsv/development/architecture.md

Logic Flow for Matching

The matching process follows a specific sequence to ensure stability and accuracy during parsing.

Grammar Primitives

Grammars combine segments or other grammars to describe the shape of SQL constructs. These primitives are located in the core parser and are utilized extensively within dialect definitions to extend or replace ANSI behavior.

Key Grammar Primitives

Primitive	Logic
Sequence	Matches elements in a specific order.
OneOf	Matches if any one of its child elements match (Choice).
Delimited	Matches a list separated by a specific delimiter (usually commas).
Bracketed	Matches content enclosed in parentheses or brackets.
Ref	References another segment by name for runtime resolution.
AnyNumberOf	Matches zero or more repetitions of the child elements.

Sources: src/sqlfluff/dialects/AGENTS.md

Usage Example in Dialects

Dialects use Ref to allow name resolution at runtime. This enables a patched grammar to override specific elements while relying on lower-level elements that remain unchanged in the parent dialect.

class SelectStatementSegment(BaseSegment):
    type = "select_statement"
    match_grammar = Sequence(
        "SELECT",
        Ref("SelectClauseSegment"),
        Ref("FromClauseSegment", optional=True),
        Ref("WhereClauseSegment", optional=True),
    )

Sources: src/sqlfluff/AGENTS.md, docsv/development/architecture.md

Immutability and Persistence

The system maintains architectural integrity through strict immutability. Segments are never modified in place. When a rule or process needs to change the tree, it produces a new version of the segment.

• Copying: Use segment.copy() to create a modified version of a segment with different attributes or children.
• LintFix: Rules generate LintFix objects that describe edits, inserts, or deletions.
• Finalization: The linter applies these fixes to create an updated tree, which is eventually written back to the file.

Sources: src/sqlfluff/AGENTS.md, docsv/development/architecture.md

Summary

The Segment and MatchResult system is the structural backbone of SQLFluff's parsing capabilities. By using immutable BaseSegment nodes and a grammar-driven MatchResult mechanism, the system transforms raw SQL into a traversable AST. This design allows for flexible dialect inheritance, where specific grammar components can be overridden without compromising the underlying parsing logic. The clear separation between matching (logic) and applying (structure) ensures that the complex task of SQL parsing remains modular and maintainable.