Lexer Functionality

Relevant source files

The following files were used as context for generating this wiki page:

• src/sqlfluff/core/parser/lexer.py
• sqlfluffrs/AGENTS.md
• docsv/development/architecture.md
• AGENTS.md
• src/sqlfluff/core/linter/linter.py
• src/sqlfluff/AGENTS.md

The Lexer serves as the second stage in the SQLFluff pipeline, following the templater. It transforms raw or post-templated SQL text into a flat sequence of typed segments. These segments represent the smallest meaningful units of the language, such as whitespace, keywords, and operators.

SQLFluff utilizes a hybrid approach for lexing. The primary implementation is written in Python, while an experimental Rust component (sqlfluffrs) provides high-performance tokenization. When enabled, the Rust lexer returns RsToken objects that the Python layer wraps into standard segments.

Sources: docsv/development/architecture.md, sqlfluffrs/AGENTS.md

Architecture and Data Flow

The lexing process begins after the templater converts raw code into valid SQL. The Lexer receives a TemplatedFile or a raw string and iterates through it to identify patterns defined by the chosen dialect.

The diagram shows the transition from templated SQL through the lexing stage to produce segments for the parser.
Sources: docsv/development/architecture.md, src/sqlfluff/core/linter/linter.py:165-177, sqlfluffrs/AGENTS.md

Lexing Stages

The Lexer performs the following operations:

1. Selection: The system chooses between the Python or Rust lexer based on configuration.
2. Matching: The lexer applies the specified dialect's LexMatcher patterns to the input string.
3. Typing: Every identified token is assigned a type, becoming a subclass of RawSegment.
4. Verification: The lexer ensures that the resulting segments maintain a sensible indentation balance unless specifically suppressed by configuration.

Sources: docsv/development/architecture.md, src/sqlfluff/core/linter/linter.py:180-210, sqlfluffrs/AGENTS.md

Component Overview

Component	Language	Description
Lexer Class	Python	The main interface for tokenizing SQL within the Python core.
RsLexer	Rust	High-performance experimental lexer exposed via PyO3 bindings.
RawSegment	Python	The base class for all flat segments produced by the lexer.
LexMatcher	Python/Rust	Defines the specific regex or logic used to identify tokens for a dialect.

Sources: docsv/development/architecture.md, sqlfluffrs/AGENTS.md, src/sqlfluff/core/linter/linter.py:175-177

Implementation Details

Python Lexer

The Python Lexer resides in src/sqlfluff/core/parser/lexer.py. It uses regular expressions to scan the input string. The _lex_templated_file method in the linter orchestrates the process, capturing any SQLLexError that occurs during the scan.

# Example of lexer orchestration in the linter
def _lex_templated_file(
    templated_file: "TemplatedFile", config: FluffConfig
) -> tuple[Optional[Sequence[BaseSegment]], list[SQLLexError]]:
    lexer = Lexer(config=config)
    try:
        segments, lex_vs = lexer.lex(templated_file)
        return segments, lex_vs
    except SQLLexError as err:
        return None, [err]

Sources: src/sqlfluff/core/linter/linter.py:165-189

Rust Lexer (sqlfluffrs_lexer)

The Rust implementation tokenizes strings into Vec<Token> using the dialect's matchers. It is exposed to Python through the sqlfluffrs module. The Rust lexer is designed for performance and is particularly beneficial for large SQL files where the overhead of the Python-Rust boundary is amortized.

The sequence diagram illustrates how the Python linter interacts with the Rust lexer to receive high-performance tokens.
Sources: sqlfluffrs/AGENTS.md

Configuration and Indentation

Lexing behavior is influenced by the FluffConfig. Specifically, the template_blocks_indent setting determines if the lexer should enforce or ignore indentation balance within template blocks.

• Standard Balance: If indent_balance is non-zero, the lexer may disable template indents for the file to prevent parsing errors.
• Forced Indent: If the configuration is set to force, the balance check is bypassed.

Sources: src/sqlfluff/core/linter/linter.py:192-210

Lexer Error Handling

If the lexer cannot match a portion of the input string to any known token type for the dialect, it generates an SQLLexError. These errors are collected and returned to the linter for reporting. Unlike the parser, which can wrap unmatchable content in an UnparsableSegment, the lexer requires a valid match for the entire string to complete successfully.

Sources: src/sqlfluff/core/linter/linter.py:182-189, docsv/development/architecture.md

Summary

The Lexer Functionality in SQLFluff provides a robust foundation for the parsing stage by converting raw SQL into a structured, flat list of segments. By supporting both a flexible Python implementation and a high-performance Rust implementation, the system balances dialect extensibility with computational efficiency. The resulting flat sequence of typed segments allows the subsequent Parser stage to build the recursive segment tree required for linting and fixing.

Sources: docsv/development/architecture.md, sqlfluffrs/AGENTS.md