Lexical Analysis (sploosh-lexer)

Relevant source files

The following files were used as context for generating this wiki page:

• crates/sploosh-lexer/src/lib.rs
• crates/sploosh-lexer/Cargo.toml
• crates/sploosh-ast/Cargo.toml
• crates/AGENTS.md
• README.md
• crates/sploosh-parser/src/lib.rs

The sploosh-lexer crate provides a UTF-8 tokenizer that implements the lexical specifications defined in §2 and §16.1 of the Sploosh language specification. Its primary responsibility is to transform raw source text into a stream of structured tokens, facilitating the work of the Recursive-Descent Parser. The lexer is designed to be allocation-free during the scanning process by carrying only source spans rather than allocating new strings for each token.

Sources: crates/sploosh-lexer/src/lib.rs:1-15, crates/AGENTS.md:14-16

Core Data Structures

The lexical analysis process centers on the Token struct and the TokenKind enumeration. Every token identified in the source code is represented by its kind and a Span, which denotes the start and end offsets in the original UTF-8 source string.

Token and TokenKind

The Token structure includes a helper method text() to slice the source on demand, ensuring efficient memory usage. The TokenKind enum covers all categories of lexical elements, including identifiers, reserved keywords, literals (integer, float, string, char), and various punctuation operators.

Data Definitions Table

Component	Description	Path
Token	Represents a single lexical unit with a TokenKind and a Span.	crates/sploosh-lexer/src/lib.rs:6
TokenKind	Enum defining all possible token types (literals, operators, etc.).	crates/sploosh-lexer/src/lib.rs:17
Keyword	Enum for reserved keywords like fn, actor, and onchain.	crates/sploosh-lexer/src/lib.rs:65
LexError	Captures failures during lexing, including an error message and the offending span.	crates/sploosh-lexer/src/lib.rs:125

Lexing Logic and Data Flow

The Lexer internal struct manages the state of the scan, maintaining the current byte position (pos) and accumulating resulting tokens or errors. The entry point is the lex function, which initializes the Lexer and executes the scanning loop.

Keywords and Identifiers

The lexer distinguishes between Reserved Keywords and Contextual Keywords. Reserved keywords (e.g., let, fn, match) are immediately classified as TokenKind::Keyword. Contextual spellings (e.g., send, storage, mut) are initially treated as identifiers (TokenKind::Ident), leaving the final classification to the parser.

Sources: crates/sploosh-lexer/src/lib.rs:103-123, crates/AGENTS.md:63-65

Literals and Validation

The lexer implements complex logic for numeric, string, and character literals, including validation for separators and escape sequences.

Numeric Literals

Numeric literals support various bases (binary, octal, hex, decimal) and specific type suffixes (e.g., u32, i128, f64). The lexer ensures that numeric separators (_) only appear between valid digits.

• Integer Suffixes: i128, u128, u256, i64, u64, i32, u32, i16, u16, i8, u8.
• Float Suffixes: f64, f32.

Sources: crates/sploosh-lexer/src/lib.rs:121-123, crates/sploosh-lexer/src/lib.rs:253-332

Escape Sequences

The lexer validates several escape sequence formats within strings and character literals:

1. Standard: \n, \r, \t, \\, \", \', \0.
2. Hexadecimal: \xHH (must be within 0x00..0x7F).
3. Unicode: \u{HHHHHH} (1-6 hex digits, must be a valid Unicode scalar).

Sources: crates/sploosh-lexer/src/lib.rs:420-474

Summary of Operators and Punctuation

The lexer recognizes a wide array of single, double, and triple-character punctuation. Notable operators include the pipeline operators (|> and |>?) and range operators (.. and ..=).

Token	Character	TokenKind
Pipe	|	Pipe
PipeGt	|>	PipeGt
DotDot	..	DotDot
DotDotEq	..=	DotDotEq
FatArrow	=>	FatArrow
ColonColon	::	ColonColon

Sources: crates/sploosh-lexer/src/lib.rs:20-59, crates/sploosh-lexer/src/lib.rs:194-230

Implementation Details

The lexer is written in idiomatic Rust, utilizing the sploosh-ast crate for shared Span definitions. It avoids external dependencies to remain "dependency-light" during the bootstrap phase of the compiler.

// crates/sploosh-lexer/src/lib.rs:139-145
pub fn lex(source: &str) -> Result<Vec<Token>, Vec<LexError>> {
    let mut lexer = Lexer {
        source,
        pos: 0,
        tokens: Vec::new(),
        errors: Vec::new(),
    };
    lexer.run();
    // ... logic for returning tokens or errors
}

Sources: crates/sploosh-lexer/src/lib.rs:139-152, crates/sploosh-lexer/Cargo.toml:1-12

Lexical analysis is the foundation of the Sploosh compiler frontend. By providing a clean, error-aware token stream, it enables the parser to enforce the language's grammar and target-specific restrictions (such as on-chain prohibitions) later in the compilation pipeline.

Sources: README.md:120-130, crates/sploosh-parser/src/lib.rs:1-15