Lexer & Parser Design
Relevant source files
The following files were used as context for generating this wiki page:
The Jazz language front-end is designed to transform human-readable source code into a structured Abstract Syntax Tree (AST). The design focuses on a functional syntax inspired by Haskell and Elixir, utilizing dot-terminated source forms and brace-bodied module declarations. The active implementation resides within the jazz-next/ directory, which serves as the canonical path for new compiler functionality.
The parsing architecture follows a standard multi-stage pipeline: tokenization via a lexer, surface parsing into a parse-specific AST, and a lowering phase that prepares the AST for semantic analysis.
Sources: README.md:14-22, jazz-next/README.md:1-20
Compiler Front-End Architecture
The jazz-next compiler employs a modular structure for its front-end components. The process begins with the lexer, which provides minimal tokenization necessary to bootstrap the parser. The parser then consumes these tokens to produce a surface AST.
This diagram illustrates the data flow from raw source code to the core AST used by the semantic analyzer.
Key Components
| Component | File Path | Description |
|---|---|---|
| Lexer | src/JazzNext/Compiler/Parser/Lexer.hs | Responsible for minimal tokenization and parser bootstrap. |
| Parser | src/JazzNext/Compiler/Parser.hs | Implements the surface parser for current syntax slices. |
| Parse AST | src/JazzNext/Compiler/Parser/AST.hs | Defines the data structures for the initial parse results. |
| Lowering | src/JazzNext/Compiler/Parser/Lower.hs | Maps the Parse-AST to the Analyzer-facing core AST. |
| Core AST | src/JazzNext/Compiler/AST.hs | The simplified AST used for analysis and type inference. |
Sources: jazz-next/README.md:14-22
Lexical Structure and Syntax
Jazz syntax prioritizes readability and standard functional programming paradigms. Key lexical and structural features include:
- Dot-Terminated Forms: Expressions and declarations are terminated with a period (
.). - Brace-Bodied Modules: Module declarations use curly braces
{}to encapsulate definitions. - Lambda Syntax: Functions are declared via assignment to a lambda using
\and->(e.g.,isEven = \(i) -> mod(i, 2) == 0.). - Impure Denotation: Functions with side effects must end with a
!(e.g.,print!). - Comments: Support for line comments starting with
//or#.
Sources: README.md:14-22, README.md:73-81, README.md:120-128
Parsing Implementation Details
The parsing logic is implemented using the megaparsec library, as indicated by the project's dependency specifications. The parser is responsible for handling various language constructs including operator precedence and fixity.
The sequence of events from source consumption to core AST generation.
Operators and Fixity
The language defines a built-in operator table and supports a staged extensibility model. A notable feature is the use of the $ operator, which is defined as a right-associative, zero-precedence function application operator used to avoid excessive parentheses.
Sources: jazz-hs/package.yaml:14-20, README.md:65-71, scripts/check-clarification-specs.sh:35-37
AST Lowering and Integration
The transition from the parser to the rest of the compiler is managed by the "Lowering Boundary." This ensures that the syntactic sugar of the surface language is removed before semantic analysis begins. The lowering phase converts the Parser/AST.hs structures into Compiler/AST.hs structures.
Lowering Responsibilities
- Desugaring: Converting high-level constructs like
if-expressionsinto canonical forms. - Adjacency Checks: Ensuring signature adjacency (e.g., type signatures appearing directly before their corresponding bindings).
- Semantic Preparation: Mapping surface syntax to the core subset used for type inference and purity enforcement.
Sources: jazz-next/README.md:14-25, scripts/check-clarification-specs.sh:31-33
Conclusion
The Lexer and Parser design in Jazz provides a structured pathway from Haskell-inspired surface syntax to a strongly typed core expression subset. By utilizing megaparsec and a dedicated lowering boundary, the architecture maintains a clear separation between syntactic representation and semantic analysis.