Compiler Pipeline & Query Engine

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• architecture/21-cli-tooling-and-code-units.md
• architecture/19-architecture-conformance-and-regression-policy.md
• architecture/08.1-closed-design-questions.md
• architecture/20-xml-documentation-comments.md
• crates/tools/architecture-tests/src/tests.rs

The Pop Lang compiler pipeline is a multi-stage semantic transformation system designed to convert Luau-inspired source code into native executables or bytecode. The architecture prioritizes strong static typing, backend neutrality, and incremental analysis through a query-based engine. This design ensures that every runtime operation has a compiler-proven type and that backends (such as LLVM or the MIR interpreter) consume a verified, unambiguous intermediate representation.

The pipeline is structured to support the Pop Lang ownership hierarchy, which organizes code into Items, Modules, Bubbles, Packages, and Workspaces. This structure enables precise incremental invalidation and stable symbol resolution across independent compilation units.

The Semantic Pipeline

The compiler processes source code through a sequence of well-defined stages. Each stage is verified before transitioning to the next to maintain architectural invariants.

The mandatory semantic stages of the Pop Lang compiler.
Sources: AGENTS.md:355-357, architecture/21-cli-tooling-and-code-units.md:65-68

Pipeline Stages Summary

Stage	Description	Key Output
Lexing & Parsing	Converts source text into tokens and a lossless syntax tree (LST) that preserves trivia.	Lossless Syntax Tree
Indexing	Maps declarations within a Bubble to stable symbol identities.	Declaration Index
Resolution	Binds identifiers to their respective declarations across namespaces and Bubbles.	Resolved AST
Typed Analysis	Performs type inference, narrowing, and compile-time evaluation.	Typed AST
HIR Lowering	Transforms the AST into High-Level IR, preserving typed language concepts.	HIR Bubble
MIR Lowering	Lowers HIR to Mid-Level IR, making control flow, effects, and GC points explicit.	MIR Bubble
Code Generation	Translates MIR into target-specific opcodes or source (LLVM, C, Bytecode).	Native/VM Artifact

Sources: AGENTS.md:355-357, architecture/21-cli-tooling-and-code-units.md:204-208

Intermediate Representations (IR)

Pop Lang utilizes two primary intermediate representations to bridge the gap between high-level source semantics and low-level machine execution. A critical architectural invariant is that both HIR and MIR must remain backend-neutral, containing no LLVM-specific objects or opcodes.

High-Level IR (HIR)

HIR preserves the typed concepts of the language. It uses stable, typed entity IDs (e.g., WorkspaceId, BubbleId, ModuleId) to track declarations across the project hierarchy. HIR is the primary target for initial semantic verification and cross-Bubble reference resolution.
Sources: AGENTS.md:364-367, architecture/19-architecture-conformance-and-regression-policy.md:10-15

Mid-Level IR (MIR)

MIR is the final portable representation of the program. It makes all executable logic explicit, including:

• Control flow and evaluation order.
• Memory allocation and Garbage Collection (GC) safe points.
• Side effects and failure paths (traps).
• Runtime operations and interface dispatch.

MIR serves as the contract for the LLVM backend, the MIR interpreter, and future virtual machine implementations. It undergoes a "hybrid lowering" process where value types and performance-critical generic instantiations are specialized, while compatible reference types share code via witness tables.
Sources: AGENTS.md:359-362, architecture/08.1-closed-design-questions.md:105-115

The Query Engine & Incremental Analysis

The compiler implementation leverages a query-based architecture to support incremental compilation and rich IDE tooling. Rather than re-parsing the entire project on every change, the query engine tracks dependencies between semantic units.

Incremental Invalidation

The engine operates at the Module and Bubble levels. A Module (a single .pop file) owns source spans and private items. When a Module is modified, only the dependent queries in the query graph are invalidated.
Sources: architecture/21-cli-tooling-and-code-units.md:65-68

Context Structures

The compiler uses aggregate context structures to manage state:

• HirBubble / MirBubble: Represent the aggregate IR for an independently compiled unit.
• BubbleContext: Manages dependencies and metadata for a compilation unit.
• SymbolId: A stable identifier for declarations, ensuring that renaming a Module does not break public item identities if their definition keys are preserved.

Sources: AGENTS.md:366-368, architecture/21-cli-tooling-and-code-units.md:70-75

Documentation Sub-Pipeline

The compiler integrates XML documentation parsing directly into the semantic pipeline. Documentation is treated as a checked companion to declarations rather than unparsed trivia.

The integrated documentation processing flow within the compiler.
Sources: architecture/20-xml-documentation-comments.md:144-154

Key features of the documentation pipeline:

1. Static Resolution: cref links resolve through the same resolver used for code, producing stable DocId values.
2. Semantic Validation: The compiler compares documented <param>, <returns>, and <error> tags against the actual typed signature of the function.
3. Separation of Concerns: Documentation is emitted into a separate documentation.xml file within .poplib artifacts, ensuring that documentation-only changes do not trigger implementation rebuilds.

Sources: architecture/20-xml-documentation-comments.md:135-142, architecture/20-xml-documentation-comments.md:160-165

Verification and Constraints

The pipeline enforces several "Lua regressions" preventions and architectural invariants through automated tests and internal verifiers:

• No Dynamic Fallback: The pipeline must never introduce Any or Dynamic types. Every operation must be statically proven.
• Backend Neutrality: HIR and MIR cannot call back into parsing or resolution; they are one-way transformations.
• Crate Boundaries: Architecture tests verify that portable crates (e.g., pop-syntax, pop-hir) never depend on backend-specific crates (e.g., pop-backend-llvm).

Sources: AGENTS.md:363-365, architecture/19-architecture-conformance-and-regression-policy.md:60-75, crates/tools/architecture-tests/src/tests.rs:340-350

Summary of Code Units

The Query Engine resolves relationships based on a strict hierarchy:

Unit	Scope	Responsibility
Item	Declaration	Smallest semantic unit (SymbolId).
Module	.pop file	Boundary for private visibility and incremental invalidation.
Bubble	Target	Unit of independent compilation and internal visibility.
Package	Directory	Versioning and distribution identity (bubble.toml).
Workspace	Monorepo	Shared dependency resolution and lockfile (bubble.lock).

Sources: architecture/21-cli-tooling-and-code-units.md:10-25