Language & Parser (AST)
Relevant source files
The following files were used as context for generating this wiki page:
- src/System.Management.Automation/engine/parser/Parser.cs
- src/System.Management.Automation/engine/parser/DebugViewWriter.cs
- src/System.Management.Automation/help/HelpCommentsParser.cs
- src/System.Management.Automation/engine/CommandCompletion/CompletionAnalysis.cs
- src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs
- src/System.Management.Automation/engine/lang/interface/PSParser.cs
The PowerShell Language and Parser system is responsible for transforming raw script text into a structured Abstract Syntax Tree (AST). This system serves as the foundation for script execution, static analysis, syntax highlighting, and IntelliSense. The core component is the Parser class, which utilizes a Tokenizer to break input into tokens before applying grammatical rules to build the tree.
The AST represents the hierarchical structure of PowerShell code, ranging from high-level script blocks down to individual expressions and literal constants. This structured representation allows other engine components, such as the Compiler, to generate executable code, and tools like CompletionAnalysis to provide context-aware suggestions.
Sources: src/System.Management.Automation/engine/parser/Parser.cs:33-45, src/System.Management.Automation/engine/lang/interface/PSParser.cs:34-45
Parser Architecture and Data Flow
The parsing process is a multi-stage pipeline that converts text into a functional ScriptBlockAst. The Parser class manages the state of the parse, including error lists and the current tokenizer mode.
Parsing Pipeline
The pipeline begins with raw input string or a file path. The Parser initializes a Tokenizer, which scans the characters to produce Token objects. The parser then consumes these tokens using recursive descent rules (e.g., StatementRule, ExpressionRule, PipelineRule) to construct AST nodes.
The parsing entry points include ParseFile and ParseInput. Both methods return a ScriptBlockAst along with an array of tokens and any discovered ParseError objects.
Sources: src/System.Management.Automation/engine/parser/Parser.cs:57-140
Tokenization Modes
The Parser switches the Tokenizer between different modes depending on the grammatical context. This is crucial for correctly interpreting tokens that may have different meanings in command versus expression contexts.
| Mode | Usage Context |
|---|---|
Command | Parsing command names, parameters, and arguments. |
Expression | Parsing mathematical operations, variables, and logic. |
TypeName | Specialized mode for parsing CLR type specifications. |
Signature | Used for method and class member definitions. |
| Source: src/System.Management.Automation/engine/parser/Parser.cs:407-440 |
Core AST Components
The Abstract Syntax Tree is composed of specialized nodes representing PowerShell language constructs. All nodes derive from a base Ast class.
Statement Structures
Statements are the primary building blocks of script logic. The parser identifies several types of statements:
- PipelineAst: Represents one or more commands linked by pipes.
- IfStatementAst: Handles conditional logic including
elseifandelseblocks. - Looping Statements: Includes
ForStatementAst,ForEachStatementAst,WhileStatementAst, andDoWhileStatementAst. - Type Definitions: Handles
ClassDefinitionAstandEnumDefinitionAst.
Sources: src/System.Management.Automation/engine/parser/Parser.cs:720-1110, src/System.Management.Automation/engine/parser/Parser.cs:1340-1550
Expression Components
Expressions are units of code that return a value. The parser evaluates these through precedence-based rules:
- BinaryExpressionAst: Operators like
+,-,*,-eq,-and. - UnaryExpressionAst: Operators like
++,--,-not. - TernaryExpressionAst: Implements the
<condition> ? <true> : <false>logic. - MemberExpressionAst: Handles property access (
.) and static member access (::).
Sources: src/System.Management.Automation/engine/parser/Parser.cs:1900-2100, src/System.Management.Automation/engine/parser/Parser.cs:2380-2450
Help and Comment Parsing
The system includes a specialized HelpCommentsParser that extracts metadata from comment blocks within the AST. This is used to generate HelpInfo objects for functions and scripts.
The parser looks for directives like .SYNOPSIS, .DESCRIPTION, and .PARAMETER to build structured help documentation.
Sources: src/System.Management.Automation/help/HelpCommentsParser.cs:181-220, src/System.Management.Automation/help/HelpCommentsParser.cs:484-520
IntelliSense and Completion Analysis
The CompletionAnalysis class utilizes the AST to provide context-aware IntelliSense. It determines the cursor's position relative to the AST tree to suggest valid commands, parameters, or file paths.
Context Extraction
The ExtractAstContext method identifies the TokenAtCursor and the RelatedAsts. This allows the engine to know, for example, if the user is currently inside a HashtableAst (suggest keys) or a CommandAst (suggest parameters).
| Completion Type | Logic Base |
|---|---|
| Variables | Scans current and parent scopes for variable names. |
| Members | Performs type inference on the target expression to list properties/methods. |
| Command Arguments | Uses parameter metadata or custom argument completers. |
| Types | Scans loaded assemblies for type name matches. |
| Sources: src/System.Management.Automation/engine/CommandCompletion/CompletionAnalysis.cs:115-150, src/System.Management.Automation/engine/CommandCompletion/CompletionAnalysis.cs:330-400 |
Script Block Compilation and Security
Once the AST is generated, it is encapsulated in a ScriptBlock. Before execution, the script block undergoes security checks and compilation.
Execution Workflow
- Security Scanning: The system uses the Anti-Malware Scan Interface (AMSI) to scan the script text. It also checks for suspicious content patterns.
- Compilation: The
Compilertransforms the AST into executable delegates. It can produce "optimized" versions (using MutableTuples for fast local variable access) or "unoptimized" versions for debugging. - Logging: If enabled via Group Policy, the full text of the script block is logged to the event log. Large scripts are split into fragments to circumvent ETW size limits.
Sources: src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs:98-130, src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs:166-190, src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs:720-750
Conclusion
The Language & Parser (AST) subsystem is a robust recursive descent parser that provides the high-fidelity representation of PowerShell code required for the modern engine. By decoupling the structure (AST) from the execution (Compiler), PowerShell supports advanced features like cross-language boundary security, deep IntelliSense, and comprehensive script logging. This architecture ensures that regardless of whether a script is being executed, analyzed for help, or used in an IDE, the interpretation of the language remains consistent and secure.
Sources: src/System.Management.Automation/engine/parser/Parser.cs:33-50, src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs:450-470