AST and PatchedAST
Relevant source files
The following files were used as context for generating this wiki page:
The Rope library utilizes a specialized version of the Python Abstract Syntax Tree (AST) called PatchedAST. While the standard Python ast module provides a tree representation of the code's structure, it loses information regarding source formatting, such as exact whitespace, comments, and parentheses. PatchedAST bridges this gap by augmenting standard AST nodes with region information and a mapping of child elements that include non-semantic tokens. Sources: rope/refactor/patchedast.py:12-25
The primary purpose of PatchedAST is to enable safe and precise refactoring. By tracking the exact offsets of code fragments and preserving the surrounding format, Rope can perform complex transformations—like Extract Method or Use Function—without corrupting the original coding style or losing user comments. Sources: rope/refactor/extract.py:140-155, rope/refactor/patchedast.py:46-54
Architecture and Core Components
The PatchedAST system functions by traversing a standard AST and mapping its nodes back to the original source string.
The Patching Process
The transformation starts with get_patched_ast(source), which parses the source using ast.parse and then applies the patching logic via the _PatchingASTWalker class. Sources: rope/refactor/patchedast.py:12-16
This diagram illustrates the flow from raw source code to a fully patched AST structure. Sources: rope/refactor/patchedast.py:12-40
Key Classes and Functions
| Component | Description |
|---|---|
get_patched_ast | High-level entry point that returns a patched AST with region and (optionally) sorted_children fields. |
patch_ast | Patches an existing AST node recursively. |
_PatchingASTWalker | Internal walker that calculates offsets and identifies formatting tokens. |
_Source | Helper class that manages the source string buffer and token consumption (e.g., strings, numbers). |
node_region | Utility to retrieve the (start, end) offset tuple from a patched node. |
write_ast | Reconstructs source code string from a node using its sorted_children field. |
Sources: rope/refactor/patchedast.py:12-65, rope/refactor/patchedast.py:480-500
Node Augmentation
When a node is patched, two critical attributes are typically added to it:
Region Information
The region attribute is a tuple containing the start and end offsets of the code that generated the node. This allows Rope to perform direct string manipulations at the correct location in the file. Sources: rope/refactor/patchedast.py:22-25
Sorted Children
If requested, the sorted_children attribute is created. It is a list containing:
- Child AST Nodes: Nested structural elements.
- Formatting Tokens: Strings containing whitespace, comments, and punctuation (like commas or parentheses) that exist between child nodes.
Sources: rope/refactor/patchedast.py:27-31, rope/refactor/patchedast.py:58-65
Source Consumption Logic
The _PatchingASTWalker uses a "consumption" strategy to identify tokens. It maintains a pointer in the source and moves forward as it matches AST node requirements against the raw text.
This diagram shows how the walker synchronizes AST structure with linear source code consumption. Sources: rope/refactor/patchedast.py:82-125
Parentheses Handling
A significant portion of the logic involves _handle_parens and _eat_surrounding_parens. Because the Python AST does not explicitly store parentheses, the walker must search backwards and forwards in the source to find and include them in the node's region if they exist. Sources: rope/refactor/patchedast.py:140-165
Applications in Refactoring
PatchedAST is the foundation for various refactoring modules in Rope.
Similar Finder
The SimilarFinder uses PatchedAST nodes to match structural patterns while accounting for wildcards. It relies on node_region to identify the exact code to be replaced or analyzed. Sources: rope/refactor/similarfinder.py:30-45, rope/refactor/similarfinder.py:80-95
Extract Method and Variable
The ExtractMethod refactoring uses patchedast.node_region to verify that selected code regions contain complete, valid statements and to reconstruct the extracted body with correct indentation. Sources: rope/refactor/extract.py:228-245, rope/refactor/extract.py:530-545
Module Imports
The ModuleImports class utilizes the AST to find global import statements and calculate separating lines between import blocks and the rest of the module body. Sources: rope/refactor/importutils/module_imports.py:20-35, rope/refactor/importutils/module_imports.py:320-335
# Example of using node_region in refactor/extract.py
def _replace_occurrences(self, content, extract_info):
for match in extract_info.matches:
# ... logic to find replacement ...
start, end = patchedast.node_region(match.get_ast(name))
mapping[name] = self.info.source[start:end]
Sources: rope/refactor/extract.py:230-238
Technical Limitations and Errors
The system can raise a MismatchedTokenError if the expected tokens in the source do not align with the AST structure provided by Python's parser. This usually occurs if the source has been modified in memory without re-parsing or if there are unexpected characters in comments. Sources: rope/refactor/patchedast.py:68-69, rope/refactor/patchedast.py:484-492
Summary
PatchedAST is a critical infrastructure component in Rope that augments standard AST nodes with source-mapping metadata. By providing both precise source regions and a list of formatting-inclusive children, it allows Rope to perform sophisticated refactoring operations while maintaining code fidelity and user formatting. Sources: rope/refactor/patchedast.py:12-31, rope/refactor/extract.py:1-20