Autoimport SQLite Database
Relevant source files
The following files were used as context for generating this wiki page:
The Autoimport SQLite Database is a performance-oriented implementation of Rope's auto-import feature. Its primary purpose is to maintain a cache of global names across Python modules, allowing the library to quickly suggest and generate import statements for names that are not currently in scope. This implementation serves as the modern successor to the deprecated Pickle-based Autoimport, offering better performance, multithreading support, and structured querying.
The system indexes both internal project files and external modules found in sys.path. It stores this metadata in an SQLite database, which can be persisted to disk within the .ropeproject folder or maintained entirely in memory for transient sessions. This allows Rope to perform complex searches, such as finding modules by name prefixes or matching module aliases, with high efficiency.
Sources: rope/contrib/autoimport/sqlite.py:65-72, rope/contrib/autoimport/pickle.py:1-12, CHANGELOG.md:195-200
Architecture and Components
The system is built around the AutoImport class, which manages database connections, cache generation, and search operations. It utilizes a separate models module to define the database schema and query logic, abstracting the SQL execution details.
Database Connection Management
The database connection can be established in two modes:
- On-Disk: Persisted in the project's
.ropeproject/autoimport.db. - In-Memory: Uses SQLite's URI filename syntax (
file:rope-<hash>:?mode=memory&cache=shared) to allow the database to be shared across multiple threads while remaining volatile.
To ensure thread safety, the AutoImport class uses threading.local() to maintain a unique SQLite connection per thread, as SQLite connections are generally not thread-safe for concurrent writes.
Sources: rope/contrib/autoimport/sqlite.py:128-164, CHANGELOG.md:46-50
Component Relationships
The following diagram illustrates how the AutoImport system interacts with the Rope project and the underlying storage.
Diagram shows the relationship between the main AutoImport class, the project, and the database storage options.
Sources: rope/contrib/autoimport/sqlite.py:100-140, rope/contrib/autoimport/sqlite.py:530-545
Data Model and Schema
The database consists of several tables designed to facilitate fast lookups of Python names and their corresponding modules.
Key Tables
| Table | Description | Derived From |
|---|---|---|
names | Stores global names, their containing modules, packages, and sources. | models.Name |
aliases | Stores module aliases (e.g., import numpy as np). | models.Alias |
packages | Tracks scanned packages to avoid redundant indexing. | models.Package |
metadata | Stores versioning information and cache creation timestamps. | models.Metadata |
Sources: rope/contrib/autoimport/sqlite.py:173-176, rope/contrib/autoimport/sqlite.py:348-355
Schema Relationships
Diagram showing the logical relationship between packages, names, and aliases within the SQLite schema.
Sources: rope/contrib/autoimport/sqlite.py:388-403, rope/contrib/autoimport/defs.py:100-135
Cache Generation and Synchronization
Cache generation is split into project resource indexing and external module indexing.
Process Flow
- Project Files: Scans the project's Python files using
ProcessPoolExecutorfor parallel parsing. - External Modules: Iterates through
sys.pathfolders to identify available packages. It uses a filtering mechanism to skip already indexed packages or those starting with underscores (unless configured otherwise). - Synchronization: The system uses
ResourceObserverto listen for file changes, moves, or removals, automatically updating the SQLite entries to keep the cache near-real-time.
Sources: rope/contrib/autoimport/sqlite.py:265-285, rope/contrib/autoimport/sqlite.py:302-320, rope/contrib/autoimport/sqlite.py:445-455
Search Logic
The system provides several search methods:
import_assist(starting): Returns names starting with a specific prefix.search(name, exact_match): Returns formatted import statements and module names.search_full(...): A comprehensive search across names, modules, and aliases.
Sequence of operations during a search request.
Sources: rope/contrib/autoimport/sqlite.py:186-215, rope/contrib/autoimport/sqlite.py:228-260
Configuration and Implementation Details
The SQLite backend introduces specific configuration parameters for performance and behavior.
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
memory | bool | True (Deprecated default) | Whether to use an in-memory database or persist to disk. |
observe | bool | True | Whether to listen for project changes to update the cache. |
underlined | bool | False | Whether to index names starting with an underscore. |
Sources: rope/contrib/autoimport/sqlite.py:100-115, rope/contrib/autoimport/sqlite.py:120-125
Versioning and Schema Migration
To prevent issues with outdated caches, the system calculates a version_hash based on the project state. During initialization, if the stored hash in the metadata table does not match the current project state, the cache is automatically cleared and regenerated.
Sources: rope/contrib/autoimport/sqlite.py:173-180, CHANGELOG.md:43-45
Conclusion
The Autoimport SQLite Database provides a robust and scalable infrastructure for name discovery in Python projects. By moving from a serialized pickle format to a structured relational database with multi-threaded indexing and shared memory support, it significantly improves the responsiveness of refactoring tools and IDE integrations within the Rope ecosystem.
Sources: rope/contrib/autoimport/sqlite.py:1-15, CHANGELOG.md:195-200