Han Shape-Based NLP Input
Relevant source files
The following files were used as context for generating this wiki page:
The Han Shape-Based NLP Input system is an experimental Natural Language Processing (NLP) module designed to provide shape-based Chinese character input (such as Cangjie or Wubi) within the keyboard. It functions as both a SuggestionProvider and a SpellingProvider, utilizing external SQLite databases bundled in language packs to map input codes to Han characters.
Sources: app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:35-40, app/src/main/res/values/strings.xml:135-136
Architecture and Components
The system is built around the HanShapeBasedLanguageProvider class, which interacts with the NlpManager to handle text suggestions and spell checking. The provider manages LanguagePackExtension objects which contain the necessary linguistic data.
Key Components
| Component | Description |
|---|---|
HanShapeBasedLanguageProvider | The primary implementation for shape-based Han input logic. |
NlpManager | Orchestrates various NLP providers, including Han shape-based and Latin providers. |
LanguagePackExtension | A dynamically importable extension grouping locale-relevant data like predictive models and SQLite databases. |
han.sqlite3 | The database file format used to store character mappings and weights. |
Sources: app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:42-60, app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/NlpManager.kt:66-70, ROADMAP.md:32-35
Implementation Diagram
The following diagram illustrates the relationship between the NLP Manager, the Han Provider, and the underlying data sources.
The NLP Manager routes input to the Han Provider, which queries a shape-based SQLite database to generate candidates.
Sources: app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:144-165, app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/NlpManager.kt:156-170
Data Flow and Suggestion Logic
The suggest function is the core of the input logic. When a user types a code corresponding to a Han character shape, the provider queries an active SQLite table.
- Context Extraction: The provider retrieves the current
EditorContent. - Language Pack Selection: It identifies the active
LanguagePackComponentbased on the keyboard subtype's locale. - Database Query: It performs a SQL query against the
han.sqlite3database using theLIKEoperator to find codes matching the user's composing text. - Result Ranking: Results are ordered by code (ascending) and weight (descending) to present the most likely characters first.
Sources: app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:144-160, app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:193-200
Database Schema
The SQLite databases used by the provider are generated from fcitx5 tables. The typical schema contains character codes, the resulting text, and priority weights.
| Field | Type | Description |
|---|---|---|
code | VARCHAR | The shape-based input code (e.g., "aaaa" for Wubi). |
text | TEXT | The Han character or phrase resulting from the code. |
weight | INT | Frequency or priority weight for ranking suggestions. |
stem | VARCHAR | Optional field used in specific tables (e.g., zhengma_large). |
Sources: utils/convert_fcitx5_sqlite.py:24-34
Composing and Input Determination
The provider implements determineLocalComposing to identify which part of the existing text should be considered "active" for Han character conversion. It uses a BreakIterator to traverse backwards from the cursor, checking characters against the allowed keyCode set defined in the language pack (typically 'a' through 'z').
The provider determines the composing range by validating backwards from the cursor position against the allowed key codes.
Sources: app/src/main/kotlin/dev/patrickgold/florisboard/ime/nlp/han/HanShapeBasedLanguageProvider.kt:233-257
Data Conversion Utility
The project includes a Python utility, convert_fcitx5_sqlite.py, to transform standard Chinese input tables into the SQLite format required by the Han Shape-Based NLP Input system. This utility parses fcitx5 table files, calculates relative weights based on entry order, and generates a han.sqlite3 database compatible with the Android provider.
Sources: utils/convert_fcitx5_sqlite.py:1-15, utils/convert_fcitx5_sqlite.py:175-185
Conclusion
The Han Shape-Based NLP Input module provides a specialized, data-driven approach to Chinese character entry. By offloading character mapping to SQLite databases within language pack extensions, it allows the keyboard to support complex input methods like Cangjie and Wubi without bloating the core application logic. This modular approach enables future expansion to other shape-based writing systems.