Natural Language Processing (Persian Offline)
Relevant source files
The following files were used as context for generating this wiki page:
The Natural Language Processing (NLP) system in Hesabyar is an offline-first engine designed to parse financial transactions from natural Persian sentences. It serves as the primary mechanism for quick data entry, allowing users to register expenses, income, loans, and installments using conversational text without requiring an internet connection. Sources: README.md, rust/hesabyar-core/README.md
The system is implemented in the Rust Core (hesabyar-core) and exposed to the Android application via UniFFI bindings. It combines rule-based heuristics, extensive keyword mapping (200+ keywords), and text normalization to achieve high accuracy in entity extraction, including amounts, dates, categories, and person names. Sources: rust/hesabyar-core/src/parser/nlp.rs:55, rust/hesabyar-core/src/lib.rs:37-40
Architecture and Pipeline
The NLP pipeline follows a linear progression from raw text input to a structured ParsedResult object. The process begins with normalization and moves through specialized detectors for different financial entities.
Data Flow Overview
The diagram shows the transition from unstructured input to the structured result used by the Android UI. Sources: rust/hesabyar-core/src/parser/mod.rs, app/src/main/java/io/github/mojri/hesabyar/rust/RustBridge.kt:71-72
Text Preprocessing
The preprocessing layer standardizes Persian text to ensure consistent parsing. It handles character variations, script-specific punctuation, and digit normalization. Sources: rust/hesabyar-core/src/parser/text_preprocessor.rs:1-13
| Function | Responsibility | Handled Characters |
|---|---|---|
preprocess_persian_text | Normalizes punctuation and spacing | \u{060C} (,), \u{066B} (.), \u{200C} (ZWNJ → space) |
to_ascii_digits | Converts Persian/Arabic numerals to ASCII | ۰-۹ and ٠-٩ → 0-9 |
normalize_money_text | Cleans thousands separators | \u{066C}, , |
Sources: rust/hesabyar-core/src/parser/text_preprocessor.rs:4-36
Entity Extraction Modules
1. Amount Parsing
The system extracts numeric values and interprets Persian multipliers. It specifically handles the Toman-to-Rial conversion, where the core logic operates on Rial (Toman * 10). Sources: rust/hesabyar-core/src/parser/nlp.rs:429-430
2. Category Inference
The system uses a list of 200+ Persian keywords to map sentences to one of the 8 core database categories. Sources: rust/hesabyar-core/src/parser/nlp.rs:55-144
3. Date and Time Extraction
The engine calculates date offsets (e.g., "Yesterday", "Tomorrow") and extracts specific times or Jalali calendar days. Sources: rust/hesabyar-core/src/parser/nlp.rs:317-360
- Date Offsets: Maps "امروز" (0), "دیروز" (-1), "فردا" (1), and "پسفردا" (2).
- Time: Detects patterns like "ساعت ۱۴" or "ساعت ۲ و نیم عصر" (converting to 24h format).
- Jalali Days: Parses specific month names (e.g., "فروردین", "اردیبهشت") to calculate days from the current date. Sources: rust/hesabyar-core/src/parser/nlp.rs:318-323, rust/hesabyar-core/src/parser/nlp.rs:381-401
Classification Logic
The classify_type function determines the transaction type by checking for specific verbs and keywords. Sources: rust/hesabyar-core/src/parser/nlp.rs:209-215
| Transaction Type | Trigger Keywords/Logic |
|---|---|
EXPENSE | Default classification if no income/loan/installment keywords found. |
INCOME | Keywords like "حقوق", "پاداش", "دستمزد", "فروش". |
LOAN_CREDITOR | Patterns like "قرض گرفتم", "بدهکار شدم", "گرفتم از". |
LOAN_DEBTOR | Patterns like "قرض دادم", "طلبکار شدم", "دادم به". |
INSTALLMENT | Presence of "قسط" without payment verbs like "تسویه" or "پرداخت". |
Sources: rust/hesabyar-core/src/parser/nlp.rs:218-283
Confidence Scoring
The system calculates a confidence level for each parse based on the number of successfully extracted "factors." Sources: rust/hesabyar-core/src/parser/nlp.rs:462-475
AI Result Sanitization
While the offline parser is rule-based, the system also includes an AiParsedTransaction module to validate and repair results returned by online LLM providers (like Gemini). This ensures that AI-generated JSON adheres to the project's strict financial constraints. Sources: rust/hesabyar-core/src/ai_validation.rs:17-22
- Clamping: Dates are clamped to $\pm365$ days, and confidence is clamped between $0.0$ and $1.0$.
- Category Normalization: Maps fine-grained AI categories (e.g., "Personal Care") to core DB categories (e.g., "Shopping").
- Repair Logging: Tracks which fields were coerced to safe defaults in
repair_notes.
Sources: rust/hesabyar-core/src/ai_validation.rs:60-100, rust/hesabyar-core/src/ai_validation.rs:125-150
Summary
The Persian Offline NLP module provides a high-performance, privacy-respecting way to input financial data. By offloading logic to a Rust core, the system ensures that complex string manipulations and keyword matching do not block the Android UI thread, while maintaining consistency across potential future platforms. Sources: README.md, app/src/main/java/io/github/mojri/hesabyar/rust/RustBridge.kt:20-25