Text Similarity and Matching Services
Relevant source files
The following files were used as context for generating this wiki page:
Text Similarity and Matching Services form the core intelligence layer of the application, responsible for automating transaction processing. These services normalize raw bank data and apply a variety of algorithms—ranging from fuzzy string matching and token overlap to LLM-powered categorization—to link transactions with subscriptions, categories, and potential duplicates.
The primary goal is to minimize manual user intervention by providing a standardized scoring mechanism for comparing text strings (descriptions, merchants, and names). This infrastructure supports the Smart Categorization and Subscription Tracking features mentioned in the project overview. Sources: backend/app/services/text_similarity.py:1-12, README.md
Text Similarity Core Logic
The TextSimilarity class provides a unified service for string comparison, primarily utilized for auto-linking transactions to subscriptions. It employs a multi-tiered scoring algorithm that prioritizes precise matches before falling back to more flexible methods.
Normalization and Tokenization
Before comparison, text is normalized by converting to lowercase, removing special characters (except spaces and hyphens), and collapsing whitespace. Tokenization filters out "noise words" (e.g., "payment", "transfer", "january", "incasso") to focus on unique merchant identifiers. Sources: backend/app/services/text_similarity.py:38-112
Scoring Hierarchy
Similarity is calculated using the following priority order:
| Priority | Method | Condition | Score |
|---|---|---|---|
| 1 | Exact Match | Normalized strings are identical | 100% |
| 2 | Substring Match | One string contains the other (min 3 chars) | 85% |
| 3 | Token Overlap | $\ge$ 50% of the smaller token set matches | 70-90% |
| 4 | Levenshtein | Fuzzy fallback based on edit distance | 0-70% |
The diagram shows the decision tree used by the calculate method to assign a similarity score. Sources: backend/app/services/text_similarity.py:126-193
Subscription Matching Service
The SubscriptionMatcher uses the core text similarity logic to link incoming bank transactions with active user subscriptions. It operates by caching active RecurringTransaction objects and comparing them against transaction fields.
Matching Logic
A transaction matches a subscription only if it passes a prerequisite amount check (within a 5% tolerance by default) and a text similarity threshold. The service compares four combinations:
- Subscription Name vs. Transaction Merchant
- Subscription Name vs. Transaction Description
- Subscription Merchant vs. Transaction Merchant
- Subscription Merchant vs. Transaction Description
Sources: backend/app/services/subscription_matcher.py:79-110, backend/app/services/text_similarity.py:195-224
Configuration Parameters
Matching behavior is controlled via environment variables:
| Variable | Description | Default |
|---|---|---|
SUBSCRIPTION_MIN_MATCH_SCORE | Minimum score (0-100) to auto-link | 60.0 |
SUBSCRIPTION_AMOUNT_TOLERANCE | Percentage difference allowed in amounts | 0.05 |
ENABLE_AUTO_SUBSCRIPTION_MATCHING | Global toggle for the service | true |
Sources: backend/app/services/subscription_matcher.py:33-41
Category Matching and LLM Fallback
The CategoryMatcher implements a two-tier approach for transaction categorization. It first attempts deterministic matching using predefined keyword rules and then falls back to an LLM-based suggestion if confidence is low.
Deterministic Rules
The system maintains a registry of keywords for categories like "groceries", "transport", and "utilities". Matches are filtered by transaction type (income vs. expense). For example, "uber" or "lyft" keywords trigger the "transport" category. Sources: backend/app/services/category_matcher.py:241-310
LLM Categorization
When rules fail, the system uses the gpt-4o-mini model (default) to analyze the transaction. The prompt includes available categories and user-specific guidelines to ensure accuracy. This service also supports batch processing to reduce API latency and costs. Sources: backend/app/services/category_matcher.py:378-430, backend/app/services/category_matcher.py:657-690
The sequence diagram illustrates the tiered lookup process where local rules are prioritized over external AI calls. Sources: backend/app/services/category_matcher.py:590-630
Frontend Duplicate Detection
During CSV imports, the application detects potential duplicates between new data and existing database records to prevent redundant entries.
Detection Criteria
The detectDuplicates utility identifies a match if:
- Date Match: Transactions occurred on the same UTC day.
- Amount Match: The absolute values of the amounts are equal (within a 0.01 tolerance).
- Fuzzy Description: A Levenshtein-based similarity score exceeds 0.85 (85%).
Sources: frontend/lib/utils/duplicate-detection.ts:68-120
Deterministic ID Generation
To ensure consistency, the system generates a "deterministic external_id" for CSV imports. This hash is derived from the account ID, date, amount, description, and merchant. If a duplicate is detected in a single batch, a counter is appended to the hash. Sources: frontend/lib/actions/csv-import.ts:553-596
Summary of Matching Components
| Component | Logic Source | Primary Use Case |
|---|---|---|
TextSimilarity | Token Overlap / Levenshtein | Foundation for all string comparisons |
SubscriptionMatcher | Similarity + Amount Check | Linking transactions to recurring payments |
CategoryMatcher | Keywords + LLM | Assigning spending categories to transactions |
DuplicateDetection | Exact Date/Amount + Fuzzy Desc | Cleaning CSV import data |
MerchantExtractor | Regex/NLP | Extracting vendor names from raw descriptions |
Sources: backend/app/services/text_similarity.py, backend/app/services/subscription_matcher.py, backend/app/services/category_matcher.py, frontend/lib/utils/duplicate-detection.ts
The Text Similarity and Matching Services provide a robust framework for financial data enrichment. By combining efficient local algorithms with powerful LLM fallbacks and strict duplicate prevention, the system ensures high data quality with minimal manual overhead.