Celery Workers & Async Tasks
Relevant source files
The following files were used as context for generating this wiki page:
The TrueLens system utilizes an asynchronous task queue architecture powered by Celery and Redis to handle computationally intensive Machine Learning (ML) pipelines. This ensures that the user interface remains responsive while the backend performs multi-signal analysis on text, images, URLs, and documents.
The primary role of the worker system is to orchestrate the "TrueLens ML Pipeline," which aggregates scores from various specialized analysis services to produce a final authenticity verdict and trust score.
Sources: truelens-backend/app/tasks/ml_pipeline.py:1-15, truelens-backend/requirements.txt:18-20
Task Architecture & Orchestration
The core of the asynchronous system is the process_scan_task, which is registered as a Celery task. It manages the lifecycle of a scan, transitioning it through different statuses while invoking sub-services based on the input type (URL or raw text).
ML Pipeline Flow
When a scan is initiated, the worker updates the scan status to "processing" and begins a multi-stage evaluation:
- Input Identification: Detects if the input is a URL or raw text.
- Service Invocation: Calls specialized services such as domain analysis, image forensics, or transformer-based text ensembles.
- Heuristic Checks: Applies logic to determine if the content resembles reviews or news articles to trigger additional specific checks.
- Ensemble Scoring: Aggregates all signal scores into a weighted final trust score.
The diagram above illustrates the logical flow of the process_scan_task from initiation to database completion.
Sources: truelens-backend/app/tasks/ml_pipeline.py:114-190
Signal Weighing & Trust Score Calculation
The system uses a weighted average mechanism to calculate the final trust score. Each signal provides a "fake probability" score (0-100) and a confidence level (0-1). The final trust score is derived by subtracting the weighted average fake probability from 100.
Signal Weight Distribution
The following weights are applied to different analysis types within the pipeline:
| Signal Type | Weight | Description |
|---|---|---|
text | 0.40 | Transformer ensemble and stylometry analysis |
metadata | 0.25 | Domain WHOIS, TLD reputation, and EXIF signals |
image | 0.20 | ELA, GAN detection, and frequency domain analysis |
news | 0.20 | Specific checks for fake news indicators |
review | 0.10 | Behavioral checks for fake product/business reviews |
provenance | 0.05 | Historical context and Wayback Machine checks |
Sources: truelens-backend/app/tasks/ml_pipeline.py:34-42, truelens-app/src/lib/analysis.ts:241-247
Scoring Logic Implementation
# From truelens-backend/app/tasks/ml_pipeline.py:45-73
def _calculate_trust_score(signals: List[Dict[str, Any]]) -> int:
if not signals:
return 50
weighted_sum = 0.0
weight_total = 0.0
for signal in signals:
sig_type = signal.get("type", "text")
weight = SIGNAL_WEIGHTS.get(sig_type, 0.10)
confidence = signal.get("confidence", 0.5)
effective_weight = weight * confidence
weighted_sum += signal["score"] * effective_weight
weight_total += effective_weight
avg_fake_probability = weighted_sum / weight_total
return int(100 - avg_fake_probability)
Specialized Analysis Services
The Celery worker orchestrates several deep-analysis modules, often requiring specialized libraries defined in the project requirements.
Text & News Analysis
The worker utilizes RoBERTa and stylometry ensembles to detect AI-generated patterns. It also triggers analyze_news if the text contains keywords like "breaking," "update," or "journalist."
Sources: truelens-backend/app/tasks/ml_pipeline.py:126-141, truelens-backend/requirements.txt:30-31
Image Forensics
Image tasks involve three distinct forensic signals:
- Error Level Analysis (ELA): Detects compression inconsistencies.
- EXIF Analysis: Flags editing software signatures (e.g., Photoshop, GIMP) or stripped metadata.
- GAN Fingerprint Detection: Uses frequency-domain analysis (FFT) to find periodic spectral artifacts typical of AI generation.
Sources: truelens-backend/app/services/image_analysis.py:20-25, truelens-backend/app/services/image_analysis.py:380-415
Document Verification
The document analysis module performs OCR (via pytesseract) and PDF-specific integrity checks.
- Font Consistency: Multiple font families detected may indicate content spliced from different sources.
- Arithmetic Validation: For invoices, the system verifies if line items sum to the total amount.
- Aadhaar Validation: Uses the Verhoeff checksum algorithm to verify 12-digit identification numbers.
Sources: truelens-backend/app/services/document_analysis.py:100-145, truelens-backend/app/services/document_analysis.py:330-360
Task State Management
Workers interact with the database using SessionLocal to update the Scan and Signal models. The VerdictEnum determines the final classification presented to the user.
| Verdict | Trust Score Range | Description |
|---|---|---|
Authentic | 80 - 100 | Content shows high indicators of human/genuine origin. |
Suspicious | 40 - 79 | Mixed signals or minor forensic anomalies detected. |
AI-Generated | 0 - 39 | Strong indicators of AI generation or significant manipulation. |
Sources: truelens-backend/app/tasks/ml_pipeline.py:76-83, truelens-backend/app/models/models.py (inferred from pipeline logic)
The sequence above shows the interaction between the API, the asynchronous worker, and the persistent storage.
Sources: truelens-backend/app/tasks/ml_pipeline.py:114-190
Integration with External APIs
The workers also leverage live web searches to verify claims. The LiveFactChecker service uses the wikipedia library to gather context and google-generativeai (Gemini) to evaluate claims against that context in real-time.
Sources: truelens-backend/app/services/live_fact_checker.py:27-55, truelens-backend/app/services/live_fact_checker.py:75-100
Summary
The Celery Worker system in TrueLens is the engine for the multi-signal ML pipeline. By offloading heavy processing—such as ELA image forensics, transformer-based text analysis, and live fact-checking—to background workers, the system provides a robust and scalable architecture for content verification. All results are aggregated into a weighted ensemble score, providing users with an explainable trust verdict and detailed forensic signals.