Backend Analysis Services Engine

Relevant source files

The following files were used as context for generating this wiki page:

• truelens-backend/app/tasks/ml_pipeline.py
• truelens-backend/app/services/document_analysis.py
• truelens-backend/app/services/news_analysis.py
• truelens-backend/app/services/review_analysis.py
• truelens-backend/app/services/live_fact_checker.py
• truelens-app/src/lib/analysis.ts

The Backend Analysis Services Engine is the core intelligence layer of TrueLens, responsible for evaluating the authenticity of various content types including text, URLs, and documents. It utilizes a multi-signal machine learning pipeline to detect AI-generated content, fake reviews, sensationalized news, and tampered documents.

The engine operates as an asynchronous pipeline, typically orchestrated via Celery tasks, allowing for deep analysis involving transformer models, OCR, and external web searches without blocking the main application flow. Sources: truelens-backend/app/tasks/ml_pipeline.py:1-18, truelens-app/src/lib/analysis.ts:167-175

Core ML Pipeline & Orchestration

The engine follows a multi-signal ensemble approach where individual specialized services contribute to a final weighted trust score. The primary entry point for this process is the process_scan_task.

Pipeline Data Flow

1. Input Categorization: The engine identifies if the input is a URL, raw text, or a file.
2. Scraping (if URL): For URLs, text is extracted using a custom TextExtractor based on HTMLParser.
3. Signal Generation: Concurrent analysis is performed across text ensemble, domain reputation, provenance, and specialized domain heuristics (news/reviews).
4. Score Aggregation: Signals are weighted and combined into a final verdict.

Sources: truelens-backend/app/tasks/ml_pipeline.py:116-160, truelens-backend/app/tasks/ml_pipeline.py:84-113

The diagram shows the parallel signal processing and weighting logic used to determine content authenticity. Sources: truelens-backend/app/tasks/ml_pipeline.py:43-73

Analysis Services

1. Document Analysis Service

This service detects forgeries in PDFs, images, and DOCX files. It combines metadata inspection with structural forensics.

• Tamper Detection: Checks for inconsistencies between creation and modification dates.
• Font Consistency: Analyzes embedded PDF fonts; more than 3-5 distinct font families often indicate copy-paste forgeries.
• Arithmetic Validation: For invoices, it extracts currency amounts and verifies if line items sum correctly to the total.
• Identity Verification: Uses the Verhoeff algorithm to validate 12-digit Aadhaar numbers.

Sources: truelens-backend/app/services/document_analysis.py:109-158, truelens-backend/app/services/document_analysis.py:236-281

2. News Integrity Analysis

Evaluates journalistic standards using a sensationalism index and transformer models (bert-tiny-finetuned-fake-news-detection).

• Sensationalism Index: Flags words like "bombshell," "shocking," or "exposed."
• Clickbait Detection: Scans the first 300 characters for phrases like "you won't believe" or "this one trick."
• Citation Check: Analyzes the presence of source attribution (e.g., "according to," "reported by").

Sources: truelens-backend/app/services/news_analysis.py:53-110

3. Review Authenticity Service

Detects fraudulent product or business reviews.

• Heuristics: Checks for extreme sentiment (1 or 5 stars), spammy capitalization, and low lexical diversity (keyword stuffing).
• ML Model: Utilizes nlptown/bert-base-multilingual-uncased-sentiment to identify unnatural sentiment patterns.

Sources: truelens-backend/app/services/review_analysis.py:44-88

4. Live Fact-Checking

Provides real-time verification of claims by searching external databases and the live web.

• Wikipedia Integration: Searches for claim keywords and extracts summaries.
• LLM Evaluation: Uses Google Gemini (Generative AI) to synthesize search results into a verdict (True, False, or Partially True).

Sources: truelens-backend/app/services/live_fact_checker.py:46-75, truelens-backend/app/services/live_fact_checker.py:113-146

Scoring & Weights

The engine calculates a trust_score (0-100), where 100 is most authentic. The score is derived by subtracting the weighted average of "fake probability" from 100.

Signal Type	Weight	Primary Purpose
Text	0.40	Transformer-based AI detection and stylometry.
Metadata	0.25	Domain WHOIS, TLD reputation, and EXIF data.
News	0.20	Journalistic integrity and fake news detection.
Image	0.20	ELA (Error Level Analysis) and GAN fingerprinting.
Review	0.10	Behavioral checks for paid/fake reviews.
Provenance	0.05	Historical presence (Wayback Machine, Google Cache).

Sources: truelens-backend/app/tasks/ml_pipeline.py:31-40, truelens-app/src/lib/analysis.ts:167-175

Implementation Example: Verhoeff Checksum

The backend implements the Verhoeff algorithm to identify tampered or invalid Aadhaar numbers during document scanning.

def _verhoeff_check(number: str) -> bool:
    try:
        c = 0
        for i, digit in enumerate(reversed(number)):
            c = _VERHOEFF_TABLE_D[c][_VERHOEFF_TABLE_P[i % 8][int(digit)]]
        return c == 0
    except (ValueError, IndexError):
        return False

Sources: truelens-backend/app/services/document_analysis.py:338-345

Summary

The Backend Analysis Services Engine serves as a comprehensive forensic suite. By combining deterministic heuristics (like checksums and arithmetic checks) with probabilistic ML models (like BERT-based classifiers), it provides a layered defense against digital misinformation and document fraud. Sources: truelens-backend/app/tasks/ml_pipeline.py:11-18, truelens-backend/app/services/document_analysis.py:11-20