OCR & Mistral Integration

Relevant source files

The following files were used as context for generating this wiki page:

• src/lib/pdf/mistral-ocr.ts
• src/lib/pdf/mistral-ocr-client.ts
• src/lib/ocr/mistral-batch.ts
• src/lib/ocr/mistral-direct.ts
• src/lib/ocr/config.ts
• src/app/api/pdf/ocr/route.ts
• src/lib/utils/format-workspace-context.ts

The OCR & Mistral Integration provides ThinkEx with the capability to transform static PDF documents and images into structured, machine-readable Markdown. By leveraging Mistral's advanced vision-language models, the system extracts not only plain text but also complex elements such as headers, footers, and tables, ensuring that the AI assistant can reason over the visual structure of uploaded media.

This integration serves as the backbone for the "See Everything" and "Capture Insights" features of the platform. It allows users to bring external documents into the spatial canvas where the AI can analyze specific pages or sections. The system supports both direct, real-time OCR requests for single files and batch processing for handling multiple candidates efficiently within a workspace.
Sources: README.md:19-25, src/lib/pdf/mistral-ocr.ts:5-8

Architecture and Execution Modes

The integration is architected to support two primary execution paths depending on the urgency and volume of the OCR tasks: Direct OCR and Batch OCR.

Direct OCR Flow

Direct OCR is used for immediate processing of single documents or images. It communicates directly with Mistral's /v1/ocr endpoint. This mode includes robust retry logic to handle rate limits and transient network failures.

The diagram above illustrates the synchronous path taken when a single file URL is submitted for immediate extraction.
Sources: src/app/api/pdf/ocr/route.ts:16-52, src/lib/pdf/mistral-ocr.ts:38-89, src/lib/pdf/mistral-ocr-client.ts:143-160

Batch OCR Flow

Batch OCR is designed for workspace-level operations where multiple items are processed simultaneously. It utilizes Mistral's batch job system, which involves creating a job, polling for completion, and downloading results in JSONL format.

Feature	Direct OCR	Batch OCR
Endpoint	/v1/ocr	/v1/batch/jobs
Concurrency	Single request	Multiple requests in one job
Response	Immediate (Synchronous)	Polling-based (Asynchronous)
Max Wait	Base Timeout	Up to 10 Minutes (MAX_POLL_MS)
Sources: src/lib/ocr/mistral-batch.ts:4-10, src/lib/ocr/config.ts:15-18

Technical Implementation

Configuration and Client

The OcrConfig interface manages the authentication and endpoint settings required for the integration. It defaults to the mistral-ocr-latest model.

export interface OcrConfig {
  apiBaseUrl: string;
  ocrEndpoint: string;
  batchJobsEndpoint: string;
  apiKey: string;
  model: string;
}

Sources: src/lib/ocr/config.ts:4-12

The callMistralOcr client implements a sophisticated retry mechanism. It distinguishes between standard timeouts and rate limits (429 errors), calculating delays using exponential backoff with jitter. For 429 errors, it attempts to parse the retry-after header or provider-specific error text to determine the optimal wait time.
Sources: src/lib/pdf/mistral-ocr-client.ts:88-121, src/lib/pdf/mistral-ocr-client.ts:199-234

Data Normalization

A critical component of the integration is the normalization of raw API responses into a consistent OcrPage structure. This involves mapping fields such as markdown, header, footer, and nested tables.

The mapping logic ensures that even if the API returns an optional index, the system maintains a 0-indexed sequence for workspace management.
Sources: src/lib/pdf/mistral-ocr-client.ts:11-20, src/lib/pdf/mistral-ocr-client.ts:44-57

Context Integration in Workspaces

Once extracted, OCR data is integrated into the AI assistant's context. The formatOcrPagesAsMarkdown utility converts the structured page objects back into a single Markdown string for the AI to read.

Table Placeholder Replacement

Mistral OCR often returns table placeholders in the format [id](https://github.com/ThinkEx-OSS/thinkex/blob/HEAD/id). The ThinkEx integration includes a replaceOcrPlaceholders function that scans the Markdown for these patterns and replaces them with the actual content found in the page's tables array.

// Example of placeholder replacement logic
out = out.replace(
  new RegExp(`\\[${escapeRegex(id)}\\]\\(${escapeRegex(id)}\\)`, "g"),
  `\n\n[Table ${id}]\n${content}\n\n`,
);

Sources: src/lib/utils/format-workspace-context.ts:285-300

AI Awareness

The system prompt explicitly informs the AI about its OCR capabilities and limitations. It specifies that while textual structure and inline tables are provided, visual elements like charts or diagrams require the user to provide separate screenshots.
Sources: src/lib/utils/format-workspace-context.ts:133-138

Error Handling and Resilience

The system provides detailed logging and error reporting for OCR failures.

• Validation: API routes validate file URLs via isAllowedOcrFileUrl and check for appropriate file extensions (e.g., .pdf) before dispatching.
• Retries: The direct client attempts processing up to MAX_ATTEMPTS (default 4).
• Status Reporting: Items in the workspace are marked with ocrStatus = "failed" and an ocrError message if extraction fails after all retries.
  Sources: src/app/api/pdf/ocr/route.ts:31-45, src/lib/pdf/mistral-ocr-client.ts:181-197, src/lib/utils/format-workspace-context.ts:390-395

Conclusion

The OCR & Mistral Integration is a sophisticated pipeline that bridges the gap between static media and the ThinkEx reasoning engine. By providing both direct and batch processing modes, the system ensures responsiveness for individual user actions while maintaining the ability to process large amounts of data for complex workspaces. The focus on structured Markdown extraction, including tables and headers, allows the platform to provide a highly contextual and accurate "spatial canvas" experience.