Image-to-Molecule OCR Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Image-to-Molecule OCR Integration is a feature within the cheMD platform designed to streamline the digitization of chemical structures. It allows users to upload or paste images (such as screenshots from papers or hand-drawn sketches) and automatically convert them into structured molecule blocks within a Chemd document.
This integration bridges the gap between raster graphics and the structured semantic layer of the application. By leveraging Optical Character Recognition (OCR) tailored for chemistry, the system extracts SMILES strings from images and either inserts new blocks or updates existing ones in the Markdown-based source.
Sources: apps/web/src/app/page.tsx:112-132, apps/web/src/features/ocr/hooks/useImageOcr.ts:34-45
Architecture and Data Flow
The OCR system follows a clear pipeline from the user interface to the backend API and back to the document editor.
Input Mechanisms
Users can trigger the OCR process through two primary methods:
- File Selection: An
OcrImportButtonallows manual file picking. - Clipboard Monitoring: An
OcrPasteListenermonitors global paste events on the page, specifically looking forimage/*MIME types.
Sources: apps/web/src/app/page.tsx:167-168, apps/web/src/features/ocr/components/OcrPasteListener.tsx:10-25
Core Processing Pipeline
The following sequence diagram illustrates the flow from an image paste event to the document update:
The workflow ensures that the image is processed asynchronously, providing the user with a "loading" state before the Markdown source is automatically modified.
Sources: apps/web/src/features/ocr/hooks/useImageOcr.ts:47-92, apps/web/src/features/ocr/components/OcrPasteListener.tsx:10-25
Components and Hooks
useImageOcr Hook
The central logic resides in the useImageOcr custom hook. It manages the state of the OCR process, including loading indicators and error messages.
| Property | Type | Description |
|---|---|---|
loading | boolean | Indicates if an OCR request is currently in flight. |
error | string | null | Contains error messages if the API request or processing fails. |
runOcr | Function | The primary method to send a File object to the backend. |
The hook accepts a sessionId and documentId to maintain context and uses a getLatestSource callback to ensure it modifies the current state of the document.
Sources: apps/web/src/features/ocr/hooks/useImageOcr.ts:25-45
OcrPasteListener Component
This is a headless component that attaches a paste event listener to the window. It filters for image files in the clipboard and triggers the provided onPickFile callback.
Sources: apps/web/src/features/ocr/components/OcrPasteListener.tsx:1-30
Backend API Integration
The frontend communicates with a dedicated API endpoint at /api/chem/ocr.
Request Structure
The request is a POST method using multipart/form-data.
- file: The image blob.
- documentId: The ID of the current document (derived from the source).
- sessionId: The current structure session ID.
Response Schema
The API returns a JSON object that dictates how the frontend should modify the Markdown source.
{
"smiles": "CC(=O)O",
"blockId": "mol-123",
"action": "create_new" | "update_existing"
}
Sources: apps/web/src/features/ocr/hooks/useImageOcr.ts:60-80, apps/web/src/app/api/chem/ocr/route.ts
Document Modification Logic
Once the SMILES string is retrieved, the system uses internal library functions to manipulate the Markdown string.
Insertion vs. Updating
- Insert Molecule: If the
actioniscreate_new,insertMoleculeBlockis called to append a new:::moleculeblock to the bottom of the source. - Update Molecule: If the
actionisupdate_existing,updateMoleculeBlocklocates the existing block by its#IDand replaces itssmilesfield.
// Example of the resulting Markdown block
const nextSource = insertMoleculeBlock(latestSource, blockId, smiles);
/* Result:
:::molecule #mol-123
smiles: CC(=O)O
:::
*/
Sources: apps/web/src/features/ocr/lib/insert-molecule-block.ts, apps/web/src/features/ocr/lib/update-molecule-block.ts, apps/web/src/app/page.tsx:125-132
Summary
The Image-to-Molecule OCR Integration provides a seamless UX for importing chemical data. By combining event-driven listeners (paste/upload) with a structured API response, it automates the creation of semantic chemical data blocks, ensuring that cheMD documents remain the "source of truth" for experimental data even when starting from graphical inputs.
Sources: AGENTS.md:12-15, apps/web/src/app/page.tsx:185-190