PDF Extraction Workflow
Relevant source files
The following files were used as context for generating this wiki page:
The PDF Extraction Workflow automates the transformation of the GestaltView_Master_Wiki_v4.0.pdf into a source-traceable, offline static visual story. The process identifies, crops, and extracts 54 distinct visuals, including 13 curated diagrams and 39 captioned figures, alongside rendering embedded Mermaid snippets into image assets.
The system utilizes PyMuPDF for PDF parsing and coordinate-based extraction, ensuring that each visual remains tied to its original page reference and context within the source document.
Sources: build_story_artifact.py:2-3, README.md:1-5, diagram_inventory.json:5-10
Pipeline Architecture
The extraction pipeline operates through three primary phases: document analysis, asset generation, and artifact compilation. The build_story_artifact.py script serves as the orchestrator for this workflow.
Extraction Flowchart
This diagram illustrates the sequence of operations from the raw PDF input to the final self-contained HTML artifact.
Core Extraction Modules
1. Curated Diagram Extraction
The workflow targets 13 specific "Curated" diagrams defined in the CURATED mapping. These represent high-level architectural views and are extracted based on specific section heading numbers found within the first ten pages of the PDF.
- Targeting: It looks for headings formatted as
number. Space(e.g., "1. GestaltView at a Glance"). - Cropping: The script calculates a rectangular area starting from the heading and ending just before the next detected section.
Sources: build_story_artifact.py:16-30, build_story_artifact.py:228-255
2. Captioned Figure Detection
Automated figure detection identifies diagrams throughout the 116-page document. It uses the make_figure_items function to scan for the regular expression Figure [\d.]+ :.
| Step | Action | Description |
|---|---|---|
| 1 | Scan Blocks | Retrieves all text blocks from a page. |
| 2 | Regex Match | Identifies figure labels and their associated titles. |
| 3 | Context Mapping | Finds the H2 or H3 heading immediately preceding the figure. |
| 4 | Coordinate Calculation | Sets crop boundaries using the heading top and the caption bottom. |
| Sources: build_story_artifact.py:263-290 |
3. Mermaid Rendering
While most diagrams are extracted as crops, the workflow identifies raw Mermaid source blocks on specific pages (e.g., page 78). The render_mermaid_items function writes these to temporary .mmd files and uses the manus-render-diagram utility to produce PNG assets.
Sources: build_story_artifact.py:316-324, diagram_inventory.json:13-17
Technical Implementation Details
Text Normalization
PDF extraction often produces artifacts such as broken ligatures or inconsistent spacing. The norm(text) function cleans extracted strings using regular expressions.
def norm(text: str) -> str:
text = text or ""
# Fix common PDF ligature breaks
text = re.sub(r"\bfi\s+les\b", "files", text, flags=re.I)
text = re.sub(r"\b([A-Za-z]+)\s+(fi|ffi)\s+([A-Za-z]+)\b", r"\1\2\3", text, flags=re.I)
return re.sub(r"\s+", " ", text).strip()
Sources: build_story_artifact.py:177-183
Image Processing and Persistence
The system uses a scale matrix of 2.4 during extraction to ensure high visual fidelity for the resulting JPEGs. These images are saved to the story_assets/ directory before being converted to Base64 data URIs for the final artifact.
- Library:
pymupdf(fitz) for cropping. - Scale:
pymupdf.Matrix(2.4, 2.4). - Optimization: PIL saves JPEGs with a quality setting of 91 and progressive encoding.
Sources: build_story_artifact.py:203-207, build_story_artifact.py:347-353
Configuration and Dependencies
The workflow requires specific Python libraries and environment setups to function.
| Dependency | Purpose | Source |
|---|---|---|
pymupdf | PDF parsing and vector-to-raster extraction. | requirements-build.txt |
Pillow (PIL) | Image optimization and format handling. | requirements-build.txt |
manus-render-diagram | External utility for Mermaid rendering. | README.md:38 |
story_template.html | The skeletal UI used to house extracted data. | build_story_artifact.py:11 |
Summary
The PDF Extraction Workflow bridges the gap between static documentation and interactive analysis by programmatically decomposing the GestaltView Master Wiki. By combining coordinate-based cropping with automated text normalization, the system generates a self-contained, searchable visual atlas that preserves the provenance of every architectural claim found in the source document.
Sources: README.md:15-20, diagram_inventory.json:11-15