LLM Chunking & Progressive Streaming

Relevant source files

The following files were used as context for generating this wiki page:

• backend/config/prompts.json
• backend/lib/openrouter.ts
• tasks/lessons.md
• tasks/todo.md
• extension/src/background/index.ts
• CHANGELOG.md

LLM Chunking and Progressive Streaming is a core architectural feature of Oddity 1 that manages long-form text processing and real-time annotation delivery. The system divides extensive prose into manageable segments and uses Server-Sent Events (SSE) to stream AI-generated insights to the user interface as they are created.

This system ensures that the application remains responsive even when processing documents exceeding the output token limits of Large Language Models (LLMs). It integrates with site adapters for ChatGPT and Claude to provide real-time support during active chat sessions.
Sources: README.md, CHANGELOG.md:23-28, tasks/todo.md:73-77

Text Chunking Logic

The backend implements chunking to prevent LLM truncation and improve annotation diversity. The system targets a specific word count per chunk to ensure the model does not "stop early" or become "lazy" due to excessive input length.

Chunking Rules and Targets

The system uses the following configuration for dividing input prose:

• Target Size: 250–350 words per chunk.
• Minimum Size: 150 words per chunk.
• Contiguity: The end of one chunk must touch the start of the next with zero gap to ensure total coverage.
• Structural Boundaries: Cuts occur at natural breaks like topic shifts or new arguments.

Sources: backend/config/prompts.json:44-53, tasks/lessons.md:71-73

Handling Input Diversity

The system employs multiple strategies to identify chunk boundaries, especially when standard paragraph markers are missing:

1. Double newlines (\n\n+).
2. Single newlines (\n).
3. Sentence boundaries using lookahead regex.
4. Hard word-count splits as a final fallback.

Sources: tasks/lessons.md:104-106

Progressive Streaming Architecture

Oddity 1 utilizes Server-Sent Events (SSE) to stream annotations from the OpenRouter API to the browser extension. This prevents the user from waiting for a full batch of annotations to complete before seeing results.

Streaming Data Flow

The following diagram illustrates the flow from the extension background script to the LLM and back to the content script.

The Service Worker acts as a proxy, parsing incremental JSON fragments and forwarding valid annotation objects to the active tab.
Sources: backend/lib/openrouter.ts:468-508, extension/src/background/index.ts:241-262

Reliability and Fallbacks

To handle network or parsing failures during a stream, the system implements a "Buffered Fallback":

• If a stream fails to parse or is interrupted, the system automatically falls back to a non-streaming batch request.
• The usedBufferedFallback flag in StreamResult indicates when this recovery path is taken.
• Identities of annotations are checked to prevent duplicate rendering if the final batch contains items already streamed.

Sources: backend/lib/openrouter.ts:553-585, tasks/lessons.md:154-162

Prompt Engineering for Chunking

The overview_prompt_template contains specific instructions to guide the LLM's chunking behavior. These instructions are critical for maintaining structural integrity across a fragmented text.

Instruction Set

Field	Requirement	Description
chunk_start	8–15 words	Exact verbatim phrase from the beginning of the chunk.
chunk_end	8–15 words	Exact verbatim phrase from the end of the chunk.
anchor	8–25 words	Central or important sentence within the chunk.
TOTAL COVERAGE	Rule	Union of all chunks must equal the full text.

Sources: backend/config/prompts.json:28-40, backend/config/prompts.json:48-52

Contextual Awareness

For chunked inputs, the system injects section context (e.g., "[This is section 2 of 4]") into the prompt. This prevents the model from assuming it has reached the end of the document prematurely and encourages consistent annotation density.
Sources: tasks/lessons.md:124-126

Implementation Details

Key Functions

• requestAnnotationsStreaming: Initiates the SSE connection and handles incremental JSON extraction via extractCompleteObjects.
• mapLlmOutput: Normalizes the raw LLM JSON into the internal Annotation schema, assigning unique IDs immediately to prevent collisions across regions.
• acquireSlot: A concurrency limiter in the background script that prevents a tab from overwhelming the API with simultaneous chunk requests.

Sources: backend/lib/openrouter.ts:742-788, backend/lib/openrouter.ts:208-232, extension/src/background/index.ts:43-52

Configuration Parameters

The following environment variables in openrouter.ts control the performance of the streaming pipeline:

• OPENROUTER_TIMEOUT_MS: Defaults to 45,000ms.
• OPENROUTER_RETRY_BASE_DELAY_MS: Initial retry backoff of 250ms.
• OPENROUTER_REASONING_EFFORT: Set to "medium" to balance speed and quality.

Sources: backend/lib/openrouter.ts:39-55

Conclusion

The LLM Chunking & Progressive Streaming system is designed to provide high-quality, real-time annotations for documents of arbitrary length. By combining surgical text splitting with incremental SSE parsing, Oddity 1 avoids common LLM limitations such as output truncation and context loss, resulting in a responsive and robust reading experience.