Chat Interface & Compression
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Onyx Chat Interface provides a modular, real-time environment for users to interact with Gen-AI agents. It supports multi-turn conversations, tool-assisted search (web and internal), and rich content rendering including Markdown, LaTeX, and code blocks. The system architecture is designed to handle complex context management, including document citations and file uploads, while ensuring the Large Language Model (LLM) remains within its token limits through an automated compression system.
The compression system manages long chat histories by summarizing older messages while preserving recent context verbatim. This architecture ensures that the AI assistant maintains high performance and relevance even in extended sessions. It utilizes a branch-aware tree structure to store summaries, allowing for consistent history even across divergent conversation paths.
Sources: backend/onyx/chat/README.md:1-10, backend/onyx/chat/COMPRESSION.md:1-5
Chat Flow Architecture
The chat system is organized into three primary layers: the setup and validation layer (process_message), the turn management loop (run_llm_loop), and the individual inference step (run_llm_step).
Top-Level Processing
The process_message function serves as the entry point, handling database validation, history building, and tool preparation. It executes the LLM flow in a background thread using a wrapper that monitors for stop signals from Redis.
The Emitter is used as an object queue so lower levels can yield packets back to the interface without complex generator logic. The State Container accumulates tokens, reasoning, and tool call metadata for database persistence.
Sources: backend/onyx/chat/README.md:160-200
LLM Loop and Step Logic
The run_llm_loop manages the conversation turn by translating and truncating context, adding reminders, and executing tool calls. Each inference is a run_llm_step, which handles streaming packet translation and tracks reasoning vs. answer sections.
Sources: backend/onyx/chat/README.md:215-230
Chat History Compression
Compression is triggered when the token count of a chat history exceeds a defined portion of the LLM's context window.
Architecture Decisions
Onyx implements a branch-aware summary system using a tree structure. Summaries are stored as ChatMessage records.
- parent_message_id: Points to the tip of the branch when compression was triggered.
- last_summarized_message_id: A pointer to the cutoff point; all messages before this are summarized, and messages after are kept verbatim.
This design ensures that context from one branch does not bleed into another during multi-path conversations.
Sources: backend/onyx/chat/COMPRESSION.md:5-15
Progressive Summarization
To prevent information loss, subsequent compressions incorporate existing summary text alongside new messages. The LLM uses specific markers to identify the cutoff points.
| Prompt Marker | Usage |
|---|---|
<context_cutoff> | Marks the exact point where summarization must stop. |
CONTEXT_CUTOFF_END_MARKER | Closing tag for the cutoff region. |
SUMMARIZATION_PROMPT | Instructions to produce information-dense, paragraph-format summaries. |
Sources: backend/onyx/prompts/compression_prompts.py:5-25, backend/onyx/chat/COMPRESSION.md:17-20
Token Budgeting
Compression occurs based on the following logic:
- Trigger: History exceeds
max_context_tokens * COMPRESSION_TRIGGER_RATIO(default 0.75). - Preservation: The system keeps a portion of history verbatim based on
RECENT_MESSAGES_RATIO(default 0.2).
Sources: backend/onyx/chat/COMPRESSION.md:25-35
Interface Components & Rendering
The frontend renders AI and human messages with specific interactive elements and supporting sidebars for document management.
Message Rendering Features
The interface supports complex Markdown and math rendering. E2E tests verify the correct display of:
- LaTeX Math: Inline
\( ... \)and display\[ ... \]blocks. - Code Blocks: Syntax highlighting for various languages (e.g., Python).
- Tables: Formatted Markdown tables for feature overviews.
- Citations: Interactive links to documents referenced by the LLM.
Sources: web/tests/e2e/chat/chat_message_rendering.spec.ts:80-150
Document Sidebar
The DocumentsSidebar displays sources cited in the current AI response versus other relevant documents found during search.
The sidebar builds MinimalOnyxDocument objects from citations to allow users to view source blurbs and metadata.
Sources: web/src/sections/document-sidebar/DocumentsSidebar.tsx:100-150
File Interaction
Uploaded files are treated as "point in time" context. If a user clicks a file link in a message, a PreviewModal or TextViewModal is opened based on the file type (e.g., .txt, .py, .docx).
Sources: backend/onyx/chat/README.md:20-30, web/tests/e2e/chat/file_preview_modal.spec.ts:90-120
Context Management
Onyx prioritizes different types of context differently within the chat window to improve model performance:
- System Prompt: Default packaged prompt, dynamically updated with datetime and tool descriptions.
- Reminders: Specific instructions (like citation rules) placed at the very end of the chat context to take advantage of LLM recency bias.
- Project Files: If a project fits in context, it is kept near the end of history to ensure easy access for the LLM.
- Tool Call Responses: Long tool responses are often replaced with hardcoded summaries to save tokens, keeping only rich metadata like search queries.
Sources: backend/onyx/chat/README.md:1-15, backend/onyx/chat/README.md:75-90, backend/onyx/chat/README.md:100-110
Conclusion
The Onyx Chat Interface and Compression system provide a robust framework for enterprise-scale AI interactions. By combining a multi-layered backend flow with an intelligent, branch-aware compression strategy, Onyx maintains context accuracy across long and complex conversations. The frontend complements this with rich rendering and document-centric navigation, ensuring that users can easily verify AI claims through the citation and sidebar systems.