AI Orchestration & Assistant UI
Relevant source files
The following files were used as context for generating this wiki page:
ThinkEx implements a sophisticated AI orchestration layer that connects a spatial canvas workspace with multi-modal LLM capabilities. The system is designed to break the isolation of traditional chat threads by making context explicit, organized, and persistent through a specialized Assistant UI and backend orchestration.
The orchestration involves three primary pillars:
- Dynamic Context Assembly: Transforming workspace items (PDFs, documents, videos) into structured LLM prompts.
- Context-Aware Chat: A streaming API that integrates workspace metadata and user-selected cards.
- Interactive Selection: UI components that allow users to target specific regions of text or workspace items for AI analysis.
Sources: README.md:14-30, src/lib/utils/format-workspace-context.ts:114-140
Backend AI Orchestration
The orchestration backend is centered around the /api/chat and /api/workspaces/autogen endpoints, utilizing the Vercel AI SDK and Google's Gemini models.
Chat Request Pipeline
The chat route processes user messages by injecting workspace-specific context before sending them to the LLM. It standardizes model IDs, manages tool definitions, and prunes message history to optimize token usage.
The diagram above illustrates the sequential processing of a chat request, showing how selection context is merged into the user's message.
Sources: src/app/api/chat/route.ts:153-240
Workspace Context Formatting
The system converts complex workspace state into a format the LLM can interpret. This includes "Workspace FS" (File System) metadata and "Selected Cards" content.
| Component | Description | Format |
|---|---|---|
| Workspace FS | Metadata and paths of all content items in the workspace. | XML-wrapped <workspace> tags |
| Selected Cards | Full content or metadata of items the user has explicitly targeted. | XML-wrapped <context> tags |
| Active Focus | Flags for items currently being viewed or specific PDF pages. | (currently viewing) or activePage=N markers |
Sources: src/lib/utils/format-workspace-context.ts:57-80, src/lib/utils/format-workspace-context.ts:145-160
Assistant UI Components
The Assistant UI provides a specialized interface for interacting with the orchestration layer, emphasizing direct manipulation of context.
Text Selection and Multi-Mode
The AssistantTextSelectionManager enables a "Multi-Select Mode" where users can accumulate multiple highlights across different messages before performing an action like "Create Document" or "Reply".
This sequence shows the transition into multi-select mode, where the UI maintains a stable reference point using a hidden DOM marker.
Sources: src/components/assistant-ui/AssistantTextSelectionManager.tsx:376-415
Markdown and Citation Rendering
ThinkEx uses a custom Streamdown renderer to handle multi-modal outputs, including LaTeX math, Mermaid diagrams, and inline citations.
- Math: Supports single
$for inline and$$for block LaTeX. - Citations: Renders as interactive badges (e.g.,
<citation>Title | p. 5</citation>). Clicking a citation navigates the user to the specific workspace item or PDF page. - Mermaid: Dynamically renders architecture and flow diagrams within the chat.
Sources: src/components/assistant-ui/markdown-text.tsx:18-45, src/lib/utils/format-workspace-context.ts:188-210
Automated Workspace Generation (Autogen)
The autogen system orchestrates multiple AI phases to build a complete workspace from a single prompt or a set of files/links.
Multi-Phase Execution
- Search Phase: A lightweight model decides if a web search is needed to gather current context.
- Distillation Phase: Extracting a concise title, icon, color, and a 200-800 word content summary.
- Content Generation: Parallel generation of a study document, a quiz, and a YouTube video search.
- Bulk Creation: Final assembly of items onto the workspace canvas.
| Phase | Model Used | Output |
|---|---|---|
| Search Decision | gemini-2.5-flash-lite | Boolean + Search Query |
| Distillation | gemini-2.5-flash-lite | Workspace Metadata + Summary |
| Document & Quiz | gemini-2.5-flash | Markdown Document + JSON Quiz |
Sources: src/app/api/workspaces/autogen/route.ts:101-135, src/app/api/workspaces/autogen/route.ts:460-510
Tool Integration
The orchestration layer uses several tools to extend LLM capabilities:
- Web Search: Utilizes Google Search grounding to return synthesized findings and resolved source URLs. Sources: src/lib/ai/tools/web-search.ts:101-125
- Process URLs: Uses Google's URL Context API to analyze web pages and documentation directly. Sources: src/lib/ai/tools/process-urls.ts:20-40
- Workspace Search: Allows the AI to
readWorkspaceorsearchWorkspaceon-demand when full item content isn't in the immediate context window. Sources: src/lib/utils/format-workspace-context.ts:114-140
Conclusion
The AI Orchestration and Assistant UI in ThinkEx represent a shift from "chat-first" to "context-first" design. By orchestrating complex data flows between the spatial canvas and high-performance LLMs, the system ensures that AI reasoning is grounded in the user's explicit workspace context, enabling persistent knowledge compounding rather than isolated conversations.