LLM Provider Integration

Relevant source files

The following files were used as context for generating this wiki page:

• spec/11-llm-integration.md
• plans/completed/2026-02-llm-provider-integration.md
• spec/adr/011-llm-cloud-and-local-providers.md
• plans/active/2026-05-ai-setup-ux.md
• plans/completed/2026-02-llm-settings-state-refactor.md

The LLM Provider Integration system allows MacParakeet to deliver advanced text-based features such as transcript summarization, chat-with-transcript, and AI-powered text transforms. The system follows a "Bring Your Own Provider" (BYOP) philosophy, where the application provides the orchestration layer but does not bundle a local LLM runtime or model, preserving the app's local-first and lightweight posture.

The architecture is designed around a single shared service layer that routes requests to various external endpoints. These include cloud APIs (Anthropic, OpenAI, Gemini), local runtimes (Ollama, LM Studio), and command-line tools (Claude Code, Codex). While the core speech-to-text (STT) processing remains strictly on-device, transcript text is sent to the user-configured provider only when an explicit AI action is triggered.

Sources: spec/11-llm-integration.md, spec/adr/011-llm-cloud-and-local-providers.md

Architectural Overview

The integration utilizes a multi-layered approach to resolve and execute LLM requests. The high-level LLMService builds prompts with transcript context and passes them to the RoutingLLMClient. This router determines whether to dispatch the request to a local subprocess (via LocalCLILLMClient) or a standard HTTP endpoint (via LLMClient).

Request Flow

The following diagram illustrates how a user action triggers the processing pipeline:

This diagram shows the routing logic from the high-level service to specific transport implementations.
Sources: spec/11-llm-integration.md, spec/adr/011-llm-cloud-and-local-providers.md

Provider Management

MacParakeet supports several categories of providers, each with specific authentication and transport requirements. Provider configurations (ID, base URL, and model name) are stored in UserDefaults, while sensitive API keys are stored securely in the macOS Keychain using KeychainKeyValueStore.

Supported Provider Types

Provider ID	Type	Default Base URL	Auth Mechanism
anthropic	Cloud	https://api.anthropic.com/v1/	Authorization: Bearer
openai	Cloud	https://api.openai.com/v1	Authorization: Bearer
ollama	Local	http://localhost:11434/v1	Bearer ollama (Placeholder)
lmstudio	Local	http://localhost:1234/v1	Optional API Key
localCLI	CLI	N/A	Subprocess Auth
gemini	Cloud	https://generativelanguage...	Authorization: Bearer

Sources: spec/11-llm-integration.md, plans/completed/2026-02-llm-provider-integration.md

Readiness State Model

To simplify UI logic, the system utilizes an AIReadinessState to track the configuration and health of the AI integration.

Core Components and Logic

Context Assembly and Truncation

Because different providers have varying context window limits, the LLMService manages context budgets. The system uses a "truncate from the middle" strategy to keep transcripts within limits while preserving the beginning and end of the conversation.

• Cloud Providers: Budgeted at approximately 500,000 characters.
• Local Runtimes: Budgeted at 80,000 characters.
• LM Studio: Specifically limited to 8,000 characters due to effective context depending on the specific model loaded.

Sources: spec/11-llm-integration.md, plans/completed/2026-02-llm-provider-integration.md

Streaming Implementations

Streaming is handled in a provider-specific manner behind a unified AsyncThrowingStream<String, Error> interface:

• Anthropic: Parses native Messages API event frames.
• OpenAI-Compatible: Parses SSE data: lines, terminating on [DONE].
• Ollama: Parses NDJSON (Newline Delimited JSON) chat chunks.
• Local CLI: Yields stdout incrementally as the subprocess executes.

Sources: spec/11-llm-integration.md, plans/completed/2026-02-llm-provider-integration.md

Local CLI Executor

The LocalCLILLMClient executes tools like claude -p or codex exec using posix_spawn. It delivers prompts via standard input (stdin) and utilizes environment variables prefixed with MACPARAKEET_ to pass context to the tool. This allows advanced users to leverage existing command-line AI tools without needing separate API keys.

Sources: spec/11-llm-integration.md, spec/adr/011-llm-cloud-and-local-providers.md

UI and UX Strategy

The AI setup experience is designed to be low-friction, prioritizing local apps.

Setup Paths

1. Local AI App (Recommended): Recommends LM Studio (port 1234) or Ollama (port 11434). The app performs speculative detection on these ports to offer "One-Click Adoption."
2. API Key: Standard path for Claude, OpenAI, and Gemini.
3. Command-Line Tool: Path for localCLI users.

AI Settings State Refactor

The settings UI utilizes a "Draft" pattern to separate editable UI state from the persisted configuration. This prevents race conditions during async connection tests.

Summary of LLM Provider Integration

The LLM Provider Integration enables sophisticated AI features while maintaining MacParakeet’s commitment to privacy and performance. By abstracting the transport layer, the system supports a wide range of providers from high-performance cloud APIs to private local runtimes. The architecture ensures that audio data never leaves the device, and transcript text is only transmitted through secure, user-authorized channels. Through features like middle-truncation and provider-specific SSE parsing, the integration provides a seamless experience for summarization, chat, and text transformation.