Multimodal Input & Files
Relevant source files
The following files were used as context for generating this wiki page:
Multimodal Input & Files enables users to interact with AI models using diverse data formats beyond plain text. The system manages file uploads, image processing, voice activity detection, and audio synthesis to support a rich conversational experience.
The architecture bridges Flutter's UI with native platform capabilities to perform resource-intensive tasks like image downsampling and real-time audio processing. It supports various providers, including Open WebUI, OpenAI-compatible endpoints, and Apple Intelligence.
Sources: README.md:144-149, README.md:37-67
Image Processing & Display
The system processes images using native bridges to ensure performance and memory efficiency. It downsamples large images before they reach the UI to prevent memory overflows.
Native Downsampling
The nativeDownsampleImageFile function creates a thumbnail from a file URL using a specific maxPixelSize. It uses CGImageSource to apply transformations and cache settings during the creation process.
The diagram shows the sequence for reducing image resolution natively before UI rendering.
Sources: ios/Runner/NativeSheetBridge.swift:26-44
Image Loading & Caching
The NativeSheetImageLoader manages asynchronous image fetching from URLs or Data URIs. It implements a thread-safe caching mechanism and prevents redundant network requests for identical assets.
| Component | Responsibility |
|---|---|
| Cache | Stores UIImage objects indexed by URLs or SHA256 fingerprints. |
| InFlightRequest | Tracks active downloads to deduplicate requests for the same image. |
| ProcessingQueue | Offloads image decoding and downsampling to a background thread. |
Sources: ios/Runner/NativeSheetBridge.swift:1690-1740, ios/Runner/NativeSheetBridge.swift:1768-1808
Voice & Audio Input
Conduit provides full voice-call modes and voice input using Voice Activity Detection (VAD). This allows the application to detect when a user starts and stops speaking automatically.
Voice Activity Detection (VAD)
The VAD system uses Silero models (v4 or v5) via ONNX Runtime to process audio frames. It operates on a fixed 16kHz sample rate.
The diagram illustrates how audio frames are analyzed to trigger speech events in the UI.
Sources: third_party/vad/UPSTREAM_README.md:371-395, third_party/vad/UPSTREAM_README.md:460-475
Audio Configuration
The system supports custom recording configurations for different platforms. On Android, it utilizes voiceCommunication audio sources to improve echo cancellation.
| Parameter | Default Value | Description |
|---|---|---|
sampleRate | 16000 | Fixed rate required by Silero models. |
frameSamples | 512 (v5) / 1536 (legacy) | Number of samples per processing frame. |
threshold | 0.5 | Probability required to trigger speech detection. |
Sources: third_party/vad/UPSTREAM_README.md:383-392, third_party/vad/UPSTREAM_README.md:580-598
Multimodal Output (TTS)
The text-to-speech (TTS) system provides live speech and file-based synthesis. It utilizes AVSpeechSynthesizer on iOS to generate audio from AI responses.
File Synthesis
The NativeIosTtsAudioFile class writes generated audio buffers to a local file in .caf format. This allows users to save or re-attach AI-generated speech as a file.
- Initialize: Creates an
AVSpeechUtterancefrom text. - Buffer Write: The synthesizer writes audio data into an
AVAudioPCMBuffer. - Append: The system writes the buffer into an
AVAudioFileon the local storage. - Complete: Returns the local path and total duration to the Flutter app.
Sources: ios/Runner/NativeIosTtsBridge.swift:319-354
File Management Dependencies
The multimodal system relies on several core libraries to handle different media types.
| Library | Functionality |
|---|---|
image_picker | Selects images from the gallery or camera. |
file_picker | Selects documents and other file types for upload. |
flutter_image_compress | Compresses images to reduce upload size. |
record | Captures raw audio for VAD and voice messages. |
pdfrx | Provides native rendering for PDF attachments. |
Sources: pubspec.yaml:70-75, pubspec.yaml:104-112
Summary
Multimodal Input & Files provides the infrastructure for image downsampling, voice activity detection, and audio synthesis. By leveraging native bridges, Conduit maintains high performance on mobile devices while processing large media files and real-time audio streams. The system integrates closely with platform-specific APIs like AVFoundation on iOS and AndroidAudioSource on Android to ensure compatibility and efficiency.