Voice Transcription Integration

Relevant source files

The following files were used as context for generating this wiki page:

• server/_core/voiceTranscription.ts
• server/storage.ts
• server/_core/llm.ts
• package.json
• app/(tabs)/analyze.tsx

The Voice Transcription Integration provides a backend service for converting audio into text. It leverages an internal, proxied Speech-to-Text service, specifically the whisper-1 model, accessed via the "Forge" API. The system is designed for a two-step client-side process: first, the client captures and uploads an audio file to a designated storage service, and second, it passes the resulting URL to the transcription endpoint for processing.

This module is self-contained on the server and exposes its functionality through a tRPC mutation. While the frontend UI for voice capture is not explicitly implemented in the provided files, the backend logic is fully defined, including error handling, data validation, and interaction with the external AI service.

Sources: server/_core/voiceTranscription.ts:1-19, 241-274

Architecture and Data Flow

The transcription process involves the client application, the application's backend server, a storage service, and the internal Forge AI service. The flow is initiated by the client.

This diagram illustrates the end-to-end data flow for transcribing an audio file.

Core Transcription Module

The central logic resides in the transcribeAudio function within server/_core/voiceTranscription.ts. This function orchestrates the entire transcription process on the server side.

Sources: server/_core/voiceTranscription.ts:70-184

transcribeAudio Function

This asynchronous function takes TranscribeOptions and returns either a TranscriptionResponse on success or a TranscriptionError on failure.

The process is as follows:

1. Configuration Validation: It first checks if the environment variables BUILT_IN_FORGE_API_URL (ENV.forgeApiUrl) and BUILT_IN_FORGE_API_KEY (ENV.forgeApiKey) are set.
2. Audio Download: The function fetches the audio file from the audioUrl provided in the options.
3. File Validation: It checks the downloaded audio file's size, enforcing a 16MB limit.
4. Form Data Preparation: A FormData object is created. The audio buffer is converted to a Blob and appended, along with the model name (whisper-1), response format (verbose_json), and a generated prompt.
5. API Call: A POST request is made to the Forge API's transcription endpoint (v1/audio/transcriptions). The request is authenticated using a Bearer token with the ENV.forgeApiKey.
6. Response Handling: If the API call is successful, the JSON response is parsed into a WhisperResponse object. The structure is validated to ensure it contains a text property before being returned.

Sources: server/_core/voiceTranscription.ts:70-184

This flowchart shows the internal logic of the transcribeAudio function.

Input Options (TranscribeOptions)

The transcribeAudio function accepts a single object with the following properties:

Parameter	Type	Description
audioUrl	string	URL to the audio file (e.g., from the storage service).
language	string	Optional: Specify language code (e.g., "en", "es").
prompt	string	Optional: A custom prompt to guide the transcription.

Sources: server/_core/voiceTranscription.ts:22-26

Error Handling

If the transcription fails at any stage, the function returns a TranscriptionError object.

Code	Description
SERVICE_ERROR	Service is not configured or returned an invalid response.
INVALID_FORMAT	Failed to download or process the audio file from the URL.
FILE_TOO_LARGE	The audio file exceeds the 16MB size limit.
TRANSCRIPTION_FAILED	The external transcription service failed to process the request.
UPLOAD_FAILED	Defined in the type, but not used in this specific function.

Sources: server/_core/voiceTranscription.ts:55-63, 80-92, 110-114, 155-160

Data Models

The module defines several types to structure the data for requests, responses, and errors.

API and Service Dependencies

Forge AI Service

The transcription relies on the internal "Forge" API, which acts as a proxy to various AI models.

• API Endpoint: The base URL is configured via ENV.forgeApiUrl (BUILT_IN_FORGE_API_URL). The specific path used is /v1/audio/transcriptions.
• Authentication: All requests to the Forge API are authenticated using a Bearer token from ENV.forgeApiKey (BUILT_IN_FORGE_API_KEY).
• Model: The service is hardcoded to use the whisper-1 model.

Sources: server/_core/voiceTranscription.ts:77-92, 132, 147-153, server/_core/llm.ts:162-165

Audio Storage (storage.ts)

A mandatory prerequisite is that the audio file must first be uploaded to a storage service to obtain a URL. The server/storage.ts file provides the storagePut function for this purpose. This function also uses the ENV.forgeApiUrl and ENV.forgeApiKey to interact with a proxied storage endpoint at /v1/storage/upload.

Sources: server/_core/voiceTranscription.ts:4-6, server/storage.ts:1-14, 71-88

Client-Side Dependencies

For the client to capture audio, the expo-audio package is included as a dependency, indicating the intended use of native device capabilities for recording.

Sources: package.json:34

tRPC Endpoint Implementation

The voiceTranscription.ts file provides a documented example of how to expose the transcribeAudio function as a tRPC mutation. This allows the frontend to call the service in a type-safe manner.

The example voiceRouter defines a transcribe mutation that:

1. Is a protectedProcedure, meaning it requires user authentication.
2. Accepts audioUrl, language, and prompt as input, validated with Zod.
3. Calls transcribeAudio with the input.
4. Checks for an error in the result and throws a TRPCError if one occurs.
5. Optionally saves the successful transcription to a database (this part is commented as an example).
6. Returns the successful TranscriptionResponse to the client.

This sequence diagram shows the client-server interaction via the tRPC mutation.