Audio Translation Pipeline

The Audio Translation Pipeline is the core system responsible for capturing a user's voice in a Discord channel, processing it through multiple stages, and playing back a translated version. This automated, multi-step process is initiated when a user speaks and has translation enabled. The pipeline is designed as a sequence of independent processors that handle audio recording, speech-to-text, translation, and text-to-speech.

The entire process is event-driven, starting when the Discord bot's VoiceReceiver detects a user has started speaking. The pipeline is orchestrated within the joinCommandHandler, which sets up the listener. Each step generates an output that serves as the input for the next, forming a chain from raw voice data to a playable translated audio file.

Sources: src/commands/join.ts:100-117, README.md

High-Level Architecture

The pipeline integrates the main Node.js application with several external microservices for specialized tasks like Speech-to-Text (STT) and translation. The flow is linear, with each step being a prerequisite for the next.

This diagram illustrates the complete end-to-end flow of the audio translation pipeline.

Pipeline Stages

The pipeline is triggered by the speaking.on('start', ...) event listener set up after the bot joins a voice channel. This listener initiates the first stage for the speaking user.

Sources: src/commands/join.ts:100-102

1. Audio Recording

The first stage captures the user's voice. The recordAudio function subscribes to the VoiceReceiver for a specific user ID. It listens for a stream of Opus audio packets from Discord.

• Opus Decoding: An OpusDecodingStream is used to decode the incoming Opus packets into raw PCM audio data. It uses the opusscript library.
• File Writing: The decoded audio is piped to a wav.FileWriter, which saves it as a .wav file in the /cache/rec directory. The file is configured for mono audio at a 16000Hz sample rate.
• End of Speech: The recording automatically stops after 100ms of silence from the user, as defined by EndBehaviorType.AfterSilence.

The function returns the path to the newly created .wav file.

2. Speech-to-Text (STT)

The recognizeRecording function takes the path to the recorded .wav file and converts it to text.

• It reads the audio file into a buffer.
• It sends an HTTP POST request to the STT service, defined by the STT_HOST environment variable. The request body contains the raw audio data with a Content-Type of audio/wav.
• The STT service, a Rust application using the whisper-rs library, processes the audio and returns the transcribed text. By default, it translates the audio to English.
• The local .wav file is deleted after the transcription is received.

The function returns the transcribed text as a string.

This sequence diagram shows the interaction between the bot and the STT service.

3. Translation

The translate function is called if the user's target language is not English (en).

• It takes the transcribed text and a target language code.
• It sends an HTTP POST request to the translation service, defined by the TRANSLATION_HOST environment variable (typically a LibreTranslate instance).
• The request body is a JSON object containing the text (q), source language (source: 'en'), and target language (target).
• The service returns a JSON object with the translatedText.

The function returns the translated text string.

// Example request body for translate function
{
  "q": "Hello, how are you?",
  "source": "en",
  "target": "de"
}

Sources: src/processors/translate.ts:7-22, src/commands/join.ts:112-115, src/env.ts:17

4. Text-to-Speech (TTS)

The readText function converts the final text (either original or translated) into a playable audio file.

• It executes a Python script (src/lib/tts/read_text.py) using execa.
• It passes the text, the TTS model name corresponding to the target language, and an output file path as command-line arguments.
• The Python script uses the TTS library to generate a .wav file from the text using the specified model.
• The generated audio file is saved in the /cache/tts directory.

The function then passes the audio file path to the AudioQueue for playback.

Sources: src/processors/readText.ts:10-21, src/lib/tts/read_text.py

5. Audio Playback

Playback is managed by the AudioQueue class to prevent multiple audio streams from overlapping.

• Queueing: The readText function calls audioQueue.play(), which adds the path of the generated TTS audio file to a channel-specific queue.
• Sequential Playback: If nothing is currently playing in the channel, the queue immediately starts playing the first item. Otherwise, the audio waits its turn.
• Player Management: The AudioQueue uses a PlayerPool to manage AudioPlayer instances for each guild. When a player becomes idle (finishes playing a file), it automatically checks the queue for the next file to play.

This ensures that translations are spoken one after another in the order they were processed.

This diagram shows the structure of the AudioQueue class.

Configuration

The pipeline's external service dependencies are configured via environment variables.

These variables are typically set in a .env file and are used by the docker-compose.prod.yml file to link the bot container to the stt and libretranslate services.

Sources: src/env.ts:10-18, docker-compose.prod.yml:12-14

Summary

The Audio Translation Pipeline is a modular, event-driven system that seamlessly integrates the Discord bot with external AI services. It follows a clear, five-stage process: recording user audio, converting it to text, translating the text to a target language, synthesizing the translated text back into audio, and playing it back into the voice channel through a queuing system. This architecture allows for robust, real-time voice communication across different languages within a Discord environment.