Audio Generation & Formats
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Generation & Formats system within Wikispeech-server is responsible for transforming processed text or SSML markup into audible speech. It serves as a middle layer that coordinates between various Text-to-Speech (TTS) engines and the end-user, handling the conversion of raw synthesizer output into standardized formats like Opus.
This module encompasses the logic for communicating with external TTS services (such as MaryTTS, Piper, and Matcha), managing temporary audio storage, and performing format transcoding to ensure compatibility across web platforms.
Core Architecture and Data Flow
The system operates via a series of adapters that normalize communication with different synthesis engines. When a synthesis request is received, the server identifies the appropriate voice and its associated adapter to generate a temporary WAV file, which is then transcoded into a compressed format for delivery.
Synthesis Process Flow
The following diagram illustrates the high-level flow from a synthesis request to the delivery of audio data.
The system relies on opusenc for converting raw audio to the Opus format. Sources: wikispeech_server/wikispeech.py:102-111, wikispeech_server/wikispeech.py:657-672
Audio Formats and Transcoding
The primary internal format generated by most adapters is the WAVE file. However, for efficient web delivery and storage, the server primarily targets the Opus format.
Transcoding Logic
The saveAndConvertAudio function in wikispeech.py manages the lifecycle of an audio file:
- Storage: Synthesized audio is initially saved to a temporary directory defined in the configuration (default:
./wikispeech_server/tmp). Sources: README.md:43-44, wikispeech_server/wikispeech.py:1016-1017 - Conversion: The system executes the
opusenccommand-line tool to transform WAV files into Opus. Sources: wikispeech_server/wikispeech.py:1041-1046 - Encoding: For JSON responses, the resulting Opus file is encoded into a Base64 string. Sources: wikispeech_server/wikispeech.py:1052-1055
- Cleanup: Temporary WAV files are unlinked after conversion to manage disk space. Sources: wikispeech_server/wikispeech.py:1066
Supported Format Specifications
| Format | Usage | Tooling |
|---|---|---|
| WAV | Intermediate synthesizer output | Internal adapters |
| Opus | Standardized output format for delivery | opusenc (opus-tools) |
| Base64 | Transmission of audio data within JSON responses | base64 library |
Sources: README.md:12, wikispeech_server/wikispeech.py:104-106, wikispeech_server/wikispeech.py:1053
Synthesis Adapters
Each synthesis engine is integrated via a specific adapter module. These modules must implement a synthesise function that returns a URL (or local path) to the generated audio and a list of timing tokens.
Adapter Implementations
| Adapter | Engine Technology | External Service Dependency |
|---|---|---|
marytts_adapter | HMM-based synthesis | MaryTTS Server |
piper_adapter | ONNX-based neural TTS | Piper Service |
matcha_adapter | Diffusion-based TTS | Matcha Service |
flite_adapter | Festival Lite (local) | engines/flite binary |
espeak_adapter | Formant synthesis | espeak-ng binary |
Sources: wikispeech_server/adapters/README.md:1-32, wikispeech_server/adapters/piper_adapter.py:53-63, wikispeech_server/adapters/marytts_adapter.py:101-118
Sequence Diagram: Piper Synthesis
This diagram shows the interaction between the Wikispeech server and the Piper TTS service.
Configuration and Management
Audio settings are controlled via a .conf file. Key settings include the directory for temporary files and the URLs for external synthesis services.
Key Configuration Parameters
| Section | Parameter | Description |
|---|---|---|
Audio settings | audio_tmpdir | Directory for synthesized audio (default: ./wikispeech_server/tmp) |
Services | marytts | URL to the MaryTTS process endpoint |
Services | piper | Base URL for the Piper TTS service |
Services | matcha | Base URL for the Matcha TTS service |
Sources: README.md:41-52, wikispeech_server/adapters/piper_adapter.py:13, wikispeech_server/adapters/matcha_adapter.py:13
Audio File Serving
The server provides a specific route /audio/<path:path> to serve the transcoded Opus files from the configured temporary directory.
@app.route('/audio/<path:path>')
def static_proxy_audio(path):
audio_tmpdir = config.config.get("Audio settings","audio_tmpdir")
return send_from_directory(os.getcwd()+"/"+audio_tmpdir, path)
Sources: wikispeech_server/wikispeech.py:698-704
Summary
The Audio Generation & Formats system provides a flexible, adapter-based framework for converting text to high-quality compressed audio. By abstracting various TTS engines into a unified interface and standardizing output via the Opus format, it ensures that Wikispeech can support a wide array of synthesis technologies while maintaining performance and compatibility across diverse client environments.