Fal AI Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Fal AI integration within the Recoup project provides a high-performance pipeline for generating AI-driven media assets, including images, videos, and lip-synced audio-video content. By leveraging the @fal-ai/client, the system orchestrates complex workflows that transform artist data, templates, and raw audio into social-ready promotional content.
This integration acts as the core generative engine for the create-content task, handling subject-specific image generation (using face guides), motion synthesis, and high-quality upscaling.
Architecture and Pipeline Overview
The integration follows a structured pipeline that communicates with Fal AI's asynchronous inference endpoints. It manages state between steps by passing Fal-hosted URLs between different AI models.
Content Creation Flow
The following diagram illustrates the standard data flow from initial configuration to the final rendered video asset.
Key Components and Functions
Request Handling with falSubscribe
The system utilizes a wrapper around the Fal SDK's subscribe method to handle polling and error management for long-running inference tasks. It ensures that requests include logging and provides safe error reporting for debugging.
- Function:
falSubscribe(model, input) - Behavior: Initiates a subscription to a Fal model, enables logs, and extracts the resulting media URL from the response payload.
Sources: src/content/falSubscribe.ts, src/content/generateContentImage.ts:67-85
Image Generation
The generateContentImage function combines subject guides (like artist headshots) with template-based reference images and prompts to create a base visual asset.
| Parameter | Type | Description |
|---|---|---|
faceGuideUrl | string | Fal storage URL of the subject (artist face or album cover). |
referenceImagePath | string | Local path to a template scene composition image. |
prompt | string | The refined AI prompt combining style guides and scene instructions. |
Sources: src/content/generateContentImage.ts:18-38
Video Synthesis and Lip-Syncing
The integration supports two primary video generation paths:
- Standard Video: Animates a static image based on a motion prompt using image-to-video models.
- Lip-Sync (Audio-to-Video): Uses the Fal LTX-2 model to animate a subject's mouth and facial expressions in synchronization with an uploaded audio clip.
Storage and Asset Management
The integration minimizes local storage usage by utilizing Fal AI's built-in storage service. Files such as face guides, reference images, and audio clips are uploaded to Fal before inference.
- Face Guide Resolution: The system fetches artist-specific assets (e.g.,
face-guide.png) from GitHub and uploads them to Fal storage to serve as a consistent subject for the AI. - Media Extraction: A utility function
extractFalUrlis used across all modules to traverse different Fal API response shapes (e.g.,data.image.url,data.images[0].url,data.video.url) to find the final hosted asset.
Sources: src/content/resolveFaceGuide.ts:25-50, src/content/generateContentImage.ts:88-105
Implementation Details
Audio Preparation for Lip-Sync
Before generating a lip-synced video, the system must prepare the audio. The clipAndUploadAudio function uses ffmpeg to extract a specific segment of a song (based on startSeconds and durationSeconds) before uploading the resulting buffer to Fal.
Sources: src/content/generateAudioVideo.ts:80-112
Model Configuration
The pipeline uses a central configuration (DEFAULT_PIPELINE_CONFIG) to manage model identifiers and inference parameters.
| Step | Default Model/Setting | File Reference |
|---|---|---|
| Image Gen | config.imageModel | src/content/generateContentImage.ts:41 |
| Video Gen | config.videoModel | src/content/generateContentVideo.ts:28 |
| Lip-Sync | config.audioVideoModel (LTX-2) | src/content/generateAudioVideo.ts:54 |
| Aspect Ratio | 16:9 | src/content/generateContentVideo.ts:56 |
Conclusion
The Fal AI integration provides a robust, asynchronous media generation framework. By abstracting the complexities of model polling, asset storage, and multi-model orchestration, it enables the Recoup project to generate high-quality, personalized artist content with minimal local compute overhead.