Media Generation Integrations
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Media Generation Integrations comprise a sophisticated pipeline designed to automate the creation of social-ready video content for musical artists. This system orchestrates various AI models and media processing tools to transform artist assets—such as face guides, audio clips, and style templates—into high-quality videos featuring synchronized audio, AI-generated imagery, and dynamic captions.
The integration leverages the fal.ai platform for core generative tasks, including image generation, video animation, and lipsyncing. The final stage of the pipeline utilizes ffmpeg for post-production tasks like aspect ratio cropping, audio overlay, and adaptive caption rendering.
Sources: src/tasks/createContentTask.ts:25-44, src/content/renderFinalVideo.ts:114-124
Pipeline Architecture
The content creation process is structured as a sequential pipeline executed within a Trigger.dev background task. It supports two primary paths: a "Normal" path for standard image-to-video generation and a "Lipsync" path that bakes audio directly into the AI-generated video.
High-Level Data Flow
The following diagram illustrates the lifecycle of a content creation request from initial payload to the final rendered video.
Component Deep Dive
1. Template Loading and Prompt Engineering
The system uses pre-defined templates to govern the aesthetic of the generated content. Templates include image prompts, style guides (camera types, lighting, realism rules), and motion prompts.
loadTemplate: Retrieves JSON-based style and caption guides from the filesystem.buildImagePrompt: Construct a detailed prompt for fal.ai by merging a base scene description with technical style rules.buildMotionPrompt: Generates instructions for the video model, typically enforcing a static camera with subtle natural movements.
Sources: src/content/loadTemplate.ts:40-84, src/content/loadTemplate.ts:128-154, src/content/loadTemplate.ts:160-173
2. Image and Video Generation (fal.ai)
The integration interfaces with fal.ai models to generate visual assets.
| Feature | Function | Description |
|---|---|---|
| Image Generation | generateContentImage | Combines a face guide headshot and a template reference image with a prompt. |
| Video Generation | generateContentVideo | Animates a static image based on motion prompts. |
| Lipsync Generation | generateAudioVideo | Uses the LTX-2 model to create lip-synced animation from an image and an audio buffer. |
| Upscaling | upscaleImage / upscaleVideo | Enhances detail and resolution (e.g., 720p to 1080p). |
Sources: src/content/generateContentImage.ts:24-45, src/content/generateAudioVideo.ts:32-60, src/tasks/createContentTask.ts:33-35
3. Final Rendering and Post-Production
The final stage uses ffmpeg to transform the generated 16:9 media into a 9:16 vertical format suitable for TikTok and Instagram Reels.
Adaptive Captioning
The system includes logic to calculate caption layouts dynamically, ensuring text fits within the frame regardless of length.
Configuration and Schemas
Payload Schema
The createContentPayloadSchema defines the requirements for triggering the pipeline.
| Parameter | Type | Default | Description |
|---|---|---|---|
accountId | string | Required | Unique identifier for the account. |
artistSlug | string | Required | Slug for the musical artist. |
template | string | Required | Name of the style template to use. |
lipsync | boolean | false | If true, enables audio-synced lip animation. |
captionLength | enum | short | Controls text length (short, medium, long). |
upscale | boolean | false | If true, triggers additional enhancement steps. |
githubRepo | string | Required | URL to fetch artist face guides and songs. |
Sources: src/schemas/contentCreationSchema.ts:6-21
FFmpeg Processing Logic
The buildFfmpegArgs function constructs the command line arguments for the render. Key operations include:
- Center Crop:
crop=ih*9/16:ih - Scaling:
scale=720:1280 - Drawtext: Multi-layered filters applying white text with a black border (
borderw) for readability. - Audio Mapping: Synchronizing the provided song clip with the video duration.
Sources: src/content/renderFinalVideo.ts:244-325
Conclusion
The Media Generation Integrations provide a robust, AI-driven workflow for automated video production. By combining the generative capabilities of fal.ai with the precise control of ffmpeg, the system efficiently produces professional-grade social media content tailored to an artist's unique identity and style.