Content Creation Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Content Creation Pipeline is a sophisticated background task system designed to generate social-ready video content for artists. It leverages AI models (primarily via fal.ai) to synthesize images, videos, audio clips, and captions into a final rendered product. The pipeline operates as a schemaTask within the Trigger.dev v4 framework, ensuring reliable execution of long-running media processing jobs.
The system is designed to be highly configurable through templates, allowing for variations in style, mood, and movement while maintaining artist identity through "face-guides" and GitHub-hosted context files. Sources: src/tasks/createContentTask.ts:25-46, src/content/defaultPipelineConfig.ts:18-38
Pipeline Architecture
The pipeline follows a sequential multi-step execution flow. It begins with data ingestion and asset fetching, moves through AI generation phases, and concludes with local media processing using FFmpeg.
Workflow Execution Flow
The following diagram illustrates the standard execution path of a content creation task:
The pipeline handles conditional paths for lipsyncing (audio-to-video) and optional upscaling steps for both images and videos. Sources: src/tasks/createContentTask.ts:48-118
Core Components
1. Template Management
Templates define the aesthetic and structural constraints of the generated content. They are loaded from a bundled directory and contain JSON-based guides for style and captions.
- Style Guide: Contains rules for camera angles, lighting, and realism.
- Motion Prompting: Uses template-specific "video moods" and "video movements" to instruct the AI on subject behavior.
- Reference Images: Used for scene composition and setting.
Sources: src/content/loadTemplate.ts:14-30, src/content/loadTemplate.ts:133-146
2. AI Generation (fal.ai Integration)
The pipeline relies on several fal.ai models to perform heavy lifting. Configuration for these models is centralized in a default config object.
| Feature | Model / Config | Description |
|---|---|---|
| Image Generation | fal-ai/nano-banana-pro/edit | Combines face-guide and reference images with prompts. |
| Standard Video | fal-ai/veo3.1/fast/image-to-video | Animates static images (Image-to-Video). |
| Lipsync Video | fal-ai/ltx-2-19b/audio-to-video | Aligns lip movement with provided audio. |
| Upscaling | fal-ai/seedvr/upscale/image | Enhances resolution for final output. |
| Resolution | 720p (Upscale to 1080p) | Standard output resolution. |
Sources: src/content/defaultPipelineConfig.ts:28-38, src/content/generateContentImage.ts:31-42, src/content/generateAudioVideo.ts:48-60
3. Media Processing and Rendering
Final assembly occurs via FFmpeg. This stage is responsible for:
- Cropping: Converting 16:9 landscape video to 9:16 portrait for social platforms.
- Audio Overlay: Baking the selected audio clip into the video.
- Caption Overlay: Adding TikTok-style text overlays with adaptive sizing and positioning.
Sources: src/tasks/createContentTask.ts:114-122, src/content/testPipeline.ts:258-290
Data Models and Validation
Content Creation Payload
The pipeline is triggered via a Zod-validated payload defined in contentCreationSchema.ts.
export const createContentPayloadSchema = z.object({
accountId: z.string().min(1),
artistSlug: z.string().min(1),
template: z.string().min(1),
lipsync: z.boolean().default(false),
captionLength: z.enum(["short", "medium", "long"]).default("short"),
upscale: z.boolean().default(false),
githubRepo: z.string().url(),
songs: z.array(z.string()).optional(),
images: z.array(z.string().url()).optional(),
});
Sources: src/schemas/contentCreationSchema.ts:6-18
Pipeline Configuration
The PipelineConfig interface governs the behavior of AI models and media output parameters.
export interface PipelineConfig {
imageModel: string;
videoModel: string;
videoModelMaxSeconds: number;
audioVideoModel: string;
audioVideoModelMaxSeconds: number;
upscaleModel: string;
aspectRatio: AspectRatio;
resolution: Resolution;
artistImage: string;
clipDuration: number;
videoResolution: string;
}
Sources: src/content/defaultPipelineConfig.ts:11-26
Testing and Simulation
The repository includes a standalone testPipeline.ts utility that allows developers to run individual steps of the pipeline locally without requiring a full Trigger.dev environment. This script manages state across runs using a .pipeline-state.json file.
Available Test Steps:
image: Tests AI image generation from a face-guide.video: Tests animation of a previously generated image.audio: Simulates song selection and transcription.render: Executes local FFmpeg commands to verify final assembly.
Sources: src/content/testPipeline.ts:12-25, src/content/testPipeline.ts:305-320
Conclusion
The Content Creation Pipeline integrates advanced AI generative models with automated media processing to provide a hands-free content generation engine. By utilizing Trigger.dev for orchestration and fal.ai for synthesis, it maintains high-quality output while scaling with the needs of various artist catalogs and social media requirements.