Source Extraction Providers
Relevant source files
The following files were used as context for generating this wiki page:
- apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt
- apps/api/src/main/kotlin/com/briefy/api/domain/knowledgegraph/source/Source.kt
- apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProviderTest.kt
- apps/api/src/test/kotlin/com/briefy/api/application/source/SourceContentFormatterServiceTest.kt
- apps/api/src/main/kotlin/com/briefy/api/application/source/SourceSortStrategy.kt
- apps/api/src/test/kotlin/com/briefy/api/api/SourceControllerTest.kt
- apps/web/src/lib/api/types.ts
Source Extraction Providers are the core infrastructure components responsible for retrieving and normalizing content from various external platforms (Web, YouTube, X/Twitter). The system abstracts the complexity of different media types and platform-specific APIs into a unified ExtractionResult, which is then used to populate the Knowledge Graph.
The extraction process is integrated into the SourceService and can be triggered by URL submission, manual content provision, or retry requests for failed extractions.
Sources: apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:53, apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:472
Extraction Architecture
The extraction system uses a "Resolver" pattern to select the appropriate provider based on the platform of the incoming URL.
The diagram shows how the SourceService uses a resolver to delegate extraction to specific platform providers, which return a standardized result used to update the Domain Model.
Sources: apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:473-475
Provider Types and Platforms
The system categorizes extraction capabilities by provider type, supporting various platforms and media formats.
| Provider ID | Target Platforms | Primary Capabilities |
|---|---|---|
JSOUP | Web (Generic) | HTML parsing and text extraction. |
FIRECRAWL | Web (Advanced) | Handling complex web structures (e.g., PostHog). |
YOUTUBE | YouTube | Transcript retrieval (Captions/Whisper) and metadata. |
X_API | X (Twitter) | Threads, Note tweets, and video transcription. |
MANUAL | N/A | User-pasted content overrides. |
Sources: apps/api/src/test/kotlin/com/briefy/api/api/SourceControllerTest.kt:65, apps/api/src/test/kotlin/com/briefy/api/api/SourceControllerTest.kt:127, apps/web/src/lib/api/types.ts:167
Extraction Workflow
The extraction lifecycle transitions a Source entity through several states, from initial submission to being "Active" in the system.
1. Source Submission and Cache Check
When a URL is submitted, the system first checks for a reusable "Shared Snapshot." If a fresh snapshot exists (within TTL), the extraction is skipped, and the cached content is reused.
Sources: apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:66-80
2. Synchronous vs. Asynchronous Extraction
- Web/X Extraction: Typically performed synchronously during the request.
- Video Extraction: For YouTube, the system enqueues an asynchronous job via
sourceExtractionJobServicebecause video processing (transcription) is time-intensive.
The sequence diagram illustrates the branching logic between immediate extraction for text-based sources and background processing for video content.
Sources: apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:85-115
Specialized Extraction Features
X (Twitter) Extraction
The XApiExtractionProvider handles various post types:
- Single Posts: Basic text and media keys.
- Threads: Automatically fetches self-replies to build a continuous narrative.
- Note Tweets: Prefers long-form
note_tweettext over standard truncated post text. - Videos: Downloads video variants (prioritizing high bitrate within size limits) and uses
MediaWhisperTranscriptionServicefor transcription.
Sources: apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProviderTest.kt:18-60, apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProviderTest.kt:225-240
YouTube Extraction
The YouTube provider retrieves transcripts and maps metadata like videoId, videoEmbedUrl, and durationSeconds.
- Transcript Sources: Can distinguish between "Captions" (provided by YouTube) and "Whisper" (AI-generated).
- Formatting: "Captions" transcripts often require an additional formatting step via
SourceContentFormatterServiceusing models likegemini-2.5-flash-lite.
Sources: apps/api/src/test/kotlin/com/briefy/api/application/source/SourceContentFormatterServiceTest.kt:150-180, apps/api/src/test/kotlin/com/briefy/api/application/source/SourceServiceEventTest.kt:332-345
Domain Model: Extraction Result
Every provider must return an ExtractionResult which maps to the following source fields:
| Field | Description | Derived Source Metadata |
|---|---|---|
text | The raw content (Markdown or Plain Text). | content.text |
title | Title of the article/video/post. | metadata.title |
author | Creator name or handle (e.g., @user). | metadata.author |
aiFormatted | Boolean indicating if content is already clean. | metadata.aiFormatted |
ogImageUrl | Preview image URL. | metadata.ogImageUrl |
transcriptSource | How video text was obtained. | metadata.transcriptSource |
Sources: apps/api/src/test/kotlin/com/briefy/api/api/SourceControllerTest.kt:54-59, apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:476-492, apps/web/src/lib/api/types.ts:16-32
Failure Handling and Retries
Extractions can fail due to network errors, unsupported platforms, or missing API keys (e.g., Firecrawl).
- Status Transitions: Failed extractions move the source to
FAILEDstatus. - Retry Endpoint: Users can trigger
POST /api/sources/{id}/retrywhich resets the status toSUBMITTEDand attempts extraction again. - Manual Override: If extraction repeatedly fails, users can provide content manually via
POST /api/sources/{id}/content, which forces the source toACTIVEstatus.
Sources: apps/api/src/domain/knowledgegraph/source/Source.kt:66-73, apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:204-219, apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:347-368
Summary
The Source Extraction Providers system enables Briefy to ingest diverse content types by abstracting platform-specific logic. Through a combination of synchronous web parsing, asynchronous video transcription, and a robust caching layer using Shared Snapshots, it ensures that the downstream AI agents have high-quality, normalized content to process into briefings.