Relevant source files

The following files were used as context for generating this wiki page: - [apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/JsoupExtractionProvider.kt](https://github.com/briefy-ai/briefy/blob/main/apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/JsoupExtractionProvider.kt) - [apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/FirecrawlAgentExtractionProvider.kt](https://github.com/briefy-ai/briefy/blob/main/apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/FirecrawlAgentExtractionProvider.kt) - [apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt](https://github.com/briefy-ai/briefy/blob/main/apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt) - [apps/api/src/main/kotlin/com/briefy/api/application/source/SourceContentFormatterService.kt](https://github.com/briefy-ai/briefy/blob/main/apps/api/src/main/kotlin/com/briefy/api/application/source/SourceContentFormatterService.kt) - [apps/web/src/lib/api/types.ts](https://github.com/briefy-ai/briefy/blob/main/apps/web/src/lib/api/types.ts) - [apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProviderTest.kt](https://github.com/briefy-ai/briefy/blob/main/apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProviderTest.kt)

# Extending Extraction Providers Extraction Providers are the core infrastructure components responsible for fetching and parsing raw content from various external URLs (web articles, X posts, YouTube transcripts, etc.) into a standardized format. The system is designed to be extensible, allowing developers to implement new providers that target specific platforms or utilize different extraction technologies. The extraction process is integrated into the source ingestion lifecycle, where a `Source` is moved from a `SUBMITTED` state through `EXTRACTING` to `ACTIVE` once the content is successfully retrieved. Sources: [apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:338-348](), [apps/web/src/lib/api/types.ts:31-35]() ## Core Architecture The system utilizes an interface-driven approach for extraction. Every extraction provider must implement the `ExtractionProvider` interface, which defines a unique ID and a single `extract` method. ### The ExtractionProvider Interface A new provider must provide an implementation for: - `id`: An `ExtractionProviderId` (e.g., `JSOUP`, `FIRECRAWL`, `X_API`, `YOUTUBE`). - `extract(url: String)`: A method that returns an `ExtractionResult` containing the text, metadata, and status flags. The following diagram illustrates the relationship between the service layer and the providers: ```mermaid classDiagram class ExtractionProvider { <> +ExtractionProviderId id +extract(url: String) ExtractionResult } class JsoupExtractionProvider { +extract(url: String) ExtractionResult } class FirecrawlAgentExtractionProvider { +extract(url: String) ExtractionResult } class XApiExtractionProvider { +extract(url: String) ExtractionResult } ExtractionProvider <|.. JsoupExtractionProvider ExtractionProvider <|.. FirecrawlAgentExtractionProvider ExtractionProvider <|.. XApiExtractionProvider SourceService --> ExtractionProviderResolver : resolve ExtractionProviderResolver --> ExtractionProvider : returns ``` Sources: [apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/JsoupExtractionProvider.kt:28](), [apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/FirecrawlAgentExtractionProvider.kt:20](), [apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:351]() ### Extraction Data Structures The output of any provider must be encapsulated in an `ExtractionResult`. | Field | Type | Description | | :--- | :--- | :--- | | `text` | `String` | The raw or partially cleaned text content extracted. | | `title` | `String?` | The title of the content, if available. | | `author` | `String?` | The author name. | | `publishedDate` | `Instant?` | The publication timestamp. | | `aiFormatted` | `Boolean` | Flag indicating if the provider output is already in high-quality markdown. | | `videoId` | `String?` | Unique identifier if the source is a video (e.g., YouTube). | | `transcriptSource` | `String?` | Indicates the source of a transcript (e.g., "captions", "whisper"). | Sources: [apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/JsoupExtractionProvider.kt:47-53](), [apps/web/src/lib/api/types.ts:18-28]() ## Implementation Patterns ### 1. Simple Synchronous Extraction (Jsoup) The `JsoupExtractionProvider` serves as the standard fallback. It uses the Jsoup library to connect to a URL, validate the host against private network blocks, and parse HTML tags like `

`, `

`, or Open Graph meta tags to find content. Key implementation details: - **Validation**: Ensures the URL is not pointing to `localhost` or private IP ranges. - **Cleaning**: Removes `