Relevant source files

The following files were used as context for generating this wiki page: - [AGENTS.md](https://github.com/jakejarvis/domainstack.io/blob/main/AGENTS.md) - [README.md](https://github.com/jakejarvis/domainstack.io/blob/main/README.md) - [server/routers/domain.ts](https://github.com/jakejarvis/domainstack.io/blob/main/server/routers/domain.ts) - [app/[domain]/page.tsx](https://github.com/jakejarvis/domainstack.io/blob/main/app/%5Bdomain%5D/page.tsx) - [lib/domain-utils.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/domain-utils.test.ts) - [lib/seo.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/seo.test.ts) - [lib/db/repos/blocked-domains.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/db/repos/blocked-domains.test.ts) - [lib/db/repos/stats.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/db/repos/stats.ts) - [components/domain/headers/headers-section.tsx](https://github.com/jakejarvis/domainstack.io/blob/main/components/domain/headers/headers-section.tsx) - [lib/domain/headers-lookup.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/domain/headers-lookup.test.ts)

# Domain Intelligence Gathering Domain Intelligence Gathering is the core system responsible for fetching, processing, and persisting public information about domain names. This includes data such as registration details (WHOIS/RDAP), DNS records, SSL certificates, HTTP headers, hosting providers, and SEO signals. The system is designed to be robust and efficient, utilizing durable workflows for heavy operations and a multi-layer caching strategy to ensure fast responses. The process is initiated when a user visits a domain report page (e.g., `/github.com`). The frontend triggers a series of parallel tRPC queries, each corresponding to a specific data category. These queries are handled by a backend system that orchestrates durable, retryable workflows for data fetching and persists the results in a PostgreSQL database. *Sources: [README.md](), [app/[domain]/page.tsx:43-65](), [AGENTS.md]()* ## Core Architecture The system's architecture is centered around Next.js, tRPC, and Vercel's Workflow DevKit. When a user requests a domain report, the Next.js page component initiates parallel, non-blocking data fetches for each report section. These requests are routed through tRPC to the backend, which then invokes the appropriate durable workflow. *Sources: [app/[domain]/page.tsx:50-65](), [server/routers/domain.ts](), [AGENTS.md]()* ### High-Level Data Flow The following diagram illustrates the end-to-end flow from a user request to data presentation. ```mermaid sequenceDiagram participant User participant Page as "Next.js Page
app/[domain]/page.tsx" participant tRPC as "tRPC Router
server/routers/domain.ts" participant Workflow as "Vercel Workflow
workflows/*" participant DB as "Postgres DB" User->>Page: Visits /example.com activate Page Note over Page: Decodes and normalizes domain Page->>tRPC: Prefetch getRegistration(domain) Page->>tRPC: Prefetch getDnsRecords(domain) Page->>tRPC: Prefetch getHosting(domain) Page->>tRPC: Prefetch getCertificates(domain) Page->>tRPC: Prefetch getHeaders(domain) Page->>tRPC: Prefetch getSeo(domain) deactivate Page activate tRPC tRPC->>DB: Check cache for data alt Cache Miss tRPC->>Workflow: start(workflow, { domain }) activate Workflow Workflow-->>DB: Persist results deactivate Workflow DB-->>tRPC: Return fresh data else Cache Hit DB-->>tRPC: Return cached data end tRPC-->>Page: Return data deactivate tRPC activate Page Page-->>User: Render DomainReportClient deactivate Page ``` *This diagram shows the parallel data fetching initiated by the page, the cache-or-fetch logic in the tRPC router, and the role of durable workflows.* *Sources: [app/[domain]/page.tsx:50-65](), [server/routers/domain.ts:22-37](), [AGENTS.md]()* ### Input Normalization All domain inputs are rigorously normalized before processing. This ensures consistency and prevents errors from malformed inputs. The normalization process includes: - Stripping scheme, authentication, port, and path information. - Lowercasing the domain. - Removing leading `www.` and trailing dots. - Rejecting IP literals. *Sources: [lib/domain-utils.test.ts:4-75](), [server/routers/domain.ts:11-20]()* ### Caching Strategy A crucial part of the architecture is its multi-layer caching strategy. Each tRPC endpoint responsible for data gathering first attempts to retrieve data from the cache. If a fresh, valid record is found in the PostgreSQL database, it is returned immediately. If the data is missing or stale, the system triggers a durable workflow to fetch the data, persists the new result, and then returns it. ```mermaid flowchart TD A[tRPC Endpoint Invoked] --> B{Check DB Cache}; B -- Cache Hit --> C[Return Cached Data]; B -- Cache Miss/Stale --> D[Start Durable Workflow]; D --> E[Fetch from External Source]; E --> F[Persist Result to DB]; F --> G[Return Fresh Data]; ``` *This flowchart demonstrates the "cache-or-fetch" pattern implemented in all data-gathering tRPC procedures.* *Sources: [server/routers/domain.ts:26-37](), [AGENTS.md]()* ## Data Gathering Workflows The project uses Vercel's Workflow DevKit for durable execution of heavy backend operations. Each data type (registration, DNS, etc.) has a dedicated workflow that orchestrates the necessary fetching, parsing, and persistence steps. These workflows are designed to be thin wrappers around the core business logic, which resides in `lib/domain/`. *Sources: [AGENTS.md]()* The following table summarizes the available workflows for domain intelligence gathering: | Workflow | tRPC Procedure | Purpose | | ---------------------- | ----------------- | --------------------------------------------------- | | `registrationWorkflow` | `getRegistration` | Performs WHOIS/RDAP lookup. | | `dnsWorkflow` | `getDnsRecords` | Resolves DNS records via multiple DoH providers. | | `headersWorkflow` | `getHeaders` | Probes for HTTP headers. | | `certificatesWorkflow` | `getCertificates` | Fetches the SSL/TLS certificate chain. | | `seoWorkflow` | `getSeo` | Fetches SEO metadata and `robots.txt`. | | `hostingWorkflow` | `getHosting` | Detects hosting, email, and DNS providers. | | `screenshotWorkflow` | (API endpoint) | Captures a server-side screenshot of the website. | | `faviconWorkflow` | `getFavicon` | Extracts the domain's favicon from multiple sources.| *Sources: [AGENTS.md](), [server/routers/domain.ts]()* ### Registration (`getRegistration`) This workflow uses the `rdapper` library to perform a Registration Data Access Protocol (RDAP) lookup, with a fallback to the legacy WHOIS protocol. It retrieves details like registrar, creation date, and expiration date. *Sources: [AGENTS.md](), [README.md]()* ### DNS (`getDnsRecords`) The `dnsWorkflow` resolves various DNS record types (A, AAAA, MX, TXT, NS) by querying multiple DNS over HTTPS (DoH) providers for redundancy. The results are then grouped by type for display. *Sources: [AGENTS.md](), [components/domain/dns/dns-section.tsx:20-32]()* ### HTTP Headers (`getHeaders`) This workflow probes the domain's web server, preferring a `HEAD` request for efficiency and falling back to a `GET` request if necessary. It captures the HTTP status code and all response headers. Important security-related headers are highlighted in the UI. *Sources: [lib/domain/headers-lookup.test.ts:39-49, 100-103](), [components/domain/headers/headers-section.tsx:49-52]()* ### SEO (`getSeo`) The `seoWorkflow` fetches the domain's homepage HTML and `robots.txt` file. - **HTML Parsing**: It extracts the title, meta description, canonical URL, and social media preview tags (Open Graph and Twitter). - **robots.txt Parsing**: It parses rules for different user-agents, sitemap locations, and handles directives like `Allow`, `Disallow`, and `Crawl-Delay`. The parser is case-insensitive and can handle comments and various line endings. *Sources: [lib/seo.test.ts:10-48, 62-79](), [AGENTS.md]()* ### Hosting (`getHosting`) The `getHosting` procedure is unique as it orchestrates other workflows to gather the necessary data for provider detection. It first ensures that DNS records and HTTP headers have been fetched before running the `hostingWorkflow`. This prevents redundant work and ensures the hosting detection logic has the data it needs. The diagram below shows the sequence for a `getHosting` request. ```mermaid sequenceDiagram participant Client participant tRPC_Router as "tRPC: getHosting" participant DNS_Workflow as "dnsWorkflow" participant Headers_Workflow as "headersWorkflow" participant Hosting_Workflow as "hostingWorkflow" Client->>tRPC_Router: getHosting({ domain: "example.com" }) activate tRPC_Router tRPC_Router->>DNS_Workflow: start(dnsWorkflow) tRPC_Router->>Headers_Workflow: start(headersWorkflow) DNS_Workflow-->>tRPC_Router: DNS results Headers_Workflow-->>tRPC_Router: Headers results tRPC_Router->>Hosting_Workflow: start(hostingWorkflow, { dns, headers }) Hosting_Workflow-->>tRPC_Router: Hosting provider results tRPC_Router-->>Client: Final result deactivate tRPC_Router ``` *This sequence illustrates how the `getHosting` procedure coordinates the `dnsWorkflow` and `headersWorkflow` before executing the `hostingWorkflow`.* *Sources: [server/routers/domain.ts:111-144]()* ### Screenshot Generation Screenshot generation is a long-running, resource-intensive operation handled differently from other data gathering tasks. It uses a dedicated polling-based API endpoint (`/api/screenshot`) instead of a synchronous tRPC query. 1. **Initiation**: The client POSTs to `/api/screenshot` with a `domainId`. 2. **Blocklist Check**: The workflow first checks if the domain is on a blocklist (e.g., for NSFW content). If blocked, the process is aborted. 3. **Workflow Start**: If not blocked and no cached screenshot exists, the `screenshotWorkflow` is started, which uses Puppeteer to capture the screenshot. The API immediately returns a `runId`. 4. **Polling**: The client then polls a GET endpoint with the `runId` until the status is `completed` or `failed`. 5. **Storage**: Successful screenshots are stored in Vercel Blob storage, and the metadata is persisted in the database. *Sources: [AGENTS.md](), [lib/db/repos/blocked-domains.test.ts:23-31]()* ## Database Schema Relationships The gathered intelligence is stored across multiple tables in a PostgreSQL database, linked by a central `domains` table. The `stats` repository provides insight into how these tables are joined to generate platform-wide statistics. The following ER diagram shows the relationships between the core data tables. ```mermaid erDiagram domains { string id PK string name string tld datetime createdAt } registrations { string domainId FK string registrarProviderId FK json data } dns_records { string domainId FK json records } http_headers { string domainId FK json headers int status } certificates { string domainId FK string caProviderId FK json chain } hosting { string domainId FK string hostingProviderId FK string dnsProviderId FK string emailProviderId FK } providers { string id PK string name string slug string category } domains ||--o{ registrations : "has one" domains ||--o{ dns_records : "has one" domains ||--o{ http_headers : "has one" domains ||--o{ certificates : "has one" domains ||--o{ hosting : "has one" registrations }o--|| providers : "belongs to" certificates }o--|| providers : "belongs to" hosting }o--|| providers : "hostingProvider" hosting }o--|| providers : "dnsProvider" hosting }o--|| providers : "emailProvider" ``` *This diagram outlines the database schema for storing domain intelligence, derived from join logic in `lib/db/repos/stats.ts` and schema descriptions in `AGENTS.md`.* *Sources: [lib/db/repos/stats.ts:37-160](), [AGENTS.md]()* ## Summary The Domain Intelligence Gathering system is a sophisticated, multi-component feature that forms the backbone of Domainstack. It leverages durable workflows for reliable data fetching, a robust caching layer for performance, and a well-structured database schema for persistence. By orchestrating various lookups for DNS, RDAP, SSL, and more, it provides a comprehensive, all-in-one report for any given domain. *Sources: [README.md](), [AGENTS.md]()*```json {"thought": "I need to create a technical wiki page about \"Domain Intelligence Gathering\" for the `domainstack.io` project.\n\n**Constraint Checklist & Confidence Score:**\n1. Start with `

` block: Yes\n2. List ALL relevant source files: Yes\n3. At least 5 source files: Yes, I have 15 files provided. I will select the most relevant ones.\n4. `

` format correct: Yes\n5. H1 heading `# Domain Intelligence Gathering`: Yes\n6. Concise introduction: Yes\n7. Detailed sections (H2/H3): Yes\n8. Explain architecture/logic from source: Yes\n9. Identify key functions/classes: Yes\n10. Extensively use Mermaid diagrams: Yes\n11. Diagrams are accurate and from source: Yes\n12. Brief explanation for diagrams: Yes\n13. Use Markdown tables: Yes\n14. Optional code snippets: Yes, I will use them if they add value.\n15. Cite EVERY significant piece of info: Yes, this is critical.\n16. Citation format `[filename:line-start-end]()`: Yes\n17. Cite at least 5 different files: Yes\n18. Technical accuracy (solely from files): Yes\n19. Clarity and conciseness: Yes\n20. Conclusion/Summary: Yes\n21. English language: Yes\n22. No preamble/filler: Yes\n23. No markdown headers/fences at start: Yes\n24. No closing fences at end: Yes\n25. Proper markdown formatting: Yes\n\nConfidence Score: 5/5\n\n**Mental Sandbox:**\n1. **File Selection:** The provided files are a good mix. I will focus on `server/routers/domain.ts`, `app/[domain]/page.tsx`, `AGENTS.md`, `README.md`, and the various `lib/domain/*-lookup.ts` files (even though they weren't in the provided list, `AGENTS.md` describes them, so I will reference them conceptually based on the router and workflow descriptions). `AGENTS.md` is a goldmine, describing the workflows and their purpose. `server/routers/domain.ts` shows how these workflows are invoked. `app/[domain]/page.tsx` shows the client-side initiation.\n2. **Structure:**\n * **Introduction:** Explain that Domainstack gathers public info (WHOIS/RDAP, DNS, SSL, etc.) for any given domain. Mention the use of durable workflows and caching.\n * **Core Architecture:**\n * Start with the user request flow from `app/[domain]/page.tsx`.\n * Show how it hits the tRPC router (`server/routers/domain.ts`).\n * Explain the role of Vercel Workflows as described in `AGENTS.md`.\n * Create a high-level flowchart.\n * **Data Gathering Modules (Workflows):**\n * Break down each major data type into a subsection (Registration, DNS, Headers, SEO, etc.).\n * For each, describe the workflow based on `AGENTS.md` and the corresponding tRPC endpoint in `server/routers/domain.ts`.\n * **Registration (`getRegistration`):** Mention RDAP/WHOIS lookup via `rdapper`. `AGENTS.md` describes the `registrationWorkflow`.\n * **DNS (`getDnsRecords`):** Mention DoH lookups. `AGENTS.md` describes the `dnsWorkflow`. `components/domain/dns/dns-section.tsx` shows how the data is grouped.\n * **HTTP Headers (`getHeaders`):** Mention probing via HEAD/GET. `lib/domain/headers-lookup.test.ts` shows the logic for handling different statuses and normalizing headers.\n * **SEO (`getSeo`):** Mention parsing HTML and `robots.txt`. `lib/seo.test.ts` provides great detail on the parsing logic.\n * **Hosting (`getHosting`):** This is an interesting orchestration. `server/routers/domain.ts` shows it calls DNS and Headers workflows first, then the hosting workflow. I'll create a sequence diagram for this.\n * **Certificates (`getCertificates`):** TLS handshake.\n * **Caching Strategy:**\n * The tRPC router (`server/routers/domain.ts`) clearly shows a \"check cache first\" pattern for every endpoint.\n * `AGENTS.md` mentions caching in Postgres with per-table TTLs and Next.js Data Cache.\n * I'll create a diagram illustrating the cache-then-workflow-run logic.\n * **Screenshot Generation:**\n * `AGENTS.md` describes a special polling-based API for screenshots using Puppeteer, which is different from the other tRPC-based workflows. It also mentions a blocklist.\n * `lib/db/repos/blocked-domains.test.ts` shows the implementation of the blocklist sync and check.\n * I'll detail this unique flow.\n3. **Diagrams Plan:**\n * High-level data gathering flow (User -> Page -> tRPC -> Workflow -> DB).\n * Sequence diagram for the `getHosting` orchestration.\n * Flowchart for the generic \"Cache-or-Fetch\" pattern used in the tRPC router.\n * Flowchart for the Screenshot API polling mechanism.\n * ER Diagram showing the relationship between `domains`, `registrations`, `dns_records`, `http_headers`, etc. based on the stats queries in `lib/db/repos/stats.ts` and descriptions in `AGENTS.md`.\n4. **Citations:** I will be very careful to cite everything. `AGENTS.md` will be cited heavily for architectural descriptions. `server/routers/domain.ts` for the API endpoints and caching logic. `app/[domain]/page.tsx` for the initial request. `lib/seo.test.ts` for SEO parsing details. `lib/db/repos/blocked-domains.test.ts` for the screenshot blocklist. `lib/db/repos/stats.ts` for database relationships. `README.md` for the high-level feature list.\n\nThis plan covers all requirements and uses the provided files effectively to build a comprehensive and accurate wiki page. I'll start writing now, following the structure I've laid out.","output":"

Relevant source files

\n\nThe following files were used as context for generating this wiki page:\n\n- [AGENTS.md](https://github.com/jakejarvis/domainstack.io/blob/main/AGENTS.md)\n- [README.md](https://github.com/jakejarvis/domainstack.io/blob/main/README.md)\n- [server/routers/domain.ts](https://github.com/jakejarvis/domainstack.io/blob/main/server/routers/domain.ts)\n- [app/[domain]/page.tsx](https://github.com/jakejarvis/domainstack.io/blob/main/app/%5Bdomain%5D/page.tsx)\n- [lib/domain-utils.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/domain-utils.test.ts)\n- [lib/seo.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/seo.test.ts)\n- [lib/db/repos/blocked-domains.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/db/repos/blocked-domains.test.ts)\n- [lib/db/repos/stats.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/db/repos/stats.ts)\n- [components/domain/headers/headers-section.tsx](https://github.com/jakejarvis/domainstack.io/blob/main/components/domain/headers/headers-section.tsx)\n- [lib/domain/headers-lookup.test.ts](https://github.com/jakejarvis/domainstack.io/blob/main/lib/domain/headers-lookup.test.ts)\n\n

\n\n# Domain Intelligence Gathering\n\nDomain Intelligence Gathering is the core system responsible for fetching, processing, and persisting public information about domain names. This includes data such as registration details (WHOIS/RDAP), DNS records, SSL certificates, HTTP headers, hosting providers, and SEO signals. The system is designed to be robust and efficient, utilizing durable workflows for heavy operations and a multi-layer caching strategy to ensure fast responses.\n\nThe process is initiated when a user visits a domain report page (e.g., `/github.com`). The frontend triggers a series of parallel tRPC queries, each corresponding to a specific data category. These queries are handled by a backend system that orchestrates durable, retryable workflows for data fetching and persists the results in a PostgreSQL database.\n\n*Sources: [README.md](), [app/[domain]/page.tsx:43-65](), [AGENTS.md]()*\n\n## Core Architecture\n\nThe system's architecture is centered around Next.js, tRPC, and Vercel's Workflow DevKit. When a user requests a domain report, the Next.js page component initiates parallel, non-blocking data fetches for each report section. These requests are routed through tRPC to the backend, which then invokes the appropriate durable workflow.\n\n*Sources: [app/[domain]/page.tsx:50-65](), [server/routers/domain.ts](), [AGENTS.md]()*\n\n### High-Level Data Flow\n\nThe following diagram illustrates the end-to-end flow from a user request to data presentation.\n\n```mermaid\nsequenceDiagram\n participant User\n participant Page as \"Next.js Page
app/[domain]/page.tsx\"\n participant tRPC as \"tRPC Router
server/routers/domain.ts\"\n participant Workflow as \"Vercel Workflow
workflows/*\"\n participant DB as \"Postgres DB\"\n\n User->>Page: Visits /example.com\n activate Page\n Note over Page: Decodes and normalizes domain\n Page->>tRPC: Prefetch getRegistration(domain)\n Page->>tRPC: Prefetch getDnsRecords(domain)\n Page->>tRPC: Prefetch getHosting(domain)\n Page->>tRPC: Prefetch getCertificates(domain)\n Page->>tRPC: Prefetch getHeaders(domain)\n Page->>tRPC: Prefetch getSeo(domain)\n deactivate Page\n\n activate tRPC\n tRPC->>DB: Check cache for data\n alt Cache Miss\n tRPC->>Workflow: start(workflow, { domain })\n activate Workflow\n Workflow-->>DB: Persist results\n deactivate Workflow\n DB-->>tRPC: Return fresh data\n else Cache Hit\n DB-->>tRPC: Return cached data\n end\n tRPC-->>Page: Return data\n deactivate tRPC\n\n activate Page\n Page-->>User: Render DomainReportClient\n deactivate Page\n```\n*This diagram shows the parallel data fetching initiated by the page, the cache-or-fetch logic in the tRPC router, and the role of durable workflows.*\n*Sources: [app/[domain]/page.tsx:50-65](), [server/routers/domain.ts:22-37](), [AGENTS.md]()*\n\n### Input Normalization\n\nAll domain inputs are rigorously normalized before processing. This ensures consistency and prevents errors from malformed inputs. The normalization process includes:\n- Stripping scheme, authentication, port, and path information.\n- Lowercasing the domain.\n- Removing leading `www.` and trailing dots.\n- Rejecting IP literals.\n\n*Sources: [lib/domain-utils.test.ts:4-75](), [server/routers/domain.ts:11-20]()*\n\n### Caching Strategy\n\nA crucial part of the architecture is its multi-layer caching strategy. Each tRPC endpoint responsible for data gathering first attempts to retrieve data from the cache. If a fresh, valid record is found in the PostgreSQL database, it is returned immediately. If the data is missing or stale, the system triggers a durable workflow to fetch the data, persists the new result, and then returns it.\n\n```mermaid\nflowchart TD\n A[tRPC Endpoint Invoked] --> B{Check DB Cache};\n B -- Cache Hit --> C[Return Cached Data];\n B -- Cache Miss/Stale --> D[Start Durable Workflow];\n D --> E[Fetch from External Source];\n E --> F[Persist Result to DB];\n F --> G[Return Fresh Data];\n```\n*This flowchart demonstrates the \"cache-or-fetch\" pattern implemented in all data-gathering tRPC procedures.*\n*Sources: [server/routers/domain.ts:26-37](), [AGENTS.md]()*\n\n## Data Gathering Workflows\n\nThe project uses Vercel's Workflow DevKit for durable execution of heavy backend operations. Each data type (registration, DNS, etc.) has a dedicated workflow that orchestrates the necessary fetching, parsing, and persistence steps. These workflows are designed to be thin wrappers around the core business logic, which resides in `lib/domain/`.\n\n*Sources: [AGENTS.md]()*\n\nThe following table summarizes the available workflows for domain intelligence gathering:\n\n| Workflow | tRPC Procedure | Purpose |\n| ---------------------- | ----------------- | --------------------------------------------------- |\n| `registrationWorkflow` | `getRegistration` | Performs WHOIS/RDAP lookup. |\n| `dnsWorkflow` | `getDnsRecords` | Resolves DNS records via multiple DoH providers. |\n| `headersWorkflow` | `getHeaders` | Probes for HTTP headers. |\n| `certificatesWorkflow` | `getCertificates` | Fetches the SSL/TLS certificate chain. |\n| `seoWorkflow` | `getSeo` | Fetches SEO metadata and `robots.txt`. |\n| `hostingWorkflow` | `getHosting` | Detects hosting, email, and DNS providers. |\n| `screenshotWorkflow` | (API endpoint) | Captures a server-side screenshot of the website. |\n| `faviconWorkflow` | `getFavicon` | Extracts the domain's favicon from multiple sources.|\n\n*Sources: [AGENTS.md](), [server/routers/domain.ts]()*\n\n### Registration (`getRegistration`)\n\nThis workflow uses the `rdapper` library to perform a Registration Data Access Protocol (RDAP) lookup, with a fallback to the legacy WHOIS protocol. It retrieves details like registrar, creation date, and expiration date.\n\n*Sources: [AGENTS.md](), [README.md]()*\n\n### DNS (`getDnsRecords`)\n\nThe `dnsWorkflow` resolves various DNS record types (A, AAAA, MX, TXT, NS) by querying multiple DNS over HTTPS (DoH) providers for redundancy. The results are then grouped by type for display.\n\n*Sources: [AGENTS.md](), [components/domain/dns/dns-section.tsx:20-32]()*\n\n### HTTP Headers (`getHeaders`)\n\nThis workflow probes the domain's web server, preferring a `HEAD` request for efficiency and falling back to a `GET` request if necessary. It captures the HTTP status code and all response headers. Important security-related headers are highlighted in the UI.\n\n*Sources: [lib/domain/headers-lookup.test.ts:39-49, 100-103](), [components/domain/headers/headers-section.tsx:49-52]()*\n\n### SEO (`getSeo`)\n\nThe `seoWorkflow` fetches the domain's homepage HTML and `robots.txt` file.\n- **HTML Parsing**: It extracts the title, meta description, canonical URL, and social media preview tags (Open Graph and Twitter).\n- **robots.txt Parsing**: It parses rules for different user-agents, sitemap locations, and handles directives like `Allow`, `Disallow`, and `Crawl-Delay`. The parser is case-insensitive and can handle comments and various line endings.\n\n*Sources: [lib/seo.test.ts:10-48, 62-79](), [AGENTS.md]()*\n\n### Hosting (`getHosting`)\n\nThe `getHosting` procedure is unique as it orchestrates other workflows to gather the necessary data for provider detection. It first ensures that DNS records and HTTP headers have been fetched before running the `hostingWorkflow`. This prevents redundant work and ensures the hosting detection logic has the data it needs.\n\nThe diagram below shows the sequence for a `getHosting` request.\n\n```mermaid\nsequenceDiagram\n participant Client\n participant tRPC_Router as \"tRPC: getHosting\"\n participant DNS_Workflow as \"dnsWorkflow\"\n participant Headers_Workflow as \"headersWorkflow\"\n participant Hosting_Workflow as \"hostingWorkflow\"\n\n Client->>tRPC_Router: getHosting({ domain: \"example.com\" })\n activate tRPC_Router\n tRPC_Router->>DNS_Workflow: start(dnsWorkflow)\n tRPC_Router->>Headers_Workflow: start(headersWorkflow)\n\n DNS_Workflow-->>tRPC_Router: DNS results\n Headers_Workflow-->>tRPC_Router: Headers results\n\n tRPC_Router->>Hosting_Workflow: start(hostingWorkflow, { dns, headers })\n Hosting_Workflow-->>tRPC_Router: Hosting provider results\n tRPC_Router-->>Client: Final result\n deactivate tRPC_Router\n```\n*This sequence illustrates how the `getHosting` procedure coordinates the `dnsWorkflow` and `headersWorkflow` before executing the `hostingWorkflow`.*\n*Sources: [server/routers/domain.ts:111-144]()*\n\n### Screenshot Generation\n\nScreenshot generation is a long-running, resource-intensive operation handled differently from other data gathering tasks. It uses a dedicated polling-based API endpoint (`/api/screenshot`) instead of a synchronous tRPC query.\n\n1. **Initiation**: The client POSTs to `/api/screenshot` with a `domainId`.\n2. **Blocklist Check**: The workflow first checks if the domain is on a blocklist (e.g., for NSFW content). If blocked, the process is aborted.\n3. **Workflow Start**: If not blocked and no cached screenshot exists, the `screenshotWorkflow` is started, which uses Puppeteer to capture the screenshot. The API immediately returns a `runId`.\n4. **Polling**: The client then polls a GET endpoint with the `runId` until the status is `completed` or `failed`.\n5. **Storage**: Successful screenshots are stored in Vercel Blob storage, and the metadata is persisted in the database.\n\n*Sources: [AGENTS.md](), [lib/db/repos/blocked-domains.test.ts:23-31]()*\n\n## Database Schema Relationships\n\nThe gathered intelligence is stored across multiple tables in a PostgreSQL database, linked by a central `domains` table. The `stats` repository provides insight into how these tables are joined to generate platform-wide statistics.\n\nThe following ER diagram shows the relationships between the core data tables.\n\n```mermaid\nerDiagram\n domains {\n string id PK\n string name\n string tld\n datetime createdAt\n }\n registrations {\n string domainId FK\n string registrarProviderId FK\n json data\n }\n dns_records {\n string domainId FK\n json records\n }\n http_headers {\n string domainId FK\n json headers\n int status\n }\n certificates {\n string domainId FK\n string caProviderId FK\n json chain\n }\n hosting {\n string domainId FK\n string hostingProviderId FK\n string dnsProviderId FK\n string emailProviderId FK\n }\n providers {\n string id PK\n string name\n string slug\n string category\n }\n\n domains ||--o{ registrations : \"has one\"\n domains ||--o{ dns_records : \"has one\"\n domains ||--o{ http_headers : \"has one\"\n domains ||--o{ certificates : \"has one\"\n domains ||--o{ hosting : \"has one\"\n\n registrations }o--|| providers : \"belongs to\"\n certificates }o--|| providers : \"belongs to\"\n hosting }o--|| providers : \"hostingProvider\"\n hosting }o--|| providers : \"dnsProvider\"\n hosting }o--|| providers : \"emailProvider\"\n```\n*This diagram outlines the database schema for storing domain intelligence, derived from join logic in `lib/db/repos/stats.ts` and schema descriptions in `AGENTS.md`.*\n*Sources: [lib/db/repos/stats.ts:37-160](), [AGENTS.md]()*\n\n## Summary\n\nThe Domain Intelligence Gathering system is a sophisticated, multi-component feature that forms the backbone of Domainstack. It leverages durable workflows for reliable data fetching, a robust caching layer for performance, and a well-structured database schema for persistence. By orchestrating various lookups for DNS, RDAP, SSL, and more, it provides a comprehensive, all-in-one report for any given domain.\n\n*Sources: [README.md](), [AGENTS.md]()*"} ```