feat(scrape): hint when an Alexandria provider covers the scraped site

NI
4d ago
+339-1

With an API key, firecrawl scrape <url> now checks whether an Alexandria provider covers the page's site and, when one does, prints one line to stderr:

Alexandria has a structured-data provider for fred.stlouisfed.org (fred). For typed records instead of page text, list its tools with `firecrawl list fred`, then run one with `firecrawl scrape fred/<capability> --options '{...}'`.

Until now the only way to learn this from a scrape was passing --domain-tools, which few callers do.

How the lookup behaves

  • It starts beside the scrape and posts only the host (https://<host>/) to /exchange/skills/resolve, a free read-only discovery route.
  • After a successful scrape it waits at most 500 ms for the answer; a late lookup is aborted so it never keeps the process alive. A failed or empty lookup prints nothing, and a failed scrape prints no hint.
  • Each host is looked up and hinted once per run, so scrape a b c and x download on one site make one request and print one line.
  • It is skipped in keyless mode, against a self-hosted --api-url, with --lockdown, and with --domain-tools (which already returns the tools).

Verification

  • New unit tests in scrape.test.ts cover the request shape, the hint text, once-per-host dedupe, failure and no-match silence, the skip cases and the late-lookup abort. The existing scrape tests now stub fetch.
  • Ran the built CLI against the live API: a FRED series URL prints the hint and an example.com scrape prints none.

Open in Capy

Summary by cubic

With an API key, firecrawl scrape now detects when an Alexandria provider covers the scraped site and prints a one-line hint to stderr naming the provider and how to run its tools. Until now the only way to learn this was --domain-tools, which few callers use.

Behavior

  • The lookup posts only the site's host to the free /exchange/skills/resolve route and never fails the scrape it accompanies.
  • Each host is checked once per run; concurrent scrapes of the same host share one lookup that stays alive until all of them finish, and a response arriving after 500 ms is aborted so it never keeps the process alive.
  • The check is skipped in keyless mode, against a self-hosted --api-url, with --lockdown, or when --domain-tools is passed. The cloud API URL is accepted with or without a trailing slash.