feat(batch-scrape): add a batch-scrape command and JSON extraction guidance

Merged
ER
1d ago
+998-4

Adds firecrawl batch-scrape, a command for the /v2/batch/scrape API, and teaches the bundled skills when to use it and how to extract JSON fields from a page.

Today firecrawl scrape url1 url2 ... runs concurrent single scrapes that keep markdown only. batch-scrape runs one async job that keeps every requested format, supports JSON extraction, webhooks and cancellation, and can be checked later by job ID.

Command

  • URLs come from arguments and/or --urls-file (one per line; blank lines and # comments ignored). URLs are normalised and de-duplicated.
  • A job ID instead of URLs checks status; --status and --cancel work like crawl.
  • Scrape options: --format, --only-main-content, --wait-for, --max-age, and --scrape-options/--scrape-options-file as the base that explicit flags override. --schema, --schema-file and --prompt add the json format; --format json without either fails locally.
  • Job options: --wait, --progress, --poll-interval, --timeout, --webhook, --max-concurrency, --ignore-invalid-urls, -o, --pretty. Without --wait it prints the job ID and a status hint, matching crawl.
  • Uses the SDK's startBatchScrape, batchScrape, getBatchScrapeStatus and cancelBatchScrape with integration: 'cli'.

Skills and docs

  • firecrawl-scrape: new "Structured JSON from a page" (scrape --format json --schema) and "Many known URLs" (batch-scrape) sections.
  • firecrawl: a command-table row and a pointer for batch scraping.
  • README: a batch-scrape section after crawl.
  • Version 1.26.10.

Tests

  • 19 unit tests for URL collection, option building, job-ID routing, wait and no-wait, progress polling and cancel; cli-aliases covers the job-status case against the built CLI.
  • pnpm run type-check, pnpm run format:check and pnpm test pass.
  • Live runs against the API: a two-URL wait with markdown,links, a no-wait start then status by job ID, --schema and --prompt extraction, --urls-file then --cancel, and the two local error paths. Webhook delivery and a live --timeout expiry were not exercised.

Open in Capy

Summary by cubic

Adds firecrawl batch-scrape, a command for the /v2/batch/scrape API, and teaches the bundled skills when to use it and how to extract JSON fields from a page. Unlike multi-URL scrape, which keeps markdown only, batch-scrape runs one async job that keeps every requested format, supports JSON extraction, webhooks, and cancellation, and can be checked later by job ID.

  • URLs come from arguments and/or --urls-file (one per line, blank lines and # comments ignored), normalised and de-duplicated.
  • A job ID instead of URLs checks status; --status and --cancel work like crawl.
  • --schema, --schema-file, or --prompt add the json format and keep existing --scrape-options formats; a json format without any of them fails locally.
  • Uses the SDK's startBatchScrape, batchScrape, getBatchScrapeStatus, and cancelBatchScrape with integration: 'cli'; version bumped to 1.26.18.
  • Skills, README, and tests updated with batch scrape usage and JSON extraction guidance.