fix: follow the API's main-content default and send crawl --delay in seconds

RA
2d ago
+144-47

Two CLI defaults disagreed with the API.

Crawl --delay is sent in seconds. --delay <ms> is documented in milliseconds, but its value went to the API unchanged, and the API reads delay as seconds (at most 60). So --delay 1000 asked for 1000 seconds and was refused. The flag keeps its millisecond unit, and the CLI now divides by 1000 before sending: --delay 1500 sends delay: 1.5.

Scrapes follow the API's main-content default. scrape, download, batch-scrape and parse always sent onlyMainContent: false, so their output kept navigation, headers and footers, unlike the API, the SDKs and search --scrape.

  • Now these commands send onlyMainContent only when a flag sets it, so by default the API returns the main content.
  • A new --full-page flag sends onlyMainContent: false for the old output.
  • --only-main-content still works.
  • Passing both flags is refused.
  • The download wizard's "Only main content?" prompt now defaults to yes, and answering no sends false.

Visible change, for the next release notes. firecrawl scrape, download, batch-scrape and parse now return the page's main content by default, without navigation, headers or footers. Add --full-page to keep the whole page as before. crawl --delay now works as documented, in milliseconds.

This is meant to ship in the release after 1.26.24. The README option tables mention --full-page.

Tests. New tests cover:

  • the unset default, --full-page, and both flags together;
  • batch-scrape with --full-page;
  • the delay conversion.

The batch-scrape and crawl tests that asserted the old request bodies now assert the new ones. I also checked the built CLI against a local capture server: plain scrape sends no onlyMainContent, --full-page sends false, --only-main-content sends true, and crawl --delay 1500 sends delay: 1.5. Type check and build pass, and so does the full suite (877 tests).

Open in Capy