Relevant source files

The following files were used as context for generating this wiki page: - [apps/test-suite/README.md](https://github.com/firecrawl/firecrawl/blob/main/apps/test-suite/README.md) - [apps/api/src/__tests__/e2e_withAuth/index.test.ts](https://github.com/firecrawl/firecrawl/blob/main/apps/api/src/__tests__/e2e_withAuth/index.test.ts) - [apps/test-suite/tests/crawl.test.ts](https://github.com/firecrawl/firecrawl/blob/main/apps/test-suite/tests/crawl.test.ts) - [apps/test-suite/load-test-results/tests-6-7/load-test-7.md](https://github.com/firecrawl/firecrawl/blob/main/apps/test-suite/load-test-results/tests-6-7/load-test-7.md) - [apps/api/src/__tests__/snips/v2/scrape.test.ts](https://github.com/firecrawl/firecrawl/blob/main/apps/api/src/__tests__/snips/v2/scrape.test.ts) - [apps/js-sdk/firecrawl/src/__tests__/e2e/v2/scrape.test.ts](https://github.com/firecrawl/firecrawl/blob/main/apps/js-sdk/firecrawl/src/__tests__/e2e/v2/scrape.test.ts) - [apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py](https://github.com/firecrawl/firecrawl/blob/main/apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py) - [CONTRIBUTING.md](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md) - [apps/api/src/scraper/scrapeURL/scrapeURL.test.ts](https://github.com/firecrawl/firecrawl/blob/main/apps/api/src/scraper/scrapeURL/scrapeURL.test.ts) - [apps/test-suite/load-test-results/tests-1-5/load-test-1.md](https://github.com/firecrawl/firecrawl/blob/main/apps/test-suite/load-test-results/tests-1-5/load-test-1.md) - [apps/test-suite/load-test-results/tests-6-7/load-test-8.md](https://github.com/firecrawl/firecrawl/blob/main/apps/test-suite/load-test-results/tests-6-7/load-test-8.md) - [CLAUDE.md](https://github.com/firecrawl/firecrawl/blob/main/CLAUDE.md)

# Testing Strategy The Firecrawl project employs a multi-layered testing strategy to ensure the reliability, performance, and correctness of its web scraping and crawling services. This strategy encompasses end-to-end (E2E) tests for the API and SDKs, comprehensive load testing to evaluate system stability under stress, and standalone unit/integration tests for core modules. The goal is to verify functionality across different configurations and usage scenarios, from basic scrapes to complex, AI-driven data extraction. The testing suite is designed to cover various aspects of the system, including crawling accuracy, API response time, and error handling. Contributions to the test suite are encouraged, and developers should run relevant tests before submitting pull requests. Sources: [apps/test-suite/README.md:5-9](), [CONTRIBUTING.md:5-7](), [CLAUDE.md:6-9]() ## Test Categories The testing strategy is broken down into three main categories: End-to-End (E2E) Testing, SDK Testing, and Load Testing. | Test Type | Purpose | Tools | Key Focus | | --- | --- | --- | --- | | **End-to-End (E2E)** | Validate the full request-response cycle of API endpoints. | Jest, Supertest | API correctness, parameter handling, format variations, caching, proxy usage. | | **SDK Testing** | Ensure SDKs provide a correct and seamless interface to the API. | Jest (JS), Pytest (Python) | Client-side method validation, schema handling, response object integrity. | | **Load Testing** | Measure system performance, stability, and resource management under stress. | Artillery | Response time, request rate, error rate, CPU/Memory utilization, autoscaling. | Sources: [apps/test-suite/README.md:11-20](), [apps/api/src/__tests__/snips/v2/scrape.test.ts](), [apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py](), [apps/test-suite/load-test-results/tests-6-7/load-test-7.md]() ## End-to-End (E2E) Testing E2E tests are the primary method for validating the Firecrawl API. These tests, located in directories like `apps/api/src/__tests__`, simulate real-world usage by sending requests to the API endpoints and asserting the responses. They are written using the Jest framework. Sources: [CONTRIBUTING.md:104-105](), [CLAUDE.md:6-9]() ### API Test Structure API E2E tests typically follow a standard structure: 1. **Setup**: A `beforeAll` block initializes a test identity using an `idmux` helper function, which provides credentials for the test run. For certain tests, this block may also pre-scrape a URL to set up a baseline for change tracking. 2. **Test Cases**: Individual test cases are defined using `it.concurrent` or `test`, allowing for parallel execution. Each test case focuses on a specific feature or scenario. 3. **Assertions**: `expect` is used to validate the response status code, body content, and specific data fields like `markdown`, `metadata`, `json`, etc. The diagram below illustrates the flow of a typical E2E test for the `/v2/scrape` endpoint. ```mermaid sequenceDiagram participant TestRunner as "Jest Test Runner" participant FirecrawlAPI as "Firecrawl API" participant ExternalWebsite as "External Website" TestRunner->>+FirecrawlAPI: POST /v2/scrape (url: '...', formats: ['...']) Note right of TestRunner: Test sends request with specific options. FirecrawlAPI->>+ExternalWebsite: HTTP GET request ExternalWebsite-->>-FirecrawlAPI: HTML/PDF/JSON content FirecrawlAPI->>FirecrawlAPI: Process content (parse, clean, extract) FirecrawlAPI-->>-TestRunner: 200 OK with JSON payload { markdown: '...', metadata: {...} } TestRunner->>TestRunner: Assert response properties (e.g., expect(response.markdown).toContain(...)) ``` This sequence diagram shows an E2E test making a POST request to the `/v2/scrape` endpoint, which in turn fetches and processes content from an external website before returning the structured data to the test runner for validation. Sources: [apps/api/src/__tests__/snips/v2/scrape.test.ts:15-32](), [apps/api/src/__tests__/e2e_withAuth/index.test.ts:20-23]() ### Key Tested Features The E2E tests cover a wide range of functionalities: * **Basic Scrapes**: Verifying successful content extraction from a URL. * **Output Formats**: Testing all supported formats, including `markdown`, `html`, `rawHtml`, `links`, `images`, `screenshot`, `summary`, and `json` extraction. * **Parameter Validation**: Ensuring the API correctly rejects invalid parameter combinations, such as a `waitFor` value that exceeds the `timeout`. * **Special Content Handling**: Scraping non-HTML content like PDFs, Google Docs, and YouTube videos. * **Caching**: Validating cache behavior (`hit`/`miss` states) based on `maxAge` and request parameters. * **Proxy Usage**: Testing different proxy strategies like `basic`, `stealth`, and `auto`. * **Change Tracking**: Verifying the detection of content changes between scrapes, including `git-diff` and structured `json` diffs. * **AI-driven Extraction**: Ensuring `json` and `summary` formats, which depend on LLMs, produce the correct output based on a given schema or prompt. The following snippet from `scrape.test.ts` demonstrates a test for the JSON extraction feature. ```typescript // Source: apps/api/src/__tests__/snips/v2/scrape.test.ts:919-962 it.concurrent( "works", async () => { const response = await scrape( { url: "http://firecrawl.dev", formats: [ { type: "json", prompt: "Based on the information on the page, find what the company's mission is and whether it supports SSO, and whether it is open source.", schema: { type: "object", properties: { company_mission: { type: "string", }, supports_sso: { type: "boolean", }, is_open_source: { type: "boolean", }, }, required: [ "company_mission", "supports_sso", "is_open_source", ], }, }, ], }, identity, ); expect(response).toHaveProperty("json"); expect(response.json).toHaveProperty("company_mission"); expect(typeof response.json.company_mission).toBe("string"); expect(response.json).toHaveProperty("supports_sso"); expect(response.json.supports_sso).toBe(false); expect(typeof response.json.supports_sso).toBe("boolean"); expect(response.json).toHaveProperty("is_open_source"); expect(response.json.is_open_source).toBe(true); expect(typeof response.json.is_open_source).toBe("boolean"); }, scrapeTimeout, ); ``` Sources: [apps/api/src/__tests__/snips/v2/scrape.test.ts](), [apps/api/src/__tests__/snips/v1/scrape.test.ts]() ## SDK Testing To ensure a high-quality developer experience, both the Python and JavaScript/TypeScript SDKs have their own E2E test suites. These tests validate that the SDKs correctly interface with the Firecrawl API and handle requests and responses as expected. Sources: [apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py](), [apps/js-sdk/firecrawl/src/__tests__/e2e/v2/scrape.test.ts]() ### Python SDK Tests The Python SDK tests use `pytest`. The test class `TestScrapeE2E` initializes a `Firecrawl` client in its `setup_method` and includes various test methods to cover different scrape scenarios. ```mermaid classDiagram class TestScrapeE2E { +client: Firecrawl +setup_method() +_assert_valid_document(doc: Document) +test_scrape_minimal() +test_scrape_with_options_markdown() +test_scrape_basic_formats(fmt, expect_field) +test_scrape_with_json_format_object() +test_scrape_invalid_url() +test_scrape_with_all_params() +test_scrape_images_format() } ``` This class diagram shows the structure of the Python SDK's E2E test class for the scrape functionality. A typical test involves calling a client method and asserting the returned `Document` object. ```python # Source: apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py:27-35 def test_scrape_minimal(self): """Scrape a URL with minimal parameters and return a document.""" doc = self.client.scrape("https://docs.firecrawl.dev") self._assert_valid_document(doc) def _assert_valid_document(self, doc: Document): assert isinstance(doc, Document) # At least one main content field should be present assert ( (doc.markdown is not None and len(doc.markdown) > 0) or (doc.html is not None and len(doc.html) > 0) or (doc.raw_html is not None and len(doc.raw_html) > 0) or (doc.summary is not None and len(doc.summary) > 0) ) # Metadata should exist with a source URL or title when available assert doc.metadata is not None ``` Sources: [apps/python-sdk/firecrawl/__tests__/e2e/v2/test_scrape.py:14-25]() ### JavaScript/TypeScript SDK Tests Similarly, the JS/TS SDK uses Jest for its E2E tests. The setup and test cases mirror the API and Python tests, ensuring consistency across the ecosystem. A `Firecrawl` client is initialized in a `beforeAll` block, and tests validate the responses from methods like `client.scrape()` and `client.search()`. ```typescript // Source: apps/js-sdk/firecrawl/src/__tests__/e2e/v2/scrape.test.ts:29-34 test("minimal: scrape only required params", async () => { if (!client) throw new Error(); const doc = await client.scrape("https://docs.firecrawl.dev"); assertValidDocument(doc); }, 60_000); ``` Sources: [apps/js-sdk/firecrawl/src/__tests__/e2e/v2/scrape.test.ts:15-20](), [apps/js-sdk/firecrawl/src/__tests__/e2e/v2/search.test.ts:16-19]() ## Load Testing Load testing is critical for understanding the system's performance, stability, and resource management under pressure. Firecrawl uses **Artillery** to conduct these tests. Sources: [apps/test-suite/README.md:15-23]() ### Configuration and Execution Load tests are defined in YAML files, which specify `phases` of traffic. Each phase has a `duration` and an `arrivalRate` (requests per second). This allows for simulating various load patterns, such as gradual ramp-ups and sustained peaks. A sample configuration from a test on the crawl routes: ```yaml # Source: apps/test-suite/load-test-results/tests-6-7/load-test-7.md:46-54 phases: - duration: 60 arrivalRate: 1 # Initial load - duration: 120 arrivalRate: 2 # Increased load - duration: 180 arrivalRate: 3 # Peak load - duration: 60 arrivalRate: 1 # Cool down ``` To run a load test, the `artillery run` command is used: ```bash artillery run load-test.yml ``` Sources: [apps/test-suite/README.md:22-23]() ### Analysis and Iteration The results of each load test are meticulously documented, including key performance metrics, resource utilization (CPU and Memory), and conclusions. This data-driven approach allows the team to identify bottlenecks and stability issues. | Metric | Value (from Test #7) | |---|---| | `http.codes.200` | 1800 | | `http.requests` | 1800 | | `http.response_time.mean` | 849.2 ms | | `http.response_time.p95` | 1043.3 ms | | `vusers.failed` | 0 | Sources: [apps/test-suite/load-test-results/tests-6-7/load-test-7.md:60-78]() The load testing process is iterative. Findings from one test directly inform the next steps, such as architectural changes or resource adjustments. For example: * **Test #1** identified a potential memory leak, leading to a recommendation for detailed analysis. * **Test #7** revealed a critical failure in `fire-engine` machines due to resource exhaustion, leading to plans to increase worker counts and implement autoscaling. * **Test #8** showed that the implemented autoscaling strategy did not work as expected, prompting further investigation. This iterative cycle is crucial for progressively hardening the system. ```mermaid flowchart TD A[Configure Load Test
(artillery.yml)] --> B[Run Test
(artillery run)] B --> C[Analyze Results
(Metrics, CPU/Mem)] C --> D{Issues Found?} D -- Yes --> E[Identify Bottleneck
(e.g., Memory Leak, Autoscaling)] E --> F[Implement Fixes
(e.g., Increase workers, Optimize code)] F --> A D -- No --> G[System Stable
(Under current load)] ``` This flowchart illustrates the iterative cycle of load testing, from configuration and execution to analysis and optimization. Sources: [apps/test-suite/load-test-results/tests-1-5/load-test-1.md:83-93](), [apps/test-suite/load-test-results/tests-6-7/load-test-7.md:104-124](), [apps/test-suite/load-test-results/tests-6-7/load-test-8.md:101-112]() ## Test Execution Developers can run tests locally to validate their changes. The process involves setting up the necessary services and then executing the test suite. ### Local Test Setup Running tests locally requires multiple components to be active simultaneously. ```mermaid graph TD subgraph Developer Machine Term1[Terminal 1: redis-server] Term2[Terminal 2: pnpm start
(API Server & Workers)] Term3[Terminal 3: Test Runner
(pnpm harness jest...)] end Developer --> Term1 Developer --> Term2 Developer --> Term3 Term3 -- HTTP Requests --> Term2 Term2 -- Jobs --> Term1 Term2 -- Results --> Term1 ``` This diagram shows the local development setup for running tests, requiring separate terminals for Redis, the API/workers, and the test runner itself. Sources: [CONTRIBUTING.md:78-96]() ### Running Commands The primary commands for running tests are: * **Install dependencies**: `pnpm install` * **Install Playwright browsers**: `npx playwright install` * **Run test suite**: `npm run test` * **Run API E2E tests (`snips`)**: `npm run test:snips` * **Run specific tests with harness**: `pnpm harness jest [test-file-pattern]` The `pnpm harness` command is a helper script that manages starting the API server and workers before executing the specified tests. Sources: [apps/test-suite/README.md:11-13](), [CONTRIBUTING.md:73-76, 104-105](), [CLAUDE.md:14-17]()