Scanner & Web Crawler
Relevant source files
The following files were used as context for generating this wiki page:
The Scanner & Web Crawler system is a core component of Comply-Quick designed to analyze websites for compliance risks. Its primary purpose is to fetch page content, execute JavaScript to identify dynamically injected trackers (such as Meta Pixel or TikTok), and generate a technical compliance score and summary. The system operates as a multi-stage pipeline involving a frontend interface, a backend processing library, and an optional standalone headless-browser worker.
Sources: src/lib/scanner/pipeline.ts:1-7, scanner-worker/README.md:1-5
System Architecture
The scanner architecture is designed with a fallback mechanism to ensure reliability. It consists of the ScannerPanel UI, the pipeline logic, and two crawling strategies: a high-fidelity headless render and a lightweight static fetch.
The system prioritizes the headless worker to capture JavaScript-injected trackers that are invisible to standard static fetches.
Sources: src/lib/scanner/pipeline.ts:51-54, src/lib/scanner/crawler.ts:106-118, scanner-worker/README.md:7-10
Core Components
1. Scanner Pipeline
The pipeline ties together the crawler and the analyzer. It orchestrates the process of fetching data, analyzing findings, and generating a plain-language summary using an AI client or a deterministic fallback.
- Key Function:
runScan(params) - Data Structure:
ScanOutcomewhich includes the URL, compliance score, detected tools, findings, and a summary.
Sources: src/lib/scanner/pipeline.ts:10-21, src/lib/scanner/pipeline.ts:50-84
2. Crawler Logic
The crawler handles the retrieval of website content. It includes SSRF (Server-Side Request Forgery) guards to prevent scanning private, loopback, or non-HTTP addresses.
| Feature | Description | File Reference |
|---|---|---|
| URL Normalization | Adds https:// if missing and validates protocol/hostname. | crawler.ts:14-31 |
| Size Limit | Caps HTML payloads at 2MB to avoid huge memory consumption. | crawler.ts:9 |
| SSRF Guard | Rejects hostnames resolving to private/internal network space. | crawler.ts:27-29 |
| Timeouts | Plain fetch times out at 10s; Worker render at 45s. | crawler.ts:10, crawler.ts:60 |
3. Scanner Worker (Headless Browser)
The scanner-worker is a standalone service using Playwright and Chromium. It renders the target URL, executes JavaScript, and captures all outbound network requests. This allows the system to detect "hidden" trackers like Google Analytics or LinkedIn Insight Tags that are injected at runtime.
Sources: scanner-worker/README.md:1-6, scanner-worker/package.json:11-13
Data Flow & Integration
The interaction between the main application and the worker is authenticated via a shared secret. If the worker is unreachable, the system gracefully degrades to a server-side fetch of the static markup.
Technical Configuration
The behavior of the scanner is controlled via environment variables.
| Variable | Default | Purpose |
|---|---|---|
SCANNER_WORKER_URL | (unset) | URL of the standalone headless worker. |
SCANNER_WORKER_SECRET | (unset) | Shared Bearer token for worker authentication. |
SCAN_NAV_TIMEOUT_MS | 30000 | Max navigation time for the headless browser. |
SCAN_SETTLE_MS | 2500 | Wait time after page load for late-firing tags. |
Sources: scanner-worker/README.md:20-26, src/lib/scanner/crawler.ts:107-111
Analysis and Findings
Once data is retrieved, the system identifies third-party tools and compliance issues. It differentiates between tools found in static HTML and those captured from network traffic during the headless render.
- Tracker Fingerprinting: Detects tools like Meta (Facebook) events, Google Analytics, and TikTok pixels.
- Severity Levels: Findings are categorized as
info,warning, orcritical. - Example Findings:
trackers_without_consent,missing_privacy_policy, andsession_replay_present.
Sources: src/app/dashboard/home/ScannerPanel.tsx:34-38, src/tests/scanner.test.ts:31-45, src/tests/scanner.test.ts:50-59
Conclusion
The Scanner & Web Crawler system provides a robust mechanism for automated compliance auditing. By utilizing a headless browser worker with a reliable static fallback, it ensures that even dynamically injected tracking technologies are identified, analyzed, and summarized for the user. This technical depth allows Comply-Quick to provide accurate risk assessments that standard scrapers would miss.