Web Scraping & Research Tools
Relevant source files
The following files were used as context for generating this wiki page:
Web Scraping & Research Tools in this project provide a tiered approach to information retrieval, ranging from simple URL fetching to autonomous AI research. The system is designed to be "offline-first," offering local alternatives for every paid or cloud-based service to ensure functionality without external API dependencies. These tools are primarily implemented as CLI-based Python scripts or Model Context Protocol (MCP) tool wrappers, allowing AI agents to discover and execute them with minimal token overhead.
The architecture distinguishes between Scripts (agent-agnostic Python workflows) and Skills (Claude Code native formats), with the former serving as the execution engine for the latter. This setup enables a 99.6% reduction in tokens by using CLI arguments instead of rewriting code for every parameter change. Key integrations include Firecrawl for advanced scraping, Nia for deep documentation research, and local alternatives like Trafilatura for zero-cost operation.
Sources: scripts/README.md:3-8, scripts/SCRIPTS.md:3-10, scripts/SCRIPTS.md:126-130
Tooling Architecture & Hierarchy
The project categorizes research capabilities into cloud-powered primary tools and local-first fallbacks. This ensures that even without API keys, core scraping and search functionality remains available.
Research Tooling Tiers
| Tool Category | Cloud/Paid Tool | Local/Offline Alternative | Key Capability |
|---|---|---|---|
| Scraping | Firecrawl | Trafilatura (web_scrape_local.py) | Main content extraction, markdown formatting. |
| Search | Perplexity / Firecrawl | WebSearch (Built-in) | Web-wide query execution. |
| Documentation | Nia | Context7 MCP | Library-specific doc retrieval and indexing. |
| Research | Nia Oracle | LLM-as-judge / Local Scripts | Autonomous multi-turn research agents. |
Sources: scripts/README.md:27-34, scripts/SCRIPTS.md:130-140
Data Flow for Web Retrieval
The following diagram illustrates the decision logic when an agent needs to retrieve web information:
The flow ensures that the agent attempts to use the most powerful available tool while maintaining a guaranteed functional path through local fallbacks.
Sources: scripts/README.md:27-34, servers/firecrawl/README.md:46-52
Firecrawl Integration
Firecrawl is the primary advanced scraper and search engine used when API keys are available. It provides high-performance scraping through caching (up to 500% faster) and structured data extraction.
Core Capabilities
firecrawl_scrape: Best for single-page extraction. Supports "branding" format to extract UI components, colors, and typography for design analysis.firecrawl_map: Discovers all indexed URLs on a website to provide a site structure before scraping.firecrawl_crawl: Starts a recursive crawl job. It is recommended to limit depth and page counts to avoid exceeding token limits.firecrawl_search: Supports advanced operators (e.g.,site:,inurl:,filetype:) and can optionally extract content from results in one call.
Sources: servers/firecrawl/README.md:6-30, servers/firecrawl/README.md:46-70, scripts/firecrawl_scrape.py:64-100
Scraping Logic
The firecrawl_scrape.py script manages API interactions, including loading keys from the environment or ~/.claude/.env.
async def firecrawl_scrape(url: str, formats: list[str], main_only: bool = True) -> dict:
# API key loading from ~/.claude/.env or OS environment
# POST to https://api.firecrawl.dev/v1/scrape
payload = {
"url": url,
"formats": formats,
"onlyMainContent": main_only
}
# Returns structured dict with markdown/html content
Sources: scripts/firecrawl_scrape.py:64-100
Local Web Scraping (web_scrape_local.py)
For environments without internet-facing API keys or for strictly local execution, the project utilizes the trafilatura library.
Features
- Boilerplate Removal: Automatically extracts the main text body while discarding navigation, ads, and footers.
- Format Flexibility: Supports output in
markdown,txt, andxml. - Reference Preservation: Optionally includes or excludes links and image references through CLI flags.
Sources: scripts/web_scrape_local.py:5-20
Local Scraping Sequence
The sequence below shows how the local scraper operates without external API calls:
Sources: scripts/web_scrape_local.py:43-68
Deep Research & Documentation (Nia API)
Nia provides the most comprehensive research tier, including autonomous agents (Oracle) and specialized documentation search.
Oracle Research Agent
The Oracle agent is a multi-turn autonomous researcher capable of investigating complex technical questions. It uses specific models like claude-opus-4-5-20251101 and supports streaming events (SSE) for real-time monitoring of the research process.
Nia Search Categories
- Universal: Search across all indexed public sources.
- Package Hybrid: Combines semantic and keyword search within specific registries (npm, PyPI, Crates, Go Modules).
- Grep: Regex-based search across repository code indexed in Nia.
- Papers: Specialized indexing and search for arXiv research papers.
Sources: scripts/nia_docs.py:5-40, scripts/nia_docs.py:73-85
Nia CLI Commands
| Action | Command Example |
|---|---|
| Autonomous Research | oracle research "query" |
| Package Search | search package fastapi --query "dependency injection" |
| Paper Indexing | papers index 2310.06825 |
| Context Saving | context save --title "Title" --content "..." |
Sources: scripts/nia_docs.py:12-35, scripts/nia_docs.py:620-680
Summary of Execution Principles
The Web Scraping & Research Tools are governed by the "Parameter Immutability" principle. Developers and AI agents are encouraged to change execution behavior via CLI arguments (e.g., --url, --limit, --format) rather than editing the underlying scripts. This ensures high token efficiency (approx. 110 tokens per execution) and allows the script logic to remain a "reusable workflow template."
Sources: scripts/SCRIPTS.md:31-45, scripts/SCRIPTS.md:162-170
The inclusion of the fetch MCP tool serves as a general-purpose fallback for basic URL retrieval, granting the agent direct internet access when sophisticated scraping or research agents are not required.
Sources: servers/fetch/README.md:1-12