fix(webfetch): block SSRF to metadata, loopback and private ranges

Merged
TE
2d ago
+2479-85

Purpose

web_fetch only checked the URL scheme, so it could reach cloud metadata, loopback (including Denkeeper's own API) and private ranges, directly or through a redirect. Found in the OWASP Agentic Top 10 review (ASI02).

Closes #589

What a reviewer needs to know

  • Checked at connect time. A new internal/netguard package refuses blocked addresses in the dialer, so redirects, robots.txt lookups and DNS rebinding are covered. A URL-string check runs first only for a clearer error.
  • Closed by default, two opt-ins. [web.fetch] allow_loopback and allow_private open their own ranges. Metadata (incl. Azure 168.63.129.16), link-local and the proxy's own address never open. The docs spell out how broad allow_private is.
  • Blocked URLs never reach Jina. The fallback used to fire on any error. A robots.txt or agents.txt disallow still falls back to Jina, as before.
  • Local-use NAT64 64:ff9b:1::/48 counts as private. A host whose DNS64 uses it needs allow_private to fetch IPv4-only sites.
  • Proxy kept. With HTTP(S)_PROXY set, the proxy must enforce egress. Documented once under "Network policy".
  • allowed_domains / blocked_domains are name-only. Bad entries (.evil.com, IP literals, unmappable hosts) fail at load.
  • Remote MCP is untouched. Its guard allows RFC1918 on purpose for LAN servers.
  • Browser allowlist now uses netguard, but has no production caller. The container's own check is Temikus/denkeeper-browser#89.

Test plan

  • just hook
  • Tests use live httptest servers and the real dialer. The redirect, metadata, IDNA, proxy and default-config tests fail on main

šŸ¤– Generated with Claude Code

https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh


Summary by cubic

Blocks web_fetch, the browser allowlist, and the Jina fallback from reaching cloud metadata, loopback, and private ranges, directly or via redirects. The check runs on the IP each connection reaches at dial time, so redirects, robots/agents.txt lookups, and DNS rebinding are covered. Closes #589.

The guard

  • New internal/netguard package classifies addresses in the dialer; loopback and private ranges open only via [web.fetch] allow_loopback and allow_private.
  • Metadata (incl. Azure 168.63.129.16, AWS fd00:ec2::254, Alibaba 100.100.100.200), link-local, unspecified, multicast, and broadcast never open, even inside private ranges.
  • IPv6 forms carrying an IPv4 address (mapped, NAT64, 6to4) are judged by that IPv4; local-use NAT64 layouts are checked at every RFC 6052 position.

Behavior changes

  • A blocked URL never reaches Jina, and Jina's client is guarded too, checking every redirect hop by URL; refusals name the setting that would open the class.
  • allowed_domains and blocked_domains match the IDNA-mapped host; IP literals, numeric last labels, and malformed entries fail at config load. They cover the URL sent to Jina, not Jina's own redirect targets, and robots/agents.txt disallows still fall back to Jina.
  • The proxy dial is exempt, so the proxy must enforce egress; a direct request to the proxy's own address is refused, and NO_PROXY hosts go through the address check.
  • Empty-label and non-canonical numeric hosts are refused so a pure-Go resolver failure cannot forward them to Jina.
  • The browser allowlist uses the same classifier, IDNA-maps hosts and patterns, and refuses unmappable patterns; the remote-MCP guard is unchanged and still allows RFC1918.
  • Enabling an opt-in logs a single startup warning, and a reload that changes an opt-in logs a warning that running agents keep the old policy until restart.