Redirect seventeen guessed Learn URLs that crawlers keep requesting
Summary
AI assistants and their crawlers keep requesting Learn paths that never existed or no longer do. Cloudflare's edge data for the last 28 days shows about 9,000 such requests and no human sessions in GA4. Most come from Meta's crawler; a few come from Googlebot and Claude.
This PR adds 17 exact rules. Each sends a guess with one clear subject to the page that covers it today.
| Requests (28 days) | Source | Target |
|---|---|---|
| 1,358 | /docs/netdata-ai/machine-learning-and-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 1,134 | /docs/netdata-agent/configuration/optimize-the-netdata-agents-performance | /docs/netdata-agent/configuration/performance-optimization |
| 1,078 | /docs/netdata-ai/machine-learning-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 1,008 | /docs/netdata-agent/configuration/running-the-agent-behind-a-reverse-proxy | .../securing-agents/running-the-agent-behind-a-reverse-proxy |
| 758 | /docs/netdata-agent/machine-learning-and-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 493 | /docs/netdata-agent/ml-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 395 | /docs/netdata-ai/anomaly-advisor | /docs/netdata-ai/troubleshooting/anomaly-advisor |
| 361 | /docs/netdata-agent/observability-centralization-points | /docs/netdata-parents |
| 283 | /docs/netdata-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 266 | /docs/collectors | /docs/collecting-metrics/collectors |
| 238 | /docs/netdata-agent/machine-learning-anomaly-detection | /docs/netdata-ai/anomaly-detection |
| 203 | /docs/netdata-agent/ml | /docs/netdata-ai/anomaly-detection |
| 153 | /docs/alerts-&-notifications/configure-health-alerts | /docs/alerts-&-notifications/alert-configuration-reference |
| 146 | /docs/collecting-metrics/system-metrics/system-statistics | .../collectors/operating-systems/system-statistics |
| 122 | /docs/netdata-agent/daemons | /docs/netdata-agent/daemon |
| 111 | /docs/netdata-agent/collectors | /docs/collecting-metrics/collectors |
| 109 | /docs/netdata-agent/streaming | /docs/netdata-parents |
Some guesses have no single clear subject, for example /docs/netdata-agent/real-time-monitoring, /docs/netdata-agent/configuration/running-the-agent and /docs/netdata-ai/alerts-and-notifications. They stay on the 404 page.
Validation
- Every source returns 404 in production today. None has an existing exact rule, and no wildcard covers any of them.
- Every target returns 200 in production and appears in the live sitemap.
- The static section of
netlify.tomlis byte-identical tostatic.toml. - I ran
scripts/verify-redirect-graph.jsagainst a route tree built from the live sitemap andstatic/. The branch verifies 3,931 redirects, which is master's 3,914 plus these 17.
Summary by cubic
Adds 17 exact redirects for Learn doc paths that never existed or no longer do, but that crawlers keep requesting — about 9,000 requests in 28 days with no human sessions. Each now goes to the page that covers its subject; guesses without one clear subject still return 404.
Validation
- Both
netlify.tomlandstatic.tomlreceive identical rules. - Every source returns 404 in production today and no wildcard covers it; every target returns 200 and appears in the live sitemap.
Summary by CodeRabbit
- Documentation
- Added redirects from alternate and outdated documentation URLs to current pages for anomaly detection, troubleshooting, performance and reverse-proxy configuration, Netdata Parents, daemons and collectors, system statistics, and health-alert configuration.
- Visitors using these URLs are now directed to the relevant current documentation.