System Observability & Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
System Observability and Monitoring ensures that the AI Network Operations Platform remains healthy, secure, and transparent. For a network engineer, this means having a "clear view of the dashboard" for the platform itself, ensuring that every AI decision is auditable, every change is tracked, and the underlying infrastructure—like the topology map and discovery engine—is performing correctly.
The platform is designed with a "local-first" and "audit everything" philosophy. It provides real-time health checks, structured logging, and performance metrics to ensure that the AI assistant behaves like a reliable member of the operations team. This system monitors the health of specialized components, such as the background engines running discovery jobs and the living topology maps that track network relationships.
Sources: CLAUDE.md, README.md, deploy/docker/README.md
1. Overview
The AI Network Operations Platform serves as an autonomous assistant for enterprise infrastructure teams. It handles complex tasks including discovery, troubleshooting, configuration management, and DDI (DNS, DHCP, IPAM) across a multi-vendor environment including Cisco, Juniper, Arista, Palo Alto, and others. The primary purpose of the observability system is to maintain the "Human-in-the-loop" requirement, where every AI reasoning step is visible and every change requires manual approval.
Sources: CLAUDE.md, README.md
Key Design Principles
- Local-first & Self-hosted: All monitoring data and AI reasoning stay within your infrastructure.
- Audit Everything: Every interaction, from a plain English question to a configuration change, is logged for review.
- Human Approval: No change is executed without a second human verifying the request.
- Explainable AI: The platform must explain its logic (e.g., why it thinks a BGP peer is down) before suggesting a fix.
Sources: CLAUDE.md, README.md
Specialized "Specialist Agent" Team
The platform routes user questions to specialized agents, which can be thought of as expert team members:
- Master Architect: The supervisor that routes your question to the right specialist.
- Discovery Agent: Enumerates inventory and crawls the network.
- Troubleshooting Agent: Diagnoses live faults like routing or ACL issues.
- DDI Agent: Manages DNS, DHCP, and IPAM records.
- Documentation Agent: Generates diagrams and runbooks.
Sources: backend/app/llm/prompts/init.py, backend/app/agents/documentation/agent.py
2. Platform Architecture
The platform consists of seven core components that work together to provide a seamless experience for network operations.
Sources: README.md, deploy/docker/README.md, deploy/kubernetes/netops/README.md
Component Breakdown
- Frontend: The web console where you chat with the AI, view the topology map, and approve changes.
- API: The central gateway that handles user requests, security, and coordinates with agents.
- Worker: The background engine that runs discovery scans, takes configuration backups, and captures packets without blocking your active session.
- Postgres: The main system of record that stores device inventory, user accounts, and change history.
- Neo4j: A living topology map that tracks how every device, interface, and subnet connects to everything else.
- Redis: A high-speed messenger that passes tasks to the workers and stores temporary data.
- Ollama: The local brain that runs the AI models without sending data to the cloud.
Sources: README.md, deploy/docker/README.md
Vendor Plugin System
The platform uses a standardized connector for each vendor family. Whether you are interacting with a Cisco IOS router or a BlueCat Address Manager, the platform normalizes the output into a consistent format before the AI processes it or the topology map is updated.
Sources: README.md, backend/pyproject.toml
Deployment Options
- Docker Compose: Ideal for labs or small teams. It runs the entire platform on a single server using a single configuration file.
- Kubernetes (Helm): Designed for production environments. It offers better resilience by distributing components across multiple nodes and provides automated backup routines.
Sources: deploy/docker/README.md, deploy/kubernetes/netops/README.md
3. System Flows and Data Models
System Overview
This diagram shows how users interact with the platform and what external systems the platform manages.
Explanation: The engineer uses a console to talk to agents, which use vendor-specific plugins to interact with the actual network hardware and cloud services.
Sources: CLAUDE.md, README.md, deploy/docker/docker-compose.yml
Agent Routing and Decision Flow
When you ask a question, the Master Architect determines which specialist should handle it and whether the resulting action requires an approval gate.
Explanation: Questions are categorized by the Master Architect. "Read" actions provide immediate answers, while "Write" actions (like changing a DNS record) are diverted to a draft state for human review.
Sources: backend/app/llm/prompts/init.py, backend/app/agents/documentation/agent.py
Topology Data Model
This represents what the platform knows about your network, stored in a way that emphasizes connections (the Neo4j "living topology map").
Explanation: The topology model tracks physical connections (interface to interface) as well as logical relationships (subnets, DNS zones, and applications).
Sources: CLAUDE.md, backend/app/agents/documentation/agent.py
4. Monitoring and Health Checks
The platform provides dedicated endpoints to monitor its internal health and the status of its dependencies.
Sources: README.md, deploy/docker/README.md, deploy/kubernetes/netops/templates/NOTES.txt
Health Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/v1/health/live | GET | Basic "liveness" check to see if the API process is running. |
/api/v1/health/ready | GET | Detailed "readiness" check that verifies connectivity to Postgres, Neo4j, and Redis. |
/docs | GET | Automated API documentation (Swagger UI). |
Observability Components
| Technology | Role for Network Engineer |
|---|---|
| Structlog | Provides structured, easy-to-search logs for every AI decision and tool call. |
| Prometheus | Tracks performance metrics, such as how long it takes to rebuild the topology map. |
| Audit Log | An append-only record of all changes, which can be verified for integrity. |
| Reasoning Traces | A detailed log of the AI's "thought process" before it arrived at an answer. |
| Sources: backend/pyproject.toml, deploy/kubernetes/netops/templates/NOTES.txt, backend/app/llm/prompts/init.py |
5. Change Approval Workflow
Every request to modify the network follows a strict lifecycle to prevent unauthorized or accidental changes.
Explanation: Before any change, the platform takes a backup. If the execution fails, it uses that backup to restore the device to its previous state.
Sources: README.md, deploy/docker/README.md, deploy/kubernetes/netops/templates/NOTES.txt
6. Summary
System Observability & Monitoring in the AI Network Operations Platform provides the transparency needed to trust an AI with critical network infrastructure. By combining real-time health checks, a standardized vendor plugin system, and a strictly enforced human-in-the-loop approval workflow, the platform ensures that the "AI Network Engineer" remains a helpful, auditable, and secure extension of the human operations team.
Sources: CLAUDE.md, README.md