Backup & Disaster Recovery
Relevant source files
The following files were used as context for generating this wiki page:
Overview
The Backup & Disaster Recovery (DR) system ensures that the AI Network Operations Platform can survive hardware failures or site outages by maintaining off-host copies of all critical data. For a network operations team, this is like having a recurring "config backup" for the entire platform, allowing you to restore the assistant to a known good state if the primary server fails.
The platform follows a "resilient-by-default" philosophy, where backup tiers are enabled automatically unless explicitly opted out. It protects three core areas: the system of record (PostgreSQL), the living topology map (Neo4j), and packet capture artifacts (pcaps). The primary tool used for data protection is pgBackRest, which handles continuous archiving and point-in-time recovery (PITR) for the main database.
Sources: deploy/kubernetes/netops/README.md:38, deploy/kubernetes/netops/templates/NOTES.txt:46, deploy/docker/README.md:144
Platform Components
To understand the backup strategy, it is helpful to view the platform as a set of specialized containers working together:
- frontend: The web console where you interact with the AI and view diagrams.
- api: The central brain that handles your requests and coordinates agents.
- worker: The background engine that runs discovery jobs, takes config backups, and captures packets.
- postgres: The system of record that stores inventory, audit logs, and credential metadata.
- neo4j: A living topology map that tracks how every device, interface, and subnet connects to everything else.
- redis: A temporary storage area used for job queuing and caching.
- ollama: An optional component for running AI models locally.
Sources: README.md:21-27, deploy/docker/README.md:10-16
Architecture
The platform supports two deployment options, each with a corresponding backup implementation.
1. Docker Compose (Lab/Small Team)
In a single-server setup, backups are managed via an opt-in profile that uses ofelia (a container-based job scheduler) to trigger pgBackRest. Data is sent to a MinIO instance (local object storage) and is encrypted using AES-256-CBC.
Sources: deploy/docker/README.md:144-150
2. Kubernetes (Production)
In production, the platform uses Kubernetes CronJobs to schedule backups. This setup is more robust, using dedicated ServiceAccounts for security and providing continuous Write-Ahead Log (WAL) archiving. This means that almost every change made to the database is streamed to off-host storage in near real-time, targeting a Recovery Point Objective (RPO) of less than 5 minutes.
Sources: deploy/kubernetes/netops/README.md:38, deploy/docker/README.md:162
System Connections & Agent Routing
The platform connects to various network vendor families (Cisco, Juniper, Palo Alto, etc.) and LLM providers. When you ask a question, the Master Architect agent routes it to a specialist "team member" (like the Discovery or DDI Agent). If the request involves a change, it must pass through a human approval gate.
This diagram shows how a user's request flows through the AI agents to either a direct answer or a human-approved network change.
Sources: CLAUDE.md:28-40, README.md:32-34
Data Protection Strategy
The platform protects different types of data using specific methods:
| Data Type | Storage Component | Backup Method | Recovery Goal |
|---|---|---|---|
| System of Record | PostgreSQL | pgBackRest (Full/Incr/WAL) | Recovery to within 5 mins of failure |
| Topology Map | Neo4j | Rebuild from Postgres | Reconstructed after DB restore |
| Packet Captures | Persistent Volume | Daily Snapshot to S3/MinIO | Restored from pcap-specific repo |
| Audit Logs | PostgreSQL | pgBackRest + Hash-Chain | Tamper-evident integrity |
Sources: deploy/kubernetes/netops/README.md:38, deploy/kubernetes/netops/templates/backup/full-platform-dr-drill-job.yaml:12-18, deploy/kubernetes/netops/templates/NOTES.txt:64
Database Backup (PostgreSQL)
PostgreSQL is the "Source of Truth." The platform uses pgBackRest to perform:
- Weekly Full Backups: A complete copy of the database.
- Daily Incremental Backups: Only the changes since the last backup.
- Continuous Archiving: Every transaction is archived as a "WAL segment," allowing the database to be restored to a specific second in time (Point-in-Time Recovery).
All backups are encrypted at rest using AES-256-CBC, ensuring that even if the backup storage is compromised, the sensitive network credentials remain secure.
Sources: deploy/kubernetes/netops/README.md:88-91, deploy/docker/README.md:144-148
Topology Rebuild (Neo4j)
The Neo4j graph database acts as a "living topology map." Rather than taking traditional backups of the graph, the platform is designed to re-project the topology from the restored PostgreSQL data. This ensures the topology is always consistent with the inventory records.
Sources: deploy/kubernetes/netops/templates/backup/full-platform-dr-drill-job.yaml:16-17
Disaster Recovery Drills
A backup is only as good as its last successful restore. The platform includes automated "DR Drills" to prove recoverability.
Full-Platform Drill
The Full-Platform DR Drill is a comprehensive test that chains multiple restoration steps together on a clean, throwaway environment. It proves that the entire system can be resurrected from backups alone with zero surviving data on the primary cluster.
This flow represents the end-to-end verification of the platform's ability to recover from total site loss.
Sources: deploy/kubernetes/netops/templates/backup/full-platform-dr-drill-job.yaml:9-19
Drill Success Criteria
Every drill must satisfy specific technical checks to be considered a "PASS":
- RPO Window: Data must be recovered to within the proposed time window (e.g., 5 minutes).
- Audit Immutability: The audit logs must show no signs of tampering during restore.
- Fail-Closed Credentials: Without the Master Key (KEK), decrypted credentials must remain inaccessible.
- Integrity: pgBackRest must verify the restored files against the repository checksums.
Sources: deploy/kubernetes/netops/templates/backup/full-platform-dr-drill-job.yaml:13-15
Configuration Options
Backup behavior is controlled via environment variables or Helm values:
| Option | Type | Default | Description |
|---|---|---|---|
NETOPS_DATABASE_URL | String | (Internal) | Connection string for the Postgres database. |
PGBACKREST_REPO1_CIPHER_PASS | Secret | (Required) | The AES-256-CBC passphrase for backup encryption. |
DRILL_RPO_WINDOW_MINUTES | Integer | 5 | The maximum allowed data loss window for a successful drill. |
backup.postgres.enabled | Boolean | true | Enables/Disables the entire pgBackRest tier. |
backup.pcap.enabled | Boolean | true | Enables/Disables daily packet capture snapshots. |
Sources: deploy/docker/README.md:113, deploy/kubernetes/netops/templates/backup/full-platform-dr-drill-job.yaml:104-106, deploy/kubernetes/netops/README.md:87
Summary
The Backup & Disaster Recovery system is a critical infrastructure component that provides the AI Network Operations Platform with "site-loss" resilience. By combining continuous database archiving, encrypted off-host storage, and automated DR drills, the platform ensures that the AI assistant, its knowledge of your topology, and its audit trails are always recoverable. For network engineers, this provides the peace of mind that the platform's "brain" is protected just as rigorously as the network devices it manages.