Monitoring & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Monitoring & Observability in the NeXifyAI platform consists of a distributed architecture of specialized agents, health check services, and automated scoring systems. The system tracks the "4 Golden Signals"—Latency, Traffic, Errors, and Saturation—to ensure operational excellence across autonomous AI services.
The architecture integrates real-time metric collection via Prometheus, visualization through Grafana, and automated health assessments via the monitoring-specialist and senior-quality-auditor agents. It enforces strict "Quality Gates" that block deployments if monitoring, health checks, or alerts are missing.
System Health Architecture
The platform calculates a composite Health Score using real-time metrics gathered from backend logs, Git deployment frequency, and API latency. This score is exposed via a dedicated dashboard and automated cron jobs.
Health Score Calculation
A Python-based automation service calculates the health score by applying weighted values to specific metrics. The system classifies health into five statuses: Excellent, Good, Fair, Degraded, and Critical.
| Metric | Weight | Description |
|---|---|---|
| Uptime | 0.25 | Backend availability over 30 days via Watchdog. |
| Error Rate | 0.20 | Percentage of errors tracked in backend logs. |
| Latency | 0.15 | P95 response time from the /api/health endpoint. |
| Deploy Frequency | 0.10 | Number of commits tracked via Git per week. |
| MTTR | 0.10 | Mean Time To Recovery calculated from incident logs. |
| Security | 0.10 | Status of CVE and secret scans. |
| Conversion | 0.10 | Events per hour as a proxy for system activity. |
Sources: services/automations/cron/health-score.py:20-30, apps/web/public/health.html:142-152
Observability Data Flow
The monitoring system utilizes the NeXify Cognitive Execution Layer (NCEL) to emit metrics to a central bus. Agents ping an orchestrator every 5 seconds to register heartbeats.
The diagram shows the flow from service execution to visualization, including the automated health scoring loop.
Sources: services/api/agents/brain_agents/monitoring-specialist.md:65-75, services/automations/cron/health-score.py:92-140
Monitoring Specialist Agent
The monitoring-specialist (agent_id: monitoring-specialist) is the autonomous authority for system observability. It operates on a "Brain-First" mandate, querying the Qdrant knowledge base for past lessons and credibility warnings before taking action.
Core Responsibilities
- Log Aggregation & Tracing: Collecting and correlating logs across distributed agent executions.
- Alerting: Implementing symptom-based alerts to minimize alert fatigue.
- Health Checks: Monitoring specific endpoints including Qdrant (
/healthz), FastAPI (/health), and Hermes Gateway.
Alerting Matrix
| Severity | Trigger | Channel |
|---|---|---|
| Critical | >5 min downtime | Immediate via Hermes Gateway Alert Channel |
| Warning | Degraded performance | Batched every 15 minutes |
| Info | System updates | Daily digest |
Sources: services/api/agents/brain_agents/monitoring-specialist.md:1-40
Quality Gates and Auditing
The senior-quality-auditor enforces System 8 (Qualitätsmanagement) mandates. No system reaches production without passing seven specific quality gates.
The Seven Quality Gates
- Documentation: Architecture specs, API docs, and SOP references must exist.
- Testing: Unit test coverage must be ≥80% for P0 systems.
- Security: Secrets must be in the vault; zero-trust must be enforced.
- Architecture: Alignment with the 12-System model and dependency maps.
- Performance: API latency <200ms p95; Agent response <5s.
- Workflow: Context enrichment from the Brain must be verified.
- Agent Quality: Profile size ≥4000 characters and heartbeats configured.
Sources: services/api/agents/brain_agents/senior-quality-auditor.md:20-68
Automated Gate Verification
The system automates the verification of performance, workflow, and agent quality gates by scraping /api/health metrics and scanning agent profiles.
The sequence shows how automated gates are checked before a deployment is approved.
Sources: services/api/agents/brain_agents/senior-quality-auditor.md:85-95
Inventory and Integrity Scanning
The inventory-brain-scanner acts as a structural guardian, cataloging every component of the NeXify AI system to detect inconsistencies before they become incidents.
Scan Domains
- Agent Inventory: Verifies all 34+ agent profiles for required blocks (Identity, Brain-First, Mission Alignment).
- Brain Integrity: Audits vector distribution in Qdrant and flags "stale" entries older than 7 days.
- Infrastructure Inventory: Tracks Docker container health, SSL certificate expiry, and resource pressure (RAM/CPU).
- Credential Inventory: Monitors 16 sets in the Data Vault for rotation status.
Sources: services/api/agents/brain_agents/inventory-brain-scanner.md:25-60
Admin Monitoring Endpoints
The NeXify AI Master routes provide administrative access to the platform's health and configuration status.
Status API
The endpoint GET /api/admin/nexify-ai/status aggregates connectivity data for the entire stack. It returns:
- OpenRouter: Connection status and active model selection.
- Qdrant: Collection list and connectivity.
- Supabase/MongoDB: Database heartbeats.
- System Resources: Disk usage percentage and memory pressure.
Sources: services/api/routes/nexify_ai_routes.py:348-410
Monitoring Tools
The platform provides a suite of administrative tools for observability, accessible via the execute-tool endpoint:
system_stats: Returns counts for contacts, leads, quotes, and projects.worker_status: Monitors background task executors.audit_log: Accesses historical action logs stored in Supabase.
Sources: services/api/tests/test_iteration_66.py:100-130, services/api/routes/nexify_ai_routes.py:160-180
Summary
The Monitoring & Observability system ensures the NeXifyAI platform remains autonomous and reliable. By combining specialized agents like the monitoring-specialist with strict Quality Gates and a weighted Health Score, the platform provides real-time transparency into the "Brain," infrastructure, and agent mesh. This multi-layered approach enables proactive incident prevention and maintains the "Founder Mandate" for production-ready AI operations.