Monitoring & Dashboards
Relevant source files
The following files were used as context for generating this wiki page:
The monitoring system provides observability for the Financial Asset Relationship Database through metrics collection, log aggregation, and visual dashboards. It tracks API performance, graph rebuild lifecycles, and Service Level Objective (SLO) compliance to ensure system reliability. The infrastructure-as-code definitions support a standardized monitoring stack consisting of Prometheus for metrics, Loki for logs, and Grafana for visualization.
Sources: monitoring/README.md:1-5, ARCHITECTURE.md:162-171
Observability Architecture
The project implements a Phase 3 Observability Layer. This layer standardizes how the system emits telemetry data.
- Metrics: The system defines standardized Prometheus metrics in the API layer to track latency and error rates.
- Logs: Structured JSON logs are emitted for domain events, such as graph rebuild failures.
- Alerts: Prometheus and Loki rules trigger notifications based on performance thresholds or specific failure events.
This diagram shows the flow of telemetry data from the application to the monitoring tools and the alerting system.
Sources: monitoring/README.md:43-47, AUDIT_REPORT.md:255-260
Monitoring Components
The monitoring directory contains specialized subdirectories for alerts, dashboards, and provisioning.
| Component | Description |
|---|---|
| Alerts | Contains Prometheus and Loki rules for API latency, error rates, and rebuild failures. |
| Dashboards | Holds Grafana JSON exports for API health, graph rebuilds, and SLO compliance. |
| Provisioning | Includes auto-configuration files for Grafana data sources (Prometheus/Loki). |
Sources: monitoring/README.md:5-13
Dashboard Overview
The system provides three primary dashboards to visualize different aspects of the environment.
API Health and Performance
The api-overview.json dashboard monitors global API status. It tracks request volume, response latency, and error frequencies. This visibility helps maintain the performance of the FastAPI backend.
Sources: monitoring/dashboards/api-overview.json, ARCHITECTURE.md:37-53
Graph Rebuild Lifecycle
The rebuild-operations.json dashboard tracks the status of the asset relationship graph rebuild process. It monitors the duration of rebuild cycles and identifies lifecycle failures. This is critical for ensuring the relationship graph remains updated with the latest financial data.
Sources: monitoring/dashboards/rebuild-operations.json, monitoring/README.md:8-10
SLO Compliance
The slo-compliance.json dashboard tracks Service Level Objectives using multi-window burn rate alerts. It measures the consumption of the error budget to determine if the system meets its reliability targets.
Sources: monitoring/README.md:11-12, monitoring/README.md:31-41
Alerting and SLOs
The system implements the Google SRE standard for Multi-Window Multi-Burn-Rate Alerting. This method uses short and long windows to detect both sudden spikes and persistent issues while reducing alert noise.
- Fast Burn Alert (Critical): Triggers if the error budget consumption rate indicates the monthly budget will be exhausted within 24 hours.
- Slow Burn Alert (Warning): Triggers if the error budget consumption rate indicates exhaustion within 3 days.
Sources: monitoring/README.md:31-41
Setup and Provisioning
Grafana provisioning files enable the automatic loading of data sources and dashboards during deployment.
Data Source Configuration
The provisioning/datasources.yml file defines the connection strings and authentication for Prometheus and Loki.
# Example configuration from provisioning context
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
Sources: monitoring/provisioning/datasources.yml, monitoring/README.md:20-25
Dashboard Providers
To load dashboards automatically, you must create a provider file in the Grafana provisioning directory that points to the local dashboard JSON files.
Sources: monitoring/README.md:26-30
Conclusion
Monitoring and Dashboards are essential for maintaining the operational health of the Financial Asset Relationship Database. By integrating Prometheus metrics and Loki logs into centralized Grafana dashboards, the system provides real-time visibility into API performance and graph operations. The implementation of SRE-standard SLO alerting ensures that potential reliability issues are identified before they impact the monthly error budget.