Monitoring & Observability
Relevant source files
The following files were used as context for generating this wiki page:
The Monitoring & Observability system in MCB (Memory Context Browser) provides real-time insights into system health, performance metrics, and operational status. It leverages a cloud-native stack comprising Prometheus for metric collection and Grafana for visualization, complemented by built-in health endpoints and an administrative UI.
This system ensures high availability and performance by tracking resource utilization, request latencies, and cache efficiency. It serves both automated scaling mechanisms in Kubernetes and manual inspection via the Admin UI.
Sources: k8s/README.md:95-103, README.md:15-20
Architecture and Data Flow
The observability architecture follows a standard pull-based model where Prometheus scrapes metrics from the MCB service. These metrics are then utilized by Grafana for dashboards and by the Kubernetes Horizontal Pod Autoscaler (HPA) for dynamic scaling.
The diagram illustrates the flow of metric data from the MCB service to monitoring and scaling components.
Sources: k8s/README.md:21-26, k8s/README.md:143-150, tests/e2e/admin-functional.spec.ts:68-80
Core Metrics
The system exposes a variety of metrics through the /api/context/metrics (or standard /health) endpoints. These are categorized into infrastructure, application performance, and Service Level Objectives (SLOs).
Infrastructure and Performance Metrics
| Metric Name | Description | Visualization Type |
|---|---|---|
mcp_cpu_usage_percent | Percentage of CPU utilization | Gauge |
mcp_memory_usage_percent | Percentage of Memory utilization | Gauge |
mcp_http_requests_total | Total count of HTTP requests handled | Graph (Rate) |
mcp_http_request_duration_seconds | Latency of HTTP requests (Histogram) | Graph (P95) |
mcp_active_connections | Number of currently active connections | Graph |
mcp_cache_hit_ratio | Efficiency of the internal cache | Gauge |
Sources: k8s/README.md:98-103, monitoring/grafana/dashboards/mcp-overview.json:28-115
SLO and Reliability Monitoring
MCB tracks Service Level Objectives (SLOs) to ensure the system meets its reliability targets.
- Uptime: Calculated using
avg_over_time(up{job="mcb"}[30d]), with a target of 99.95%. - Latency: Specifically monitors the P95 latency of the
/api/searchendpoint, aiming for under 200ms. - Error Rate: Measures the percentage of 5xx responses over a 30-day window, with a threshold for orange alerts at 0.1%.
Sources: monitoring/grafana/dashboards/mcp-overview.json:157-220
Admin UI and Health Checks
For manual observability and rapid debugging, MCB provides an Admin UI accessible via web browser. This interface interacts with specific JSON endpoints to display real-time status.
Health Endpoints
The system implements several health-related endpoints for different consumers:
- Liveness Probe:
/api/alive(scraped every 10s by K8s). - Readiness Probe:
/api/alive(scraped every 5s by K8s). - Health Status:
/health(returns JSON status for the Admin UI). - Jobs Status:
/jobs(returns counts of total, running, and queued tasks).
Sources: k8s/README.md:88-91, tests/e2e/admin-functional.spec.ts:68-100
Complexity Analysis
Beyond runtime metrics, the mcb-infrastructure layer provides deep code observability through the ValidationServiceInterface. This service calculates code quality metrics that can be viewed or used in CI gates.
fn calculate_maintainability_index(sloc: usize, avg_cyclomatic: f64) -> f64 {
if sloc == 0 {
return 100.0;
}
let raw = 171.0
- 5.2 * avg_cyclomatic.max(1.0).ln()
- 0.23 * avg_cyclomatic
- 16.2 * (sloc as f64).ln();
(raw * 100.0 / 171.0).clamp(0.0, 100.0)
}
Sources: crates/mcb-infrastructure/src/validation/service.rs:189-198
Kubernetes Integration
In a Kubernetes environment, the observability data is directly linked to the HorizontalPodAutoscaler (HPA).
- CPU-based scaling: Triggers when average utilization exceeds 70%.
- Memory-based scaling: Triggers when average utilization exceeds 80%.
- Custom Metric scaling: Can scale based on request rate (target 100 req/pod) or active connections (target 50/pod).
Sources: k8s/README.md:76-84, k8s/README.md:143-150
Summary
The Monitoring & Observability framework in MCB is a multi-layered system that provides both high-level SLO tracking and granular code-level analysis. By integrating standard tools like Prometheus and Grafana with custom Rust-based complexity analyzers and a dedicated Admin UI, the project ensures that developers and operators have full visibility into the server's lifecycle and performance.