Observability & Metrics System
Relevant source files
The following files were used as context for generating this wiki page:
The Observability & Metrics System in CodeBot is a multi-layered infrastructure designed to provide deep insights into system performance, health, and reliability. It combines automated OpenTelemetry (OTel) tracing, manual instrumentation for high-value logic, and a persistent metrics storage engine. This system enables real-time monitoring through Prometheus, persistent historical analysis via MongoDB, and smart alerting based on adaptive thresholds.
The system's scope includes tracking request latency, error ratios, and operation-specific performance
Relevant source files
The following files were used as context for generating this wiki page:
The Observability & Metrics System in CodeBot is a multi-layered infrastructure designed to provide deep visibility into application performance, error rates, and system health. It combines distributed tracing via OpenTelemetry, structured logging, and persistent metrics storage in MongoDB to enable real-time monitoring and historical analysis.
The system is primarily used by administrators through ChatOps commands to investigate latency spikes, classify errors, and monitor resource usage. It leverages adaptive thresholds to detect anomalies and provides a specialized profiler for database query performance.
Sources: GUIDES/observability_file_relationships.md, bot_handlers.py:596-758
System Architecture
The architecture consists of four primary layers: Configuration, Instrumentation, Storage, and Presentation. Data flows from incoming requests through instrumentation decorators, which emit traces to OpenTelemetry and metrics to a persistent database.
The diagram shows the dual-path for observability data: traces are exported via OTLP, while performance metrics are queued for bulk insertion into MongoDB.
Sources: GUIDES/observability_file_relationships.md, monitoring/metrics_storage.py:12-45
Distributed Tracing (OpenTelemetry)
The tracing system is built on OpenTelemetry (OTEL). It facilitates end-to-end visibility by correlating requests across different services. The setup_telemetry function initializes the tracer provider with resource attributes like service name and version.
Key Components:
- Trace Provider: Manages the lifecycle of spans.
- OTLP Exporter: Sends spans to a configured backend (e.g., Jaeger or Honeycomb) via gRPC or HTTP.
- BatchSpanProcessor: Optimizes performance by sending spans in groups rather than individually.
Sources: observability_otel.py:15-55
Manual Instrumentation with @traced
Developers use the @traced decorator to wrap critical functions. This automatically starts a new span, records the function name, and captures any exceptions as span events. It also handles Correlation IDs, linking logs to specific traces.
@traced(name="process_snippet", attributes={"category": "transform"})
async def process_snippet(code):
# Function logic...
pass
Sources: observability_instrumentation.py:10-48
Metrics Collection and Persistence
Unlike ephemeral Prometheus metrics, the metrics_storage module provides a persistent history of every request's outcome. It uses an internal asynchronous queue to prevent blocking the main application thread during database writes.
Data Flow for Metrics
- Enqueue: Metrics are added to a
dequeviaenqueue_request_metric. - Batch Processing: A background worker monitors the queue. When the batch size (default 100) or time threshold (5s) is reached, it triggers a bulk write.
- Database Write: Metrics are inserted into the
service_metricscollection in MongoDB.
Sources: monitoring/metrics_storage.py:20-85
Storage Schema
| Field | Type | Description |
|---|---|---|
request_id | String | Unique ID for correlation |
timestamp | DateTime | When the request occurred |
duration | Float | Request latency in seconds |
status | Int | HTTP status code or result |
endpoint | String | The path or command invoked |
Sources: monitoring/metrics_storage.py:110-140
Performance Profiling
The Query Profiler tracks the performance of specific database operations. It identifies "slow queries" that exceed defined thresholds and logs them with metadata for optimization.
Sources: monitoring/profiler_metrics.py:5-40
Administrative ChatOps
Administrators interact with the observability system directly through Telegram commands. These commands aggregate data from the service_metrics collection to provide instant snapshots.
Primary Commands:
/status: Provides a report for a specific time window (default 5m), showing total requests, error ratios (5xx), and latency percentiles (p50, p95, p99)./observe: A high-level overview of system health, including uptime percentage and active alerts./metrics: Generates a downloadable text file containing full Prometheus-formatted metrics./errors: Aggregates the top error signatures and provides Sentry links for deep-dive investigation.
Sources: bot_handlers.py:596-650, bot_handlers.py:821-945
Latency Percentiles Calculation
The system calculates percentiles using the aggregate_latency_percentiles function. This provides a more accurate view of user experience than simple averages by showing the "long tail" of performance.
Sources: monitoring/metrics_storage.py:180-210
Summary of Monitoring Features
| Feature | Implementation | Primary Source |
|---|---|---|
| Adaptive Thresholds | Mean + 3σ calculations | alert_manager.py |
| Adaptive Batching | 100 items / 5 seconds | metrics_storage.py |
| Error Classification | Regex-based signature matching | error_signatures.yml |
| Trace Correlation | TraceID/SpanID in logs | observability_instrumentation.py |
Sources: GUIDES/observability_file_relationships.md, monitoring/metrics_storage.py:40-60
The Observability & Metrics System ensures that CodeBot remains performant and reliable by providing the necessary tools for proactive monitoring and reactive debugging. By combining real-time tracing with persistent metrics, it allows the team to understand both immediate system behavior and long-term trends.