Smart Alerting & Incident Management
Relevant source files
The following files were used as context for generating this wiki page:
Smart Alerting & Incident Management in CodeBot is a sophisticated observability system designed to detect system anomalies, manage alert lifecycles, and provide actionable insights for rapid incident remediation. The system utilizes adaptive thresholds, structured logging, and automated forwarding to various sinks such as Telegram and Slack to ensure high availability and performance.
The architecture is built on a multi-layered detection and analysis pipeline. It continuously monitors internal metrics like error rates and response latencies while also tracking external service health. Alerts are enriched with metadata, including Sentry links and "Quick Fix" suggestions, allowing administrators to investigate and resolve issues directly through integrated ChatOps commands.
Sources: alert_manager.py:1-25, GUIDES/ARCHITECTURE.md:83-118
## Core Architecture and Detection Logic
The alerting system operates on a rolling window of request samples. It maintains in-memory state of the last 3 hours of traffic and recomputes adaptive thresholds every 5 minutes using statistical analysis (mean + 3 standard deviations).
### Adaptive Thresholding
Thresholds are calculated for two primary metrics:
- Error Rate Percent: Percentage of internal requests resulting in 5xx status codes.
- Latency Seconds: Average duration of internal request processing.
The system applies "safety floors" to prevent noise in low-traffic scenarios and can be scaled via environment variables to widen the "normal" operating band.
The diagram shows the flow from metric monitoring through adaptive analysis to multi-sink alert delivery.
Sources: alert_manager.py:27-110, GUIDES/ARCHITECTURE.md:83-118
### Key Components
| Component | Responsibility |
|---|---|
AlertManager | Maintains sample buffers, recomputes thresholds, and triggers critical alerts. |
AlertForwarder | Routes alerts to external sinks (Telegram, Slack) and manages batching of anomalies. |
AlertsStorage | Persists alerts to MongoDB for historical analysis and dashboarding. |
Observability | Provides structured logging and event emission used as alert sources. |
Sources: alert_manager.py:126-150, alert_forwarder.py:1-20, monitoring/alerts_storage.py
## Alert Enrichment and Remediation
When a threshold is breached, the system generates a critical alert. These alerts are not mere notifications; they are enriched with deep context to facilitate immediate triage.
### Enrichment Metadata
Alerts are enriched with:
- Dominant Command/Endpoint: Identification of the specific feature causing issues.
- Resource Snapshots: CPU, RAM, and active request counts at the time of the breach.
- Database Health: MongoDB connection utilization and availability.
- Quick Fix Actions: Suggestions derived from runbooks, such as "Scale Up" or "Kill Slow Queries".
- Sentry Integration: Direct links to specific Sentry issues or request IDs.
Sources: alert_manager.py:461-558, services/observability_dashboard.py:734-850
### Remediation Workflow
CodeBot implements an automated remediation path for critical incidents (excluding "Drill" events). The remediation_manager is invoked to handle incidents based on the metric type and breach severity.
The sequence illustrates the coordination between detection, automated remediation, and final notification dispatch.
Sources: alert_manager.py:770-820, GUIDES/ARCHITECTURE.md:11-23
## Incident Investigation and Triage
For manual investigation, the system provides a Triage Investigation Flow. This allows administrators to use a request_id to build a timeline of events leading to an incident.
### Triage Data Gathering
The investigation service gathers:
- Metrics: Historical performance data.
- Logs: Structured log excerpts relevant to the timeframe.
- Traces: Distributed tracing IDs (Trace/Span ID).
- Related Requests: Other requests occurring in the same window or by the same user.
Sources: services/observability_dashboard.py:1571-1650, GUIDES/ARCHITECTURE.md:119-140
### ChatOps Commands
Administrators can interact with the alerting system via Telegram commands:
/status: Current health check and recent metric summary./alerts: List the 5 most recent internal alerts./triage <request_id>: Generate a summary investigation report./silence <pattern> <duration>: Temporarily suppress specific alerts.
Sources: alert_manager.py:867-885, observability.py:347-380
## Alert Forwarding and Suppression
The AlertForwarder acts as the gateway to external communication platforms. It respects "silences" configured in the database to prevent alert fatigue during maintenance or known issues.
### Forwarding Logic
- Normalization: Alert names and severities are standardized.
- Startup Grace Period: Performance alerts are suppressed for 20 minutes after system deployment/start to avoid noise during warmup
The following files were used as context for generating this wiki page:
- alert_manager.py
- alert_forwarder.py
- monitoring/alerts_storage.py
- services/observability_dashboard.py
- observability.py
- GUIDES/ARCHITECTURE.md
Smart Alerting & Incident Management within CodeBot is a sophisticated observability framework designed to detect system anomalies, manage lifecycle alerts, and facilitate rapid incident response. It utilizes adaptive thresholds based on historical data to minimize noise and provides enriched contextual data, such as Sentry links and "Quick Fix" suggestions, to assist developers in triage and remediation.
The system encompasses several layers: a Detection Layer that monitors error rates and latency, an Analysis Layer that computes adaptive thresholds ($mean + 3\sigma$), and a Delivery Layer that dispatches alerts to Telegram, Slack, and Grafana. It also includes persistent storage in MongoDB for auditing and a comprehensive "Incident Story" module for documenting major system events.
Sources: alert_manager.py:1-25, GUIDES/ARCHITECTURE.md:88-112
System Architecture
The alerting architecture is structured as a pipeline that moves from raw metric collection to intelligent notification delivery. It is integrated deeply with the core services of the bot and the webapp.
The diagram illustrates the flow from initial detection via monitors to the final delivery to notification sinks and persistent storage.
Sources: GUIDES/ARCHITECTURE.md:91-120, alert_forwarder.py:1-15
Adaptive Threshold Management
The system avoids static thresholds by maintaining a rolling 3-hour window of request samples. Every 5 minutes, it recomputes thresholds based on the mean and standard deviation ($3\sigma$) of the samples.
Threshold Calculation Logic
- Bucketing: Samples (status, latency) are grouped into per-minute buckets.
- Statistical Analysis: The system calculates the mean and standard deviation for internal error rates and average latencies.
- Threshold Setting: The threshold is set to $mean + 3 * \sigma$, multiplied by a configurable scale factor.
- Safety Floors: Minimum floors (e.g., 5% error rate or 1s latency) are applied to prevent zero-value thresholds from triggering excessive noise during periods of very high stability.
Sources: alert_manager.py:58-95, alert_manager.py:186-235
| Metric | Calculation Method | Default Scale | Safety Floor |
|---|---|---|---|
error_rate_percent | $mean + 3\sigma$ | ALERT_ERROR_THRESHOLD_SCALE | 5.0% |
latency_seconds | $mean + 3\sigma$ | ALERT_LATENCY_THRESHOLD_SCALE | 1.0s |
Sources: alert_manager.py:72-88
Alert Enrichment and Quick Fixes
When an alert is triggered, the alert_manager and observability_dashboard service enrich it with actionable intelligence. This reduces triage time by providing immediate paths for investigation or remediation.
Enrichment Components
- Contextual Metadata: Dominant commands, endpoints, and user IDs associated with the failure.
- Sentry Integration: Dynamic construction of Sentry search queries based on
request_idorerror_signature. - System Snapshots: Real-time CPU, memory usage, and MongoDB connection pool utilization.
- Quick Fixes: Automated suggestions derived from runbooks. Examples include "Scale Up" for resource exhaustion or "Kill Slow Queries" for database pressure.
Sources: alert_manager.py:461-512, services/observability_dashboard.py:535-645
Quick Fix Logic Flow
Enrichment logic determines the most relevant "Quick Fix" by correlating different system signals (e.g., combining high latency with high DB pool utilization).
Sources: services/observability_dashboard.py:586-630, alert_manager.py:557-610
Incident Storage and Audit
Every significant alert is persisted in MongoDB. This allows for unified counts across services, historical auditing, and the generation of "Incident Stories."
Alert Data Model
The storage layer utilizes an alerts_log collection with a 30-day TTL.
alert_id: Unique UUID generated at dispatch.alert_type: Normalized category (e.g.,high_error_rate,deployment_event).severity:critical,anomaly,warning, orinfo.details: A JSON blob containing the system snapshot at the time of the alert.silenced: Boolean flag indicating if the alert was suppressed by an active silence rule.
Sources: monitoring/alerts_storage.py:1-50, alert_manager.py:643-662
Silence Enforcement
The system supports temporary suppression of alerts using pattern matching on alert names.
- Pattern: Regex or exact match.
- Scope: Can be restricted to specific severities.
- Transparency: Silenced alerts are still recorded in the database but marked as
silenced=true.
Sources: alert_forwarder.py:522-545, alert_manager.py:633-640
External Integrations
CodeBot integrates with Sentry and Grafana to provide a comprehensive monitoring interface.
Sentry Polling and Polling Loop
The system polls Sentry for new issues and maps them to internal alert severities.
- Direct Links: Construct URLs pointing directly to the issue in the Sentry UI.
- Fingerprinting: Uses
error_signatureto group similar errors and avoid duplicate alerting.
Sources: services/observability_dashboard.py:1002-1025, alert_manager.py:734-780
Grafana Annotations
Critical alerts are sent to the Grafana API as annotations, allowing developers to see alert events directly on performance dashboards.
Sources: alert_manager.py:808-825
Conclusion
The Smart Alerting & Incident Management system in CodeBot provides a robust, self-healing observability pipeline. By leveraging adaptive thresholds and automated enrichment, the project ensures that alerts are both high-signal and actionable. The integration of persistent storage and documentation tools like "Incident Stories" ensures that every anomaly contributes to the long-term reliability and knowledge base of the system.
Sources: alert_manager.py:1-20, services/observability_dashboard.py:1-35