Observability Stack
Relevant source files
The following files were used as context for generating this wiki page:
The ClassMate Observability Stack is a comprehensive monitoring, logging, and tracing system designed to provide deep visibility into the application's performance and health. It utilizes the LGTM (Loki, Grafana, Tempo, Mimir/Prometheus) stack to handle the high-throughput data generated by the AI-powered meeting assistant, ensuring that 100k+ users can be supported with minimal downtime.
The stack is integrated directly into the infrastructure through containerized services, allowing for real-time tracking of API requests, AI worker processing times, and system resource utilization. Key performance indicators (KPIs) such as P95 latency and error rates are prioritized to maintain the quality of the transcription and note-generation services.
Sources: README.md:12-24, PROJECT_STRUCTURE.md:154-159, docker-compose.prod.yml:144-150
Architecture Overview
The observability architecture follows a distributed pattern where the application services (FastAPI Backend, Celery Workers) push telemetry data to a centralized monitoring layer. This layer consists of several specialized tools for different data types: Prometheus for metrics, Loki for logs, Tempo for traces, and Grafana for visualization.
Telemetry Data Flow
The following diagram illustrates how telemetry data flows from the core application components to the observability tools.
The FastAPI backend and Celery workers are configured to export metrics directly to Prometheus and traces via OpenTelemetry (OTLP) to Tempo. Logs are captured by Promtail from the container runtime and pushed to Loki.
Sources: README.md:15-32, docker-compose.prod.yml:144-215, infra/docker/docker-compose-monitoring.yml:110-150
Monitoring Components
Metrics Collection (Prometheus)
Prometheus acts as the primary time-series database for system and application metrics. It is configured to scrape data from the API and worker services at regular intervals.
- Endpoint:
http://localhost:9090 - Retention: 30 days
- Key Metrics Tracked:
http_requests_total: Tracks request volume and status codes.http_request_duration_seconds_bucket: Used for calculating P50, P90, and P95 latencies.active_recording_sessions: Monitors concurrent live sessions.ai_agent_requests_total: Tracks activity specific to AI processing agents.
Sources: docker-compose.prod.yml:153-167, infra/monitoring/grafana/dashboards/classmate-overview.json:16-56
Distributed Tracing (Tempo)
Tempo provides high-volume, low-dependency trace storage. The API service exports traces using the OpenTelemetry protocol (OTLP) to Tempo’s gRPC receiver.
| Feature | Configuration Value |
|---|---|
| OTLP Endpoint | http://tempo:4317 |
| Listen Ports | 3200 (HTTP), 4317 (gRPC), 4318 (HTTP) |
| Storage Path | /var/tempo |
Sources: docker-compose.prod.yml:79-79, docker-compose.prod.yml:205-215
Log Aggregation (Loki & Promtail)
Loki provides log aggregation inspired by Prometheus. Promtail runs as an agent that discovers logs from the Docker socket and ships them to Loki.
- Loki Config: Uses a local-config YAML for single-binary operation.
- Promtail Config: Configured to mount
/var/run/docker.sockto automatically label logs by container name.
Sources: infra/monitoring/loki/loki-config.yml:1-10, docker-compose.prod.yml:193-203, infra/docker/docker-compose-monitoring.yml:110-118
Visualization and Dashboards
Grafana is the central portal for the observability stack, providing a "ClassMate Overview" dashboard that aggregates data from all sources.
Key Dashboard Panels
The "ClassMate Overview" dashboard is organized into several strategic panels:
| Panel Title | Datasource | Expression / Query |
|---|---|---|
| Total Requests (1h) | Prometheus | sum(increase(http_requests_total[1h])) |
| Error Rate | Prometheus | sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 |
| AI Agent Duration | Prometheus | histogram_quantile(0.95, sum(rate(ai_agent_duration_seconds_bucket[5m])) by (le, agent_type)) |
| Transcriptions | Prometheus | sum(rate(transcriptions_total[5m])) by (status) |
Sources: infra/monitoring/grafana/dashboards/classmate-overview.json:16-118
Configuration and Deployment
The observability stack is deployed using Docker Compose, typically under a monitoring profile or within a dedicated production compose file.
# docker-compose.prod.yml snippet
grafana:
image: grafana/grafana:10.4.0
environment:
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-classmate123}
volumes:
- grafana_data:/var/lib/grafana
- ./infra/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
Sources: docker-compose.prod.yml:174-185, infra/docker/docker-compose-monitoring.yml:119-130
Conclusion
The observability stack is a critical component for maintaining the "production-grade" status of ClassMate. By leveraging structured logging through Loki and performance metrics via Prometheus, the system can automatically track the health of the AI transcription pipeline and provide developers with the necessary data to perform horizontal scaling of workers when demand increases.
Sources: README.md:12-14, README.md:88-91