Monitoring & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Monitoring and Observability within the SecondLayer platform is a multi-layered system designed to provide deep visibility into service health, AI model performance, and infrastructure stability. The system integrates real-time metrics collection, structured logging, and automated alerting to ensure the reliability of legal document analysis and MCP tool execution.
The observability stack primarily utilizes Prometheus for metrics ingestion, Grafana for visualization, and Winston-based structured logging for application-level event tracking. This infrastructure allows developers and operators to monitor critical paths such as LLM execution costs, API response latencies, and database connectivity across different deployment environments (Local, Dev, Stage, and Prod).
Core Architecture
The observability architecture follows a standardized pattern across all microservices, including mcp_backend, mcp_rada, and mcp_openreyestr. Services export metrics to Prometheus, which are then visualized through Grafana dashboards.
The diagram shows the flow of observability data from application services through specialized collection utilities to the infrastructure monitoring tools.
Sources: CLAUDE.md:65-75, DEPLOY_DEV_FIXES.md:70-82
Metrics Collection
The MetricsService serves as the primary interface for recording application-level telemetry. It tracks tool execution, API performance, and resource utilization.
Key Components
- MetricsService: A singleton service responsible for initializing and updating Prometheus counters, histograms, and gauges.
- Tool Execution Tracking: Monitors the success and failure rates of the 45+ MCP tools available in the system.
- LLM Cost Monitoring: Integrated with the
CostTrackerto record token usage and monetary costs associated with OpenAI and Anthropic API calls.
Primary Metrics
| Metric Name | Type | Description |
|---|---|---|
mcp_tool_execution_total | Counter | Total number of tool executions, partitioned by tool_name and status. |
mcp_tool_execution_duration_seconds | Histogram | Latency of tool execution. |
llm_api_cost_usd_total | Counter | Cumulative cost of AI model calls in USD. |
database_connection_pool_size | Gauge | Current number of active connections in the PostgreSQL pool. |
Sources: mcp_backend/src/services/metrics-service.ts:10-45, CLAUDE.md:105-110
Structured Logging
The system uses a Winston-based structured logging utility to ensure that logs are searchable and machine-readable.
The logging flow demonstrates how events are categorized by severity and persisted with specific retention rules.
Sources: mcp_backend/src/utils/logger.ts:5-25
Logger Configuration
The logger is configured to provide different outputs based on the environment. In development, it provides colorized console output, while in production, it emphasizes JSON-formatted files for ingestion into log aggregators.
- Max File Size: 10MB per file.
- Retention: Maximum of 3 files or 14 days of history.
- Metadata: Automatically includes timestamps and service names in every log entry.
Sources: mcp_backend/src/utils/logger.ts:30-50
Infrastructure Monitoring
Beyond application logic, the platform monitors the underlying data stores and container environment using specialized Prometheus exporters.
Monitoring Exporters
| Exporter | Port (Dev) | Description |
|---|---|---|
| Postgres Exporter | 9187 / 9188 | Monitors query performance and table statistics for backend and OpenReyestr DBs. |
| Redis Exporter | 9121 | Tracks cache hit/miss ratios and memory utilization. |
| Node Exporter | 9100 | Provides hardware metrics (CPU, RAM, Disk) for the gate server. |
Sources: DEPLOY_DEV_FIXES.md:74-80, deployment/LOCAL_DEVELOPMENT.md:85-95
Alerting and Visualization
Alerting Rules
Prometheus is configured with specific rules to trigger alerts when critical thresholds are crossed. These are defined in alert-rules.yml.
groups:
- name: secondlayer_alerts
rules:
- alert: HighToolFailureRate
expr: rate(mcp_tool_execution_total{status="error"}[5m]) > 0.1
for: 2m
labels:
severity: critical
- alert: DatabaseConnectionLimit
expr: database_connection_pool_size > 90
for: 5m
labels:
severity: warning
Sources: deployment/prometheus/rules/alert-rules.yml:1-20
Grafana Dashboards
Grafana provides visual representations of the system state through several pre-configured dashboards:
- MCP Unified Gateway Dashboard: Shows real-time tool usage, top failing tools, and total system cost.
- AI Performance Dashboard: Tracks token throughput and model-specific latencies.
- System Health Dashboard: Displays resource usage (CPU/RAM) across all Docker containers.
Sources: deployment/grafana/dashboards/, CLAUDE.md:140-150
Conclusion
Monitoring & Observability in SecondLayer is designed to provide actionable insights into both the technical health of the infrastructure and the operational efficiency of its AI features. By combining Prometheus metrics, structured Winston logs, and automated alerts, the platform ensures that legal document processing remains reliable and cost-effective.