Telemetry & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Telemetry and Observability in Daytona provide a comprehensive framework for monitoring, tracing, and logging resources within sandbox environments. The system leverages the OpenTelemetry (OTEL) standard to collect and export performance data to various backends, such as Grafana, New Relic, and Jaeger. This infrastructure allows developers to gain deep insights into CPU, memory, and filesystem utilization, as well as application-level execution traces.
The observability stack is designed to be pluggable, supporting standard OTLP (OpenTelemetry Protocol) exporters. This enables the integration of external monitoring platforms by simply configuring OTLP endpoints and headers within the Daytona settings.
Sources: examples/otel-dashboards/grafana/README.md, docker/README.md:14-25
Architecture and Data Flow
The Daytona telemetry architecture follows a standard collector-exporter pattern. Metrics and traces are generated within the sandbox environments and sent to an OTLP-compatible collector or gateway.
The diagram shows the flow of telemetry data from the isolated sandbox environments to both internal APIs and external observability platforms.
Sources: examples/otel-dashboards/grafana/README.md, docker/README.md:23, libs/analytics-api-client/src/api/telemetry-api.ts
Metric Collection and Monitoring
Daytona collects specific resource utilization metrics for every active sandbox. These metrics follow the OpenTelemetry naming convention where dots are replaced by underscores in Prometheus-compatible backends.
Tracked Sandbox Metrics
| OTEL Metric | Prometheus Metric | Unit | Description |
|---|---|---|---|
daytona.sandbox.cpu.utilization | daytona_sandbox_cpu_utilization_percent | % | CPU usage (0-100) |
daytona.sandbox.cpu.limit | daytona_sandbox_cpu_limit_cores | cores | CPU cores limit |
daytona.sandbox.memory.utilization | daytona_sandbox_memory_utilization_percent | % | Memory usage (0-100) |
daytona.sandbox.memory.usage | daytona_sandbox_memory_usage_bytes | bytes | Absolute memory used |
daytona.sandbox.filesystem.utilization | daytona_sandbox_filesystem_utilization_percent | % | Disk usage (0-100) |
Sources: examples/otel-dashboards/grafana/README.md:113-125, examples/otel-dashboards/new-relic/README.md:65-73
Performance Thresholds
The system defines standard visual alerting thresholds for resource monitoring:
- CPU: Warning at 70%, Critical at 85%.
- Memory: Warning at 80%, Critical at 90%.
- Disk: Warning at 75%, Critical at 85%.
Sources: examples/otel-dashboards/grafana/README.md:104-110, examples/otel-dashboards/new-relic/README.md:58-62
API Interfaces for Telemetry
The Daytona Analytics API and Sandbox API provide endpoints for retrieving historical telemetry data. These interfaces allow the dashboard to fetch paginated logs, metrics, and trace summaries.
Telemetry API Endpoints
| Method | Endpoint | Description |
|---|---|---|
GET | /organization/{orgId}/sandbox/{sId}/telemetry/logs | Paginated logs with severity filters |
GET | /organization/{orgId}/sandbox/{sId}/telemetry/metrics | Gauge metrics in 1-minute intervals |
GET | /organization/{orgId}/sandbox/{sId}/telemetry/traces | Trace summaries and span counts |
GET | /organization/{orgId}/sandbox/{sId}/telemetry/traces/{tId} | Spans for a specific trace ID |
Sources: libs/analytics-api-client/src/api/telemetry-api.ts:44-245, libs/api-client/src/api/sandbox-api.ts:503-625
Sequence: Querying Metrics
Sequence illustrating the retrieval of sandbox metrics via the Analytics API.
Sources: libs/analytics-api-client/src/api/telemetry-api.ts:110-140
Visualization and Dashboards
Daytona provides pre-configured dashboard JSON templates for Grafana and New Relic. These dashboards are structured into several detailed views:
- Resource Overview: High-level view showing the total sandbox count, critical services, and a "Resource Pressure Score" (a weighted average of CPU, Memory, and Disk).
- CPU Details: Includes utilization timeseries, cores limits, and a usage heatmap.
- Memory Details: Visualizes absolute usage in GB and percentage patterns.
- Disk Details: Shows filesystem breakdown including available vs. used space.
Sources: examples/otel-dashboards/grafana/README.md:10-53, examples/otel-dashboards/grafana/dashboard.json:84-150
Configuration Parameters
Dashboards use template variables to allow dynamic filtering across large environments:
$datasource: Selects the Prometheus or OTLP provider.$service: Filters metrics by theservice_namelabel (corresponding to the sandbox ID).$interval: Sets the time aggregation (e.g., 1m, 5m, 1h).
Sources: examples/otel-dashboards/grafana/README.md:55-73, examples/otel-dashboards/grafana/dashboard.json:1150-1200
Distributed Tracing
Daytona supports distributed tracing, enabling developers to monitor the execution flow of code within sandboxes.
- Trace ID: A unique identifier for a single request or execution flow.
- Span: A single operation within a trace.
- Jaeger: Included in the local Docker Compose setup to provide a UI for inspecting these traces.
Sources: libs/analytics-api-client/src/api/telemetry-api.ts:24-30, docker/README.md:23
Summary
The Telemetry & Observability module provides the critical data layer for managing and troubleshooting Daytona sandboxes. By implementing the OpenTelemetry standard, it ensures compatibility with industry-standard tools while providing specific metrics for sandbox resource constraints. This allows for both real-time monitoring through dashboards and post-mortem analysis via historical logs and traces.