Observability & Telemetry
Relevant source files
The following files were used as context for generating this wiki page:
Observability and Telemetry within the LiveCanvas ecosystem are designed to provide high-fidelity insights into the system's operational health, user behavior, and performance metrics. The architecture leverages the Elixir Telemetry library to instrument critical paths, ranging from authentication attempts to real-time media session orchestration. While the current development phase prioritizes product feature completeness, the underlying infrastructure is built to support robust monitoring across distributed Kubernetes pods.
The telemetry system focuses on three primary pillars: performance metrics (like database latency and message throughput), operational health (such as authentication failures and media errors), and auditability (tracking sensitive security events). This ensures that developers can correlate requests across users, sessions, and channels without compromising sensitive data.
Sources: ARCHITECTURE.md:270-285, ARCHITECTURE.md:21-25
System Monitoring Architecture
LiveCanvas utilizes a unified telemetry approach across its modular monolith architecture. Each pod in the Kubernetes cluster runs an identical release, emitting telemetry events that cover GraphQL requests, REST webhooks, and Phoenix Channels.
The diagram shows the internal flow of telemetry data from the application logic to specific monitoring outputs within a Kubernetes pod.
Sources: ARCHITECTURE.md:73-81, ARCHITECTURE.md:264-278
Critical Instrumentation Points
The system prioritizes instrumentation for high-impact functional areas. Key flows are decorated with telemetry hooks to capture both success and failure states.
| Category | Instrumented Events | Purpose |
|---|---|---|
| Authentication | Login success/fail, credential link/unlink, token revocation | Security auditing and abuse control |
| Live Sessions | Session start, join, and end events | Usage analytics and session lifecycle tracking |
| Real-time | Channel join failures, message throughput | Troubleshooting WebSocket connectivity |
| Infrastructure | Database latency, media session errors | Performance bottleneck identification |
Sources: ARCHITECTURE.md:235-238, ARCHITECTURE.md:273-280
Development and Client-Side Telemetry
During development, the system exposes additional telemetry and debugging tools to the client. This includes the ability to stream server-side logs directly to the browser console and simulate network conditions to test system resilience.
Phoenix LiveView Debugging
The frontend liveSocket configuration allows for real-time interaction with the telemetry layer from the web console.
// expose liveSocket on window for web console debug logs and latency simulation
// >> liveSocket.enableDebug()
// >> liveSocket.enableLatencySim(1000)
window.liveSocket = liveSocket
Sources: assets/js/app.js:33-38
Live Reloader and Log Streaming
In development environments, a dedicated reloader enables server log streaming to the client. This facilitates immediate feedback during the implementation of new features or when diagnosing complex OTP process interactions.
The sequence diagram illustrates the initialization of server log streaming to a developer's browser in a development environment.
Sources: assets/js/app.js:46-55
Security and Audit Logging
The telemetry strategy includes a strict "no-leak" policy for secrets. Logs and telemetry events are structured to allow correlation between a request, user, and session without persisting raw token secrets or recoverable passwords.
Sensitive Event Auditing
The system maintains auditable records for sensitive authentication events to detect and mitigate abuse:
- Auth Challenges: Captured via
BeginAuthChallengemutations to track potential enumeration. - Identity Changes: Tracking when users link or unlink providers (Google, Apple, Passkeys).
- Token Lifecycle: Monitoring the issuance and revocation of refresh tokens.
Sources: ARCHITECTURE.md:235-238, mobile/schema.graphql:27-50
Operational Health
LiveCanvas uses standard Kubernetes probes (Liveness and Readiness) to monitor the health of the BEAM (Erlang Virtual Machine). These checks ensure that pods are not only running but are capable of handling traffic by verifying database connectivity and the state of internal supervision trees.
Failure Handling and Recovery
Telemetry plays a vital role in the system's "failover" strategy. When transient states in OTP or Presence are lost, telemetry helps identify if the state can be reconstructed or if a session should be ended explicitly.
- Durable Facts: Sourced from PostgreSQL.
- Ephemeral State: Reconstructed from OTP/Presence/PubSub where practical.
Sources: ARCHITECTURE.md:264-268, ARCHITECTURE.md:286-292
Summary
The Observability & Telemetry system in LiveCanvas is a multi-layered framework designed for both operational stability and security. By integrating Elixir Telemetry with Kubernetes health checks and client-side debugging tools, the backend ensures that developers can monitor the health of live sessions, audit security-critical authentication flows, and maintain high performance across the modular monolith. This infrastructure supports the project's long-term goal of scaling while maintaining a strict boundary between durable facts in Postgres and transient state in the BEAM.