Release Operations & Runbooks
Relevant source files
The following files were used as context for generating this wiki page:
Release Operations for LiveCanvas encompass the deployment, maintenance, and emergency procedures required to operate the Phoenix-based modular monolith. The system is designed to run as a single Elixir release within Kubernetes, where every pod is capable of handling GraphQL requests, REST webhooks, WebSockets, and media orchestration.
The operational strategy prioritizes product feature completeness while maintaining strict architectural boundaries. Key operational areas include deployment gating, runtime failover drills, staged rollouts, and data governance.
Sources: ARCHITECTURE.md:14-22, ARCHITECTURE.md:83-88, README.md:14-20
Infrastructure & Deployment Model
The LiveCanvas backend is architected as a modular monolith deployed via Kubernetes. Each pod runs an identical Elixir release, utilizing OTP for process supervision and Phoenix PubSub/Presence for distributed coordination.
Deployment Architecture
The system relies on a single persistent datastore (PostgreSQL) and external object storage for media binaries. Ephemeral state is managed in-memory across the cluster.
The diagram shows the distributed nature of the Phoenix pods and their shared dependencies on PostgreSQL and object storage.
Sources: ARCHITECTURE.md:42-48, ARCHITECTURE.md:83-93
Operational Components
| Component | Responsibility | Technical Implementation |
|---|---|---|
| Phoenix Endpoint | Primary entry point for HTTP and WS | Phoenix.Endpoint |
| GraphQL API | Client data interface | Absinthe / Relay-first |
| Realtime Transport | Chat and session events | Phoenix Channels / WebSockets |
| Media Orchestration | WebRTC and live streaming | Membrane Framework |
| State Coordination | Distributed presence and state | Phoenix PubSub / Presence |
Sources: ARCHITECTURE.md:83-100, AGENTS.md:13-14
Release Runbooks
The project maintains a set of standardized runbooks for managing the lifecycle of a release and responding to production incidents.
Deployment Gates
Before a release is promoted to production, it must pass defined deployment gates. These ensure that architectural conventions, such as boundary enforcement and type safety, are maintained.
- Boundary Validation: Running
mix compileto check for cross-context boundary violations. - Type Checking: Executing
mix typecheckto verify public function typespecs. - Security Review: Ensuring SHA3-based token hashing and IDOR protection in GraphQL resolvers.
Sources: README.md:15, AGENTS.md:10-14, ARCHITECTURE.md:65-67
Rollback and Restore
In the event of a failed deployment or data corruption, the rollback procedure involves reverting the Kubernetes deployment to the previous known-good image tag.
- Durable Facts: Since PostgreSQL is the sole system of record, restoration focuses on database point-in-time recovery if migrations cannot be safely reverted.
- Ephemeral State: Because transient state is kept in supervised processes, a pod restart or rollback will naturally clear ephemeral state, requiring clients to reconnect.
Sources: README.md:18, ARCHITECTURE.md:46-48, ARCHITECTURE.md:181-185
Staged Rollout
New features, particularly those involving the Live or Chat contexts, are rolled out incrementally to monitor impact on system latency and database load.
The flow represents the staged progression from canary deployment to full production scale.
Sources: README.md:17, ARCHITECTURE.md:175-180
Monitoring & Health
Operational health is determined by the status of the BEAM (Erlang Virtual Machine) and the connectivity to the primary database.
Critical Metrics
Instrumentation via Telemetry is used to track the following high-priority indicators:
- Authentication: Success/failure rates of auth challenges and token issuance.
- Live Sessions: Session start/join/end events and media negotiation failures.
- Realtime: WebSocket message throughput and channel join failures.
- Persistence: Database latency and repo operation errors.
Sources: ARCHITECTURE.md:175-180, mobile/schema.graphql:30-60
Failure Handling & Recovery
The system is built for resilience, emphasizing client-side recovery and server-side supervision.
This sequence illustrates how clients recover after a pod failure by re-fetching authoritative data from the persistent store.
Sources: ARCHITECTURE.md:181-188, mobile/src/config/runtime.contract.ts:10-15
Security Operations
Security operations focus on secret management and identity protection within the GraphQL API.
- Secret Rotation: Token secrets are never stored in plain text; only
secret_hash(SHA3) is persisted in theuser_tokenstable. - Authorization Gating: Ownership and visibility rules must be re-applied in every GraphQL node fetcher to prevent Insecure Direct Object Reference (
CWE-639) vulnerabilities. - Identity Integrity: Identity linking rules prevent the silent attachment of credentials to incorrect user accounts.
Sources: AGENTS.md:10-15, ARCHITECTURE.md:139-145, mobile/schema.graphql:160-170
Data Governance & Compliance
The project defines procedures for managing user data throughout its lifecycle, including export and deletion requests.
| Operation | Implementation | Description |
|---|---|---|
| Data Export | DataExportRequest | Users can request a JSON-formatted export of their account data. |
| Account Deletion | AccountDeletionRequest | Supports a grace period before a scheduled purge occurs. |
| Audit Logging | Auth Events | Audit records for sensitive events like credential linking/unlinking. |
Sources: README.md:19, mobile/schema.graphql:255-275, ARCHITECTURE.md:155-159
The release operations and runbooks for LiveCanvas ensure a reliable, observable, and secure environment for the backend modular monolith, leveraging the BEAM's native strengths for fault tolerance and realtime communication.