Deployment: Kubernetes & Terraform
Relevant source files
The following files were used as context for generating this wiki page:
The deployment infrastructure for the prediction market platform is built on a modular architecture using Terraform for Infrastructure as Code (IaC) and Kubernetes for container orchestration. The system supports multiple cloud providers, including Google Cloud Platform (via GKE and Cloud Run) and DigitalOcean, ensuring high availability and scalability for the decentralized trading platform.
The deployment focus is on containerized web services, automated database migrations, and secure environment variable management. Infrastructure is partitioned into reusable modules, such as target-kubernetes and target-gke-autopilot, allowing consistent deployment patterns across different environments and providers.
Sources: [infra/terraform/modules/target-kubernetes/main.tf](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/main.tf), [infra/terraform/environments/production/gke/main.tf](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/environments/production/gke/main.tf)
Kubernetes Architecture
The Kubernetes deployment utilizes standard resources to manage the application lifecycle. The primary component is a kubernetes_deployment_v1 that manages a set of web replicas, supported by a Service for internal networking and an Ingress for external traffic.
Core Components
- Namespace: Isolates the application resources (default:
kuest). - ConfigMap & Secrets: Manage non-sensitive and sensitive environment variables respectively.
- Deployment: Controls the
webcontainer, ensuring the desired number of replicas (default: 2) are running with defined resource limits and health checks. - Service: A ClusterIP service exposing the application on port 80, targeting the container port 3000.
- Ingress: Handles external HTTP(S) traffic, supporting NGINX ingress class and TLS termination.
Sources: [infra/terraform/modules/target-kubernetes/main.tf:18-185](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/main.tf#L18-L185), [infra/terraform/modules/target-kubernetes/variables.tf:26-30](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/variables.tf#L26-L30)
Resource Flow Diagram
The following diagram illustrates how external traffic reaches the application pods and how configuration is injected.
Sources: [infra/terraform/modules/target-kubernetes/main.tf:46-150](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/main.tf#L46-L150)
Terraform Infrastructure Provisioning
Terraform is used to provision the underlying cluster infrastructure and the Kubernetes resources. The project follows a modular structure where environment-specific configurations (e.g., production/gke) call specialized modules.
GKE Autopilot
For Google Cloud, the system utilizes GKE Autopilot, which manages the underlying node infrastructure automatically.
| Parameter | Source | Description |
|---|---|---|
cluster_name | var.cluster_name | Name of the GKE cluster |
network_name | var.network_name | VPC network for the cluster |
release_channel | var.release_channel | GKE release channel (e.g., REGULAR) |
deletion_protection | var.deletion_protection | Prevents accidental cluster deletion |
Sources: [infra/terraform/environments/production/gke/main.tf:20-33](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/environments/production/gke/main.tf#L20-L33)
Resource Limits and Health Checks
The Kubernetes deployment defines specific resource constraints and probes to maintain service stability.
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1024Mi"
readiness_probe:
http_get:
path: "/"
port: "http"
initial_delay_seconds: 10
period_seconds: 10
Sources: [infra/terraform/modules/target-kubernetes/main.tf:97-124](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/main.tf#L97-L124)
Environment & Secret Management
The application requires a robust set of environment variables for blockchain interaction, authentication, and database connectivity. These are strictly validated during the Terraform plan phase.
Required Secrets
The secret_env map must contain specific keys, or the deployment will fail validation. The storage configuration requires either a Supabase profile or an S3 profile.
| Secret Key | Purpose |
|---|---|
BETTER_AUTH_SECRET | Authentication encryption |
POSTGRES_URL | Primary database connection string |
REOWN_APPKIT_PROJECT_ID | Wallet connection integration |
KUEST_API_KEY | Platform API credentials |
SUPABASE_URL | Supabase storage (Profile A) |
S3_BUCKET | S3 asset storage (Profile B) |
Sources: [infra/terraform/modules/target-kubernetes/variables.tf:38-66](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/variables.tf#L38-L66), [infra/terraform/modules/target-digital-ocean/variables.tf:112-140](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-digital-ocean/variables.tf#L112-L140)
Database Migrations & Task Scheduling
Post-deployment, the system uses a custom migration script (scripts/migrate.ts) to handle schema updates and configure database-level schedulers (e.g., pg_cron in Supabase).
Migration Flow
The migration script ensures that only one instance runs at a time using pg_advisory_lock and tracks applied versions in a migrations table.
Sources: [scripts/migrate.ts:169-232, 427-434](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/scripts/migrate.ts#L169-L232)
Scheduled Jobs
When running in Supabase mode, the script configures several cron jobs using pg_cron and pg_net to handle background platform tasks.
- sync-events: Periodic polling of event data.
- sync-volume: Market volume aggregation.
- sync-resolution: Checking for market outcomes.
- clean-jobs: Cleanup of processed or failed background tasks.
Sources: [scripts/migrate.ts:285-408](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/scripts/migrate.ts#L285-L408)
Deployment Summary
The infrastructure is designed for portability and resilience. By utilizing Terraform modules for Kubernetes, the platform can be deployed consistently across GKE, DigitalOcean App Platform, or any standard Kubernetes cluster. The integration of automated migrations and database-level scheduling ensures that the application state remains consistent with the code versioning during CI/CD cycles.
Sources: [infra/terraform/modules/target-kubernetes/main.tf](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/modules/target-kubernetes/main.tf), [infra/cloud-run/cloudbuild.yaml](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/cloud-run/cloudbuild.yaml), [infra/terraform/environments/production/digital-ocean/main.tf](https://github.com/kuest/prediction-market/blob/927a5c8fe56ae38a7e48ffea6be92b7f5e368d1a/infra/terraform/environments/production/digital-ocean/main.tf)