RunPod GPU Orchestration

Relevant source files

The following files were used as context for generating this wiki page:

• deploy/register_template.py
• deploy/launch.py
• scripts/runpod_control.py
• scripts/bootstrap_runpod.sh
• reports/2026-09-09-runtime/README.md
• README.md

RunPod GPU Orchestration manages the lifecycle of GPU-accelerated cloud environments for training and simulating neural connectome models. The system automates the provisioning of NVIDIA GPU instances, validates the runtime environment, and enforces strict budget and time limits for experimental runs.

The orchestration layer interfaces with the RunPod REST and GraphQL APIs to manage private Pod templates and active Pod instances. It ensures that experiments run in a consistent, pre-configured environment containing CARLA 0.9.16, CUDA, and the project's Python dependencies without requiring installation at startup.

Sources: README.md:25-27, deploy/register_template.py:1-4, reports/2026-09-09-runtime/README.md:1-5

System Architecture

The orchestration system consists of a local client controller, a template registration module, and an independent guard process.

Lifecycle Management Flow

The orchestration follows a strict sequence: template registration, pod launching, active monitoring, and termination.

The flow ensures that only validated images are promoted to templates and that all active pods are monitored by a local guard.
Sources: scripts/runpod_control.py:85-155, deploy/register_template.py:27-46

Key Components

Component	Description	Primary File
Template Registry	Promotes validated image digests into private RunPod templates.	deploy/register_template.py
Runpod Control	Interfaces with REST/GraphQL APIs and manages session state.	scripts/runpod_control.py
Independent Guard	A background process that monitors spend and deadlines to stop pods.	scripts/runpod_control.py
Bootstrap Script	Configures the environment inside the container upon start.	scripts/bootstrap_runpod.sh
Launch Utility	High-level CLI for starting new experiment sessions.	deploy/launch.py

Sources: scripts/runpod_control.py:101, deploy/register_template.py:7, scripts/bootstrap_runpod.sh:1

Template Management

The system uses private Pod templates to ensure immutable deployments. A template named Flyhard / CARLA 0.9.16 is used to store the specific container image digest and resource requirements.

Validation Requirements

Before an image is promoted to a template, the register_template.py script enforces the following checks:

1. Image Digest: Must be an immutable GHCR digest.
2. Runtime Checks: Requires passed status for cuda, mujoco, vtk, and carla.
3. Local Exports: Requires verified local exports before promotion.

Sources: deploy/register_template.py:27-37

Template Configuration

The template is configured with specific resource allocations to support the simulation workload:

• NVIDIA GPU: Typically A40 or A6000.
• Storage: 100GB container disk and 40GB persistent volume.
• Runtime Limit: Default FLYHARD_MAX_RUNTIME_SECONDS is set to 3600 (1 hour).

Sources: deploy/register_template.py:38-46, reports/2026-09-09-runtime/README.md:7-13

Execution Control and Guarding

The runpod_control.py module manages the active lifecycle of a Pod. It uses a session state file (work/runpod-session.json) to track the current experiment's parameters.

The Guard Process

When a Pod is launched, a background "watch" process is spawned. This process performs the following actions every 30 seconds:

1. Checks the current client balance.
2. Compares elapsed time against the deadline_epoch.
3. Evaluates total spend against the spend_cap_usd.
4. Forces a Pod stop if any limit is breached.

Sources: scripts/runpod_control.py:85-115

Budget Constraints

The orchestrator enforces strict financial safeguards:

• Minimum Credit: Requires at least $10 existing RunPod credit to launch.
• Hourly Limit: Hourly cost must not exceed $0.90.
• Spend Cap: Default session spend cap is $6.00.

Sources: scripts/runpod_control.py:126-134

Container Initialization

Upon Pod startup, scripts/bootstrap_runpod.sh initializes the environment. It handles:

1. SSH Configuration: Appends the supplied PUBLIC_KEY from environment variables to ~/.ssh/authorized_keys.
2. Workspace Setup: Links /workspace/flyhard to the home directory.
3. CARLA Server: Starts the CARLA simulator in offscreen mode using scripts/start_carla.sh.

Sources: scripts/bootstrap_runpod.sh:4-16

Operational Commands

The orchestration system is interacted with via scripts/runpod_control.py using specific commands.

Command	Action
balance	Retrieves current account balance and hourly spend.
launch	Starts a Pod based on a JSON config path.
status	Shows the current Pod status and remaining budget.
stop	Halts the GPU instance but retains storage.
terminate	Deletes the Pod and releases storage (requires verified exports).

Sources: scripts/runpod_control.py:160-178

The orchestration system ensures that high-cost GPU resources are used efficiently and that experiment results are safely exported before resources are terminated. Monitoring via the local guard process provides a safety net against API outages or accidental overruns.

Sources: scripts/runpod_control.py:86, reports/2026-09-09-runtime/README.md:37-41