@pi/pods CLI Commands

Relevant source files

The following files were used as context for generating this wiki page:

• packages/pods/src/commands/pods.ts
• packages/pods/src/commands/models.ts
• packages/pods/src/commands/prompt.ts
• packages/pods/src/cli.ts
• packages/pods/src/config.ts
• packages/pods/src/ssh.ts
• README.md

The @pi/pods package provides a command-line interface (CLI) named pi for managing vLLM (a fast LLM inference and serving library) deployments on remote GPU pods. It simplifies the process of setting up remote environments, deploying models as Docker containers, and interacting with them. The CLI is structured into two main categories: pod management and model management.

This tool acts as a control plane, using SSH to execute commands on remote pods. It maintains a local configuration file to track pod details, including SSH commands, GPU information, and running models.

Sources: README.md, packages/pods/src/cli.ts:14-48

Command Structure

The pi CLI parses commands and arguments to route them to the appropriate handler for pod or model management. It also supports a global --pod <name> flag to override the currently active pod for any model-related command.

Sources: packages/pods/src/cli.ts:53-149

This diagram illustrates the initial command routing logic.

Pod Management

These commands are used to configure and manage the remote GPU pods where models will be deployed.

Sources: packages/pods/src/commands/pods.ts, packages/pods/src/cli.ts

pi pods setup

This command initializes a new pod, sets up the necessary environment, and saves its configuration.

The setup process involves:

1. Validating that HF_TOKEN and PI_API_KEY environment variables are set.
2. Testing the SSH connection to the pod.
3. Copying a setup script (pod_setup.sh) to the remote pod.
4. Executing the script on the pod to install Docker, NVIDIA drivers, vLLM, and other dependencies.
5. Detecting the GPU configuration on the pod.
6. Saving the pod's details to the local config.json file.

Sources: packages/pods/src/commands/pods.ts:40-149

The following flowchart shows the pod setup process.

Options:

Option	Description
<name>	A unique name for the pod.
<ssh>	The SSH command to connect to the pod (e.g., "ssh user@host").
--mount "<mount>"	The command to mount a shared file system on the pod.
--models-path <path>	The path on the pod where models are stored. If not provided, it's inferred from the --mount command.
--vllm <type>	Specifies the vLLM version to install. Valid types are release (default), nightly, or gpt-oss.

Sources: packages/pods/src/cli.ts:70-117

pi pods

Lists all configured pods, indicating the active one with an asterisk (*). The output includes the pod name, GPU details, vLLM version, and SSH command.

Sources: packages/pods/src/commands/pods.ts:13-35

pi pods active <name>

Switches the active pod to the one specified by <name>. All subsequent model commands will target this pod unless overridden with the --pod flag.

Sources: packages/pods/src/commands/pods.ts:154-167

pi pods remove <name>

Removes a pod's configuration from the local config.json file. This does not affect the remote pod itself.

Sources: packages/pods/src/commands/pods.ts:172-181

pi shell [<name>]

Opens an interactive SSH shell on the active pod or a specified pod.

Sources: packages/pods/src/cli.ts:151-182

pi ssh [<name>] "<command>"

Executes a single shell command on the active or specified pod and streams the output.

Sources: packages/pods/src/cli.ts:184-233

Model Management

These commands manage the lifecycle of vLLM models running as Docker containers on the active pod.

Sources: packages/pods/src/commands/models.ts

pi start

Starts a model on the active pod. The command constructs and executes a docker run command on the remote pod via SSH.

Sources: packages/pods/src/commands/models.ts:40-155

The following sequence diagram illustrates the pi start workflow.

Options:

Option	Description
<model>	The HuggingFace model ID to deploy (e.g., meta-llama/Llama-2-7b-chat-hf).
--name <name>	Required. A unique name for the running model instance.
--memory <percent>	The percentage of GPU memory to allocate (e.g., 90%). Translates to --gpu-memory-utilization.
--context <size>	The context window size (e.g., 4k, 16k). Translates to --max-model-len.
--gpus <count>	The number of GPUs to use. Translates to --tensor-parallel-size.
--vllm <args...>	Passes all subsequent arguments directly to the vLLM container, ignoring other options like --memory or --context.

Sources: packages/pods/src/cli.ts:241-294

pi stop [<name>]

Stops a running model container. If <name> is provided, it stops that specific model. If no name is provided, it stops all models running on the active pod.

Sources: packages/pods/src/commands/models.ts:160-214

pi list

Lists all running models on the active pod by executing docker ps and formatting the output.

Sources: packages/pods/src/commands/models.ts:219-278

pi logs <name>

Streams the logs from a specific running model container using docker logs -f.

Sources: packages/pods/src/commands/models.ts:283-298

pi agent <name>

Starts an interactive chat session with a deployed model using the @mariozechner/pi-coding-agent. This command acts as a bridge, constructing the necessary arguments to point the agent at the model's OpenAI-compatible API endpoint.

The promptModel function finds the running model's port on the pod and constructs the pi-agent command with the following key arguments:

• --provider openai
• --model <model_id>
• --api-key ${process.env.PI_API_KEY}
• --api-base http://<pod_ip>:<port>/v1

All other arguments are passed through to the pi-agent process, allowing for features like --continue to resume sessions.

Sources: packages/pods/src/commands/prompt.ts:12-68, packages/pods/src/cli.ts:323-339

Configuration

The CLI stores its configuration in ~/.pi/config.json. This file contains information about all configured pods and the models running on them.

Sources: packages/pods/src/config.ts:10-14

The structure of the configuration is as follows:

{
  "active": "my-pod-name",
  "pods": {
    "my-pod-name": {
      "ssh": "ssh user@host",
      "gpus": [
        {
          "id": 0,
          "name": "NVIDIA A100-SXM4-80GB",
          "memory": "81920 MiB"
        }
      ],
      "models": {
        "llama2-7b": {
          "id": "meta-llama/Llama-2-7b-chat-hf",
          "name": "llama2-7b",
          "containerId": "a1b2c3d4e5f6",
          "port": 8000
        }
      },
      "modelsPath": "/mnt/models",
      "vllmVersion": "release"
    }
  }
}

Sources: packages/pods/src/config.ts, packages/pods/src/commands/pods.ts:133-142

Environment Variables

The CLI relies on several environment variables for its operation.

Variable	Description
HF_TOKEN	Required for setup. Your HuggingFace token, used to download models on the pod.
PI_API_KEY	Required for setup. The API key used to secure the vLLM OpenAI-compatible endpoints.
PI_CONFIG_DIR	Optional. Overrides the default configuration directory (~/.pi).

Sources: packages/pods/src/cli.ts:44-48, packages/pods/src/commands/pods.ts:43-55