CLI Command Reference
Relevant source files
The following files were used as context for generating this wiki page:
The Osmosis AI CLI is the primary interface for managing LLM training workflows, rubric evaluations, and remote rollout agent servers. It provides a suite of tools for authentication, workspace management, data previewing, and performance testing.
The CLI is built using Python's argparse library and follows a sub-command pattern. It supports environment variable loading via python-dotenv and integrates with the Osmosis Platform for remote registration and authentication.
Sources: osmosis_ai/cli.py:10-31, README.md:167-172
Core Command Architecture
The CLI uses a central entry point in osmosis_ai/cli.py that dispatches to specific command handlers. Each command is encapsulated in a class that provides a configure_parser method and a run (or handler) method.
The diagram shows the dispatch logic from the main entry point to specialized command modules.
Sources: osmosis_ai/cli.py:34-92, osmosis_ai/cli_commands.py:27-31
Authentication and Workspace Management
The CLI manages authentication through an OAuth-like flow that opens a local browser for platform login. Credentials are stored locally to allow persistent sessions across multiple workspaces.
Authentication Commands
| Command | Description | Key Arguments |
|---|---|---|
login | Authenticates with Osmosis AI platform. | --force, --no-browser |
logout | Revokes tokens and clears local credentials. | --all, -y |
whoami | Displays current user and active workspace. | N/A |
Sources: osmosis_ai/cli.py:39-55, osmosis_ai/cli_commands.py:149-165
Workspace Management Logic
The workspace command allows users to list, switch, and view their current active workspace. This enables developers to work across different organizational contexts within the same CLI session.
Rubric Evaluation and Data Preview
The CLI provides tools to validate and run rubric evaluations locally or against remote providers using LiteLLM.
Preview Command
The preview command allows inspecting YAML rubric configurations or JSONL dataset records before processing.
- YAML Support: Parses versioned rubric configs and displays parsed items.
- JSONL Support: Validates conversation records, including fields like
solution_strandoriginal_input.
Sources: osmosis_ai/cli_commands.py:33-66, osmosis_ai/cli_services/config.py:133-160
Evaluation Command (eval)
The eval command executes a batch evaluation of a dataset against a specific rubric.
The sequence diagram illustrates the flow of a batch evaluation request through the evaluation engine.
Sources: osmosis_ai/cli_commands.py:70-137, osmosis_ai/cli_services/reporting.py:236-267
Remote Rollout SDK Commands
These commands manage the lifecycle of a RolloutAgentLoop, facilitating the integration of agent frameworks with the Osmosis training platform.
Rollout Server Lifecycle
| Command | Description | File Path |
|---|---|---|
serve | Starts a FastAPI server for the agent. | osmosis_ai/rollout/cli.py:32 |
validate | Checks agent implementation (tools, async compatibility). | osmosis_ai/rollout/cli.py:126 |
test | Runs a local test suite against a dataset using LLMs. | osmosis_ai/rollout/test_mode/cli.py:108 |
Test Runner Logic
The test command supports both batch and interactive modes. Interactive mode allows for step-by-step execution of agent turns to debug logic and tool usage.
Reporting and Output Formats
The CLI generates detailed reports for both rubric evaluations and rollout tests.
Evaluation Reports
Evaluation results include:
- Overall Statistics: Average, variance, standard deviation, min, and max scores.
- Baseline Comparison: Deltas against a prior result file provided via
--baseline. - Record Details: Individual run logs, provider payloads, and durations.
Sources: osmosis_ai/cli_services/reporting.py:21-65, osmosis_ai/cli_services/reporting.py:145-180
Test Mode Output
The test command can emit a JSON summary including:
row_index: The dataset row being tested.success: Boolean status of the agent run.token_usage: Detailed prompt and completion token counts.reward: The final score assigned by the agent loop or judge.
Sources: osmosis_ai/rollout/test_mode/cli.py:325-353
The Osmosis CLI serves as a robust bridge between local agent development and platform-scale training. By providing unified authentication and standardized evaluation tools, it ensures consistency across the LLM training lifecycle.