CLI Services & Sessions
Relevant source files
The following files were used as context for generating this wiki page:
The CLI Services and Sessions module provides the core infrastructure for the Osmosis AI command-line interface. It handles command routing, authentication flows via local servers, rubric configuration parsing, and evaluation session management. This system ensures that users can interact with the Osmosis Platform securely and perform complex LLM evaluation workflows with strict type enforcement and reliable session handling.
Sources: osmosis_ai/cli.py:1-40, osmosis_ai/cli_commands.py:46-133
Architecture and Command Routing
The CLI is built on a modular architecture using argparse to route user inputs to specific command handlers. Each command (e.g., login, eval, preview) is encapsulated in a class that configures its own parser and executes its logic.
The diagram shows how the entry point delegates responsibilities to specialized command handlers and underlying services.
Sources: osmosis_ai/cli.py:30-74, osmosis_ai/cli_commands.py:1-20
Key Command Handlers
| Command | Class | Description |
|---|---|---|
login | LoginCommand | Authenticates with Osmosis AI using an OAuth browser flow. |
preview | PreviewCommand | Inspects and pretty-prints YAML rubrics or JSONL data. |
eval | EvalCommand | Runs rubric evaluation sessions against datasets. |
workspace | WorkspaceCommand | Manages active workspaces and environment context. |
Sources: osmosis_ai/cli.py:41-76, osmosis_ai/cli_commands.py:23-440
Authentication Sessions
Authentication is managed through a local HTTP server that captures OAuth callbacks. This allows the CLI to securely receive tokens from the Osmosis Platform after a browser-based login.
OAuth Callback Flow
The sequence illustrates the interaction between the local server and the platform to secure a session token.
Sources: osmosis_ai/auth/local_server.py:28-110, osmosis_ai/cli_commands.py:159-215
Local Auth Server Configuration
The LocalAuthServer searches for available ports within a pre-defined range to avoid conflicts.
| Parameter | Range/Value | Source |
|---|---|---|
| Port Start | LOCAL_SERVER_PORT_START | osmosis_ai/auth/local_server.py:238 |
| Port End | LOCAL_SERVER_PORT_END | osmosis_ai/auth/local_server.py:238 |
| Callback Path | /callback | osmosis_ai/auth/local_server.py:28 |
| Health Path | /health | osmosis_ai/auth/local_server.py:30 |
Evaluation Session Management
The EvaluationSession handles the lifecycle of a rubric evaluation run. It integrates dataset loading, config discovery, engine execution, and result reporting.
Session Lifecycle Logic
- Discovery: Locates the rubric configuration file near the data path if not explicitly provided.
- Loading: Parses the Rubric Suite and JSONL records.
- Execution: Invokes the
RubricEvaluationEnginefor the specified number of samples. - Reporting: Calculates aggregate statistics and writes the result to a JSON file.
Sources: osmosis_ai/cli_commands.py:84-133, osmosis_ai/cli_services/config.py:233-255
# osmosis_ai/cli_commands.py:108-115
request = EvaluationSessionRequest(
rubric_id=rubric_id,
data_path=data_path,
number=number,
config_path=Path(config_path_value).expanduser() if config_path_value else None,
output_path=Path(output_path_value).expanduser() if output_path_value else None,
baseline_path=Path(baseline_path_value).expanduser() if baseline_path_value else None,
)
Configuration and Schema Parsing
The system uses a versioned schema approach to parse rubric YAML files. The RubricConfigParser selects a schema based on the version field in the document.
Rubric Suite Data Model
| Field | Type | Description |
|---|---|---|
rubric_id | str | Unique identifier for the rubric. |
rubric_text | str | Natural language instructions for the judge. |
model_info | dict | Provider and model configuration (e.g., LiteLLM). |
score_min | float | Minimum possible score (default 0.0). |
score_max | float | Maximum possible score (default 1.0). |
Sources: osmosis_ai/cli_services/config.py:22-31, osmosis_ai/cli_services/config.py:157-194
Configuration Hierarchy
The parser supports defaults defined at the document level which are overridden by individual rubric entries.
Shared CLI Utilities
The module includes several utility functions for data processing and statistical analysis used across different sessions.
- Statistics Calculation: Computes mean, variance, standard deviation, min, and max for evaluation scores.
- Text Processing: Collapses and previews long strings for CLI output.
- Fragment Collection: Recursively traverses complex message objects to extract textual snippets for display or analysis.
Sources: osmosis_ai/cli_services/shared.py:29-45, osmosis_ai/cli_services/shared.py:65-156
Statistical Data Structures
# osmosis_ai/cli_services/shared.py:38-44
return {
"average": average,
"variance": variance,
"stdev": std_dev,
"min": min(scores),
"max": max(scores),
}
Summary
The CLI Services & Sessions module provides a robust foundation for the Osmosis SDK, managing everything from the initial login via local_server.py to complex evaluation sessions orchestrated by EvalCommand. By enforcing strict schema validation in config.py and providing comprehensive statistical utilities in shared.py, it ensures that LLM training workflows are both reliable and measurable.