Configuration Management
Relevant source files
The following files were used as context for generating this wiki page:
Configuration Management in the Osmosis SDK provides a structured approach to managing environmental variables, SDK settings, and rubric evaluation definitions. The system is split into two primary domains: the Rollout SDK configuration, which handles client and server operational parameters via Pydantic settings, and Rubric Configuration, which manages YAML-based definitions for LLM-based evaluation workflows.
The SDK utilizes python-dotenv to load environment variables and pydantic-settings (when available) for type-safe validation. For rubric evaluation, the system supports versioned YAML documents and discovery logic to locate configuration files within project directories.
Sources: osmosis_ai/rollout/config/settings.py:1-25, osmosis_ai/cli_services/config.py:126-146, README.md:12-25
SDK Settings and Environment Variables
The Osmosis Rollout SDK uses a hierarchical settings model managed by Pydantic. Settings are divided into client-side HTTP configurations and server-side operational limits. These settings can be populated through environment variables, .env files, or programmatic overrides.
Configuration Hierarchy
The system uses specific environment variable prefixes to avoid collisions:
OSMOSIS_ROLLOUT_CLIENT_*: Used forRolloutClientSettings.OSMOSIS_ROLLOUT_SERVER_*: Used forRolloutServerSettings.OSMOSIS_ROLLOUT_*: Used for globalRolloutSettings.
Sources: osmosis_ai/rollout/config/settings.py:10-20, osmosis_ai/rollout/config/settings.py:133-140
Rollout Settings Schema
The following table details the primary configuration options available in the Rollout SDK:
| Category | Parameter | Type | Default | Description |
|---|---|---|---|---|
| Client | timeout_seconds | float | 300.0 | HTTP request timeout (1.0 to 3600.0s) |
| Client | max_retries | int | 3 | Max retry attempts for 5xx errors |
| Server | max_concurrent_rollouts | int | 100 | Concurrency limit for agent rollouts |
| Server | record_ttl_seconds | float | 3600.0 | TTL for completed rollout records |
| Global | max_metadata_size_bytes | int | 1MB | Maximum size allowed for metadata payloads |
Sources: osmosis_ai/rollout/config/settings.py:44-90, osmosis_ai/rollout/config/settings.py:102-130, osmosis_ai/rollout/config/settings.py:146-151
Programmatic Configuration Flow
Developers can interact with the global configuration state using the get_settings(), configure(), and reset_settings() functions.
The diagram shows how the global settings singleton is initialized and can be programmatically overridden.
Sources: osmosis_ai/rollout/config/settings.py:157-200, osmosis_ai/rollout/config/__init__.py:1-25
Rubric Configuration Management
Rubric configurations define the parameters for evaluating LLM solutions against natural language rubrics. These are typically stored in YAML files and parsed into typed RubricSuite and RubricConfig objects.
Discovery Logic
The SDK employs a discovery strategy to find rubric_configs.yaml if a path is not explicitly provided. It checks the following locations in order:
- The directory containing the data file.
- The current working directory.
- The
examples/directory relative to the current working directory.
Sources: osmosis_ai/cli_services/config.py:126-146
YAML Schema and Versioning
The SDK supports versioned YAML documents. Currently, Version1RubricConfigSchema is implemented to handle standardized rubric definitions.
The class diagram illustrates the relationship between a suite of rubrics and individual configurations derived from source files.
Sources: osmosis_ai/cli_services/config.py:22-55
Default Values and Overrides
YAML documents can specify top-level defaults that apply to all rubrics within that document. These include default_model_info, default_score_min, and default_score_max.
version: 1
default_score_min: 0.0
default_score_max: 1.0
rubrics:
- id: support_followup
rubric: "Evaluate the response quality..."
model_info:
provider: openai
model: gpt-5-mini
api_key_env: OPENAI_API_KEY
Sources: examples/rubric_configs.yaml:1-15, osmosis_ai/cli_services/config.py:221-240
CLI Configuration Commands
The Osmosis CLI provides tools to preview and validate configurations before execution.
- Preview:
osmosis preview --path <file>validates YAML/JSONL files and echoes the discovered configuration blocks. - Evaluation:
osmosis eval --rubric <id> --config <path>executes runs based on the provided configuration. - Auth:
osmosis loginmanages workspace-specific credentials stored in~/.config/osmosis/credentials.json.
Sources: osmosis_ai/cli.py:35-80, README.md:144-165, osmosis_ai/cli_commands.py:40-70
Configuration Data Flow (Evaluation)
This flowchart describes the path from CLI input to the generation of evaluation results using parsed configurations.
Sources: osmosis_ai/cli_commands.py:82-140, osmosis_ai/cli_services/config.py:80-120
Conclusion
Configuration Management in the Osmosis SDK is designed for flexibility, supporting both high-level environment-based settings for infrastructure (Rollout SDK) and detailed, per-task configurations for LLM evaluations (Rubric Management). By utilizing Pydantic for validation and YAML for task definitions, the system ensures type safety while remaining accessible for development and production environments.