Getting Started & Operator Environment
Relevant source files
The following files were used as context for generating this wiki page:
The "Getting Started & Operator Environment" defines the foundational setup, configuration, and operational discipline required to work with the regime-detection engine. This project unifies V1 and V2 regime detection, requiring a strict environment that handles market data, macro-economic series, and event artifacts while ensuring deterministic results.
The environment is designed for three coupled product areas: a runtime classifier for building market contexts and features, a data acquisition system for collecting external artifacts, and a suite of operational scripts for calibration, profiling, and auditing.
Sources: pyproject.toml:9, TECH_DEBT_AUDIT.md:27-31
Development Environment Setup
The project requires Python 3.11 or higher and manages dependencies through a structured pyproject.toml. It utilizes several optional dependency groups to separate core engine logic from data acquisition and auditing tools.
Core Dependencies
| Dependency | Version | Purpose |
|---|---|---|
numpy | 2.3.0 | Numerical computing |
pandas | 2.3.0 | Data manipulation and time-series analysis |
pydantic | 2.13.4 | Data validation and wire models |
hmmlearn | 0.3.3 | Hidden Markov Model fitting |
pyarrow | 23.0.0 | High-performance Parquet storage |
Sources: pyproject.toml:11-25
Installation and Dependency Groups
Users can install specific environments using the following optional groups:
- s3: AWS S3 integration via
boto3. - browser: Web scraping/automation via
playwright. - data-sources: Market data providers like
alpaca-pyand datasets. - audit: Security and code quality tools including
pip-audit,pydeps, andvulture. - dev: Development tools including
pytest,ruff,pyright, andblack.
Sources: pyproject.toml:27-52
Operator Configuration & Controls
Operators interact with the system primarily through environment variables and configuration files. A critical component is the operator.env file, which contains environment-specific pointers and credentials.
Environment Variable Management
The system uses lazy accessor functions for environment-backed tunables to ensure that overrides (e.g., via pytest monkeypatching) are respected.
- Lazy Accessors: Validation (e.g., integer parsing, range checks) lives inside the accessor.
- Thread Management: To prevent CPU stall on multi-core systems during testing, the environment explicitly caps BLAS/OpenMP thread fanout to 1.
# Thread capping examples in tests/conftest.py
for _envkey in (
"OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS",
"VECLIB_MAXIMUM_THREADS", "NUMEXPR_NUM_THREADS", "LOKY_MAX_CPU_COUNT",
):
os.environ.setdefault(_envkey, "1")
Sources: AGENTS.md:128-133, tests/conftest.py:17-26
Operator Environment Pointers
Operational scripts utilize an OPERATOR_ENV_POINTER_FILE to resolve local repository credentials.
- Default Path:
~/.config/regime-detection/operator.env. - CLI Override: Use
--operator-env-file <path>in profiling and calibration scripts.
Sources: scripts/profile_engine.py:44-47, scripts/profile_engine.py:322-329
Operational Discipline (AGENTS.md)
The project enforces "Non-negotiable rules" to maintain technical integrity, especially when using coding agents or automated operators.
Core Rules
- Intent Announcement: Operators must state "what, files touched, tests, and expected output" before work begins.
- Secret Management: No secrets or raw request/response payloads are allowed in code or logs; they must be read from env or secret managers.
- Validation: Every layer boundary must be validated, specifically using source-specific models before data normalization.
- UTC Standard: All persistent state must be stored in UTC with timezones. Conversion to local time is restricted to display edges.
Sources: AGENTS.md:7-14, AGENTS.md:59-61
Execution Flow for Agents
The following diagram illustrates the lifecycle of a task performed by an agent or operator in this environment:
The diagram shows the mandatory sequence from intent announcement through to final CI verification.
Sources: AGENTS.md:7-30
Performance & Observability
The operator environment includes extensive profiling and observability tools to monitor the RegimeEngine.
Engine Profiling
The profile_engine.py script measures wall-clock stages of the classification window. It uses a StageTimer to capture durations across various components.
| Profiling Stage | Description |
|---|---|
build_market_context | Time to construct the SPY-based trading environment |
build_feature_store_total | Total time for calculating V2 features (HMM, Fragility, etc.) |
build_axis_series_bundle | Execution time for rule-based axis assessment |
build_regime_timeline_total | Final emission of the Pydantic timeline |
Sources: scripts/profile_engine.py:90-110, scripts/profile_engine.py:539-550
Observability Architecture
The project integrates OpenTelemetry and Sentry for deployment observability and error tracking.
The diagram represents how different observability components wrap the core engine execution.
Sources: scripts/profile_engine.py:55-68, pyproject.toml:23-24
Testing & Quality Gates
The environment enforces strict testing protocols, distinguishing between local "targeted" checks and full-suite verification in CI.
- Local Testing: Operators run "cheap targeted local checks" for debugging.
- CI Verification: Full-suite verification is performed in GitHub Actions.
- V2 Slice Gates: V2 components must pass a specific gate (
evaluate_v2_gate) before shipping to the main branch. - Type Checking: The project uses
pyrightfor static type checking, though enforcement is currently focused on configuration modules.
Sources: AGENTS.md:21-25, AGENTS.md:88-91, TECH_DEBT_AUDIT.md:120-125
Conclusion
The regime-detection environment is a disciplined ecosystem that prioritizes deterministic outcomes and strict data provenance. By adhering to the AGENTS.md operational rules and utilizing the provided profiling and auditing scripts, operators can maintain the stability of both V1 and V2 detection models while managing complex macro-economic and market data dependencies.