Harden offline incidents and virtual eIM transitions

Merged
HD
12d ago
+189-17

Summary

This change hardens two reliability boundaries:

  • AWS-independent offline incidents: --no-bedrock incidents now receive a private provider-neutral Strands Model. Offline replay, trace validation, benchmark helpers, and replay-backed UI flows no longer initialize AWS. Accidental inference fails closed with the fixed sanitized error inference is disabled for direct-tool-only incidents.
  • Atomic virtual-eIM target handling: FleetTwin.enable_profile now validates uncached targets before proposal checks or operational mutation. Non-string targets raise TypeError("target must be a string"); unknown and already-active strings produce uncached, audit-visible refusals. Approval-keyed duplicate commands remain cache-first and return the identical result object without new events, profile changes, or simulated time.

The demonstrated benefits are reliability, dependency isolation, and atomic state handling, not measured performance. No comparable raw before/after performance evidence was collected, so this PR makes no performance-improvement claim.

Reproduced failures and fixes

  1. Offline execution consulted AWS. Before the change,
    env AWS_PROFILE=simshift-profile-that-does-not-exist DECIDER_URL=http://127.0.0.1:1 .venv/bin/python -m simshift.agent TRANSIENT --approve --no-bedrock --replay docs/sample-runs/TRANSIENT.json
    exited 1 with botocore.exceptions.ProfileNotFound because Incident always constructed a Bedrock model. Offline construction now uses _DirectToolModel; the enabled branch still obtains and retains the exact cached real BedrockModel.
  2. An invalid profile could partially mutate the twin. Before the change, a proposal-bound FleetTwin("BR_PROFILE").enable_profile("profile_C", proposal_id) raised KeyError after changing active_profile to an uninstalled profile and emitted no audit event. Uncached malformed, unknown, and active targets are now rejected before mutation. Unknown/active string refusals retain the existing single-event, 30-minute accounting and remain uncached.

Compatibility boundaries

The change preserves Python >=3.12, Strands Agents 1.61.1 and its provider-neutral Model interface, Pydantic 2, the standard-library loopback web stack, public module/CLI commands, enabled Bedrock model ID/region/token settings, advisory plan review and explanations, orchestrator behavior, approval/proposal/capability/owner/reachability precedence for valid targets, rollback behavior, routing prompts and thresholds, policy rules, six scenario outcomes, trace schema and recordings, benchmark records, HTTP/JavaScript contracts, dependencies, packaging configuration, and frontend behavior.

Published traces and docs/benchmark/benchmark-2026-10-19.json have no branch or working-tree changes. The complete diff contains only the seven planned source, test, and documentation files; inspection found no credentials, generated artifacts, cache files, or unrelated churn.

Verification evidence

Focused and full tests

  • Agent/orchestrator/web focus, Python 3.12: 58 passed, 1 skipped (the optional real-Decider integration).
  • Twin/policy focus, Python 3.12: 39 passed.
  • Full Python 3.12 suite: 147 collected; 146 passed, 1 skipped.
  • Full Python 3.13 suite: 147 collected; 146 passed, 1 skipped.
  • The only skip on each interpreter was the optional real-Decider test because no server was running.

All 12 published trace validations

All 12/12 validations reported ok:

  • docs/sample-runs/BR_PROFILE.json
  • docs/sample-runs/DE_OPERATOR.json
  • docs/sample-runs/FW_LOOP.json
  • docs/sample-runs/JP_INGEST.json
  • docs/sample-runs/OFFLINE.json
  • docs/sample-runs/TRANSIENT.json
  • docs/sample-runs/orchestrated/BR_PROFILE.json
  • docs/sample-runs/orchestrated/DE_OPERATOR.json
  • docs/sample-runs/orchestrated/FW_LOOP.json
  • docs/sample-runs/orchestrated/JP_INGEST.json
  • docs/sample-runs/orchestrated/OFFLINE.json
  • docs/sample-runs/orchestrated/TRANSIENT.json

Six deterministic replay routes and outcomes

  • BR_PROFILE: gather_evidence -> query_apn_session_app_diagnostics; gather_evidence -> query_firmware_modem_diagnostics; act -> prepare_guarded_profile_transition; restored=True.
  • DE_OPERATOR: gather_evidence -> query_apn_session_app_diagnostics; act -> simulate_operator_reselection; restored=True.
  • FW_LOOP: gather_evidence -> query_firmware_modem_diagnostics; gather_evidence -> query_apn_session_app_diagnostics; fallback -> defer_or_escalate; restored=False, safely escalated.
  • JP_INGEST: act -> query_apn_session_app_diagnostics; gather_evidence -> query_firmware_modem_diagnostics; fallback -> defer_or_escalate; restored=False, safely escalated.
  • OFFLINE: gather_evidence -> query_apn_session_app_diagnostics; fallback -> defer_or_escalate; restored=False, safely escalated.
  • TRANSIENT: act -> simulate_retry_with_backoff; restored=True.

CLI, HTTP, JavaScript, and distribution checks

  • Under hostile AWS_PROFILE=simshift-profile-that-does-not-exist, the documented TRANSIENT --approve --no-bedrock --replay command exited 0, identified no Bedrock, routed to simulate_retry_with_backoff, ended with restored=True, and contained no ProfileNotFound.
  • Replay-backed real loopback HTTP checks: 2 passed. They loaded the inspector page and exactly six scenarios, advanced BR_PROFILE to proposal-bound human approval, rejected another step with HTTP 409 while pending, accepted literal JSON true, and completed in restored SUCCEEDED state.
  • node --check simshift/static/app.js exited 0.
  • uv build produced exactly 1 sdist and 1 wheel. Wheel assertions found all 3/3 required assets: simshift/static/app.js, simshift/static/index.html, and simshift/static/style.css.
  • The existing setuptools warning that simshift.static is absent from explicit package configuration was observed and intentionally not changed.
  • Working-tree and origin/main...HEAD git diff --check commands exited 0; immutable fixture/benchmark diff checks also exited 0.
  • pyproject.toml, Makefile, and CI configure no lint command, formatter command, or static type-check command. These checks are recorded as absent rather than misrepresented as passing; Python tests/import execution, JavaScript parsing, package build, and whitespace checks are separate evidence.

No live Decider/Bedrock benchmark was run. The measured facts above support dependency isolation and state-boundary reliability; any future latency or recovery-quality claim requires comparable raw before/after measurements.

Deferred recommendations

  • Resolve simshift.static discovery and add artifact-content CI. Impact: medium packaging reliability. Evidence: the wheel currently contains all static assets, but setuptools emits an ambiguous package-discovery warning and CI does not build artifacts. Tradeoff: package-data/discovery policy must be chosen deliberately to avoid including unwanted packages or tests.
  • Test Python 3.12 in CI. Impact: medium lower-bound compatibility assurance. Evidence: metadata declares >=3.12, CI runs only 3.13, while local 3.12.13 and 3.13.13 both passed 146 tests with one skip. Tradeoff: a version matrix increases CI time and unpinned resolution may differ by interpreter.
  • Coordinate frontend requests and incident snapshots. Impact: medium UI robustness under rapid/concurrent requests. Evidence: start() checks but does not set busy, controls are not uniformly disabled, and snapshots do not take the incident lock. Tradeoff: naive locking can deadlock and stale-response handling needs dedicated browser/concurrency tests.
  • Validate router configuration at construction. Impact: medium reusable-pattern developer experience. Evidence: fallback/options/tools/read-only consistency is currently assumed. Tradeoff: new validation tightens the public API and may reject configurations that previously failed only at runtime.
  • Harden benchmark CLI validation. Impact: low-to-medium local developer experience. Evidence: unknown chooser names raise KeyError, zero samples fail late, and --stamp is joined into an output path. Tradeoff: this trusted local CLI work is unrelated to the fixed runtime boundaries.
  • Adopt narrowly selected quality automation separately. Impact: medium maintainability. Evidence: no lint, formatter, static type, JavaScript, or dependency-vulnerability check is configured. Tradeoff: adds dependencies/policy and can create cosmetic churn, especially around dynamic Strands tool wrappers.
  • Expand web security/logging only if deployment scope changes. Impact: high for remote production use but low for the declared loopback demo. Evidence: the app intentionally has no authentication, suppresses access logs, and may expose exception details, while binding to loopback and enforcing host/origin/content-type/body-size controls. Tradeoff: authentication, TLS, CSRF, logging, and rate limiting are a large architectural expansion.
  • Revisit model/policy outcome optimization only with comparable live evidence. Impact: potentially high but unproven. Evidence: the published benchmark notes firmware misses and synthetic/cross-hardware caveats; no comparable Decider/AWS environment was available. Tradeoff: tuning risks overfitting, changing recordings, and overstating performance; it requires held-out data, repeated runs, comparable hardware, and confidence intervals.