Background Jobs
Relevant source files
The following files were used as context for generating this wiki page:
Background Jobs in Noesis constitute a durable work orchestration system designed to handle non-blocking, asynchronous tasks such as reflection, evaluation, and capability authorship. These jobs are managed by an in-runtime coordinator that ensures idempotency, lease-based execution, and recovery from crashes. The system is built to operate under a specific risk-based autonomy model where background work is constrained by scoped grants and budgets.
The architecture separates the principal acting in the foreground from background principals like the reflector or evaluator. This separation ensures that background jobs cannot silently widen their own authority or modify protected kernel components. Jobs are persisted in a centralized WorkspaceStore using SQLite, which serves as the authoritative source for job states, leases, and completion records.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html, AGENTS.md, plans/noesis-first-complete-iteration.html
Job Lifecycle and Orchestration
Background jobs follow a strict lifecycle managed by the Automatic in-runtime coordinator. The lifecycle transitions through states such as hypothesis formation, authoring, preflight testing, and observation. The coordinator uses a durable queue with leases to prevent redundant execution and ensure that interrupted jobs can be recovered without repeating dangerous external effects.
Lifecycle States
The following diagram illustrates the typical progression of a background learning job:
The diagram shows the transition from initial signal detection through LLM reflection, preflight evaluation, and final activation or rejection.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html
Lease and Recovery Mechanism
Jobs are governed by durable leases and heartbeats. If a job crashes or the process restarts, the coordinator rehydrates the job state from SQLite. Idempotency is maintained by binding stable operation identities to specific principals and resources.
| Feature | Description |
|---|---|
| Durable Leases | Prevents multiple workers from picking up the same job simultaneously. |
| Idempotency Keys | Binds keys to principal, effect class, and resource to prevent duplicate side effects. |
| Budget Limits | Restricts background jobs to a specific estimated cost and attempt count. |
| Lease Expiration | Allows expired leases to be retried by the coordinator after a timeout. |
Sources: plans/noesis-autonomous-compounding-implementation-plan.html, plans/noesis-high-leverage-correction-plan.html, plans/noesis-first-complete-iteration.html
Technical Data Model
Background jobs are represented as BackgroundWorkItem types, which encompass various learning and maintenance operations. These are stored within the DurableJobStorePort in the workspace.
Core Data Structures
The system defines specific interfaces for tracking work and ensuring authority boundaries:
type BackgroundWorkItem =
| ReflectWork | AuthorRevisionWork | GenerateCasesWork | PreflightWork
| ActivateWork | ObserveWork | ResolveExperimentWork | ReviseWork | RevertWork;
interface StableOperation {
operationId: string;
idempotencyKey: string;
principal: Principal;
effect: EffectClass;
resource: string;
estimatedCost: number;
attempt: number;
}
Sources: plans/noesis-autonomous-compounding-implementation-plan.html, packages/domain/test/architecture.test.ts:318
Principal Separation
Different background tasks operate under distinct principals to enforce security boundaries:
- Reflector: May read traces and write proposals but cannot promote behavior.
- Evaluator: Runs sandboxes and records scores; isolated from the authoring principal.
- Scheduler: Manages job-scoped, expiring grants.
- Promoter: The only principal capable of activating qualified revisions via the protected control plane.
Sources: plans/noesis-first-complete-iteration.html, AGENTS.md
Authority and Security Rails
Background jobs are strictly prohibited from modifying the protected kernel or widening their own permissions. All side effects must pass through the EffectGateway and AuthorityBoundary.
Risk-Based Autonomy
The system applies different policies based on the nature of the background task:
- Low Risk: Read-only skills or learned prompt modules can be auto-activated after a successful preflight.
- Medium Risk: Routing changes among already-authorized tools require an open real-use experiment.
- High Risk: New write/network effects or credential access require explicit user approval.
- Prohibited: Modification of authority rules, workspace ownership, or execution boundaries is impossible for generated behavior.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html, AGENTS.md
Effect Reservation
Before any background job executes a dangerous external effect, it must reserve the grant use and cost in the authoritative operational state. If a reservation exists without an unambiguous completion record (e.g., due to a crash), the system fails closed rather than retrying blindly.
Sources: plans/noesis-high-leverage-correction-plan.html, AGENTS.md
Implementation Status (AC Series)
The implementation of Background Jobs was phased across several tasks (AC-08 through AC-12) to ensure a stable coordinator and lifecycle management.
- AC-08: Established the in-runtime coordinator, durable queue, and lease management.
- AC-09: Implemented preflight policy interpretation and atomic activation for background revisions.
- AC-10: Added continuous feedback attribution and automated
revertoutcomes for regressions detected by background monitors. - AC-12: Introduced the resident background worker (
noesis worker) which resumes the durable queue upon startup.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html