Knowledge Base Curation

Relevant source files

The following files were used as context for generating this wiki page:

• src/curate.rs
• src/gates.rs
• src/worker.rs
• src/provider.rs
• src/main.rs
• src/search.rs

Knowledge Base Curation transforms raw coding session records into structured memory claims. The curation system identifies technical decisions, bug fixes, features, and preferences from interactions between developers and coding agents. This process occurs in the background as part of the system's worker loop, utilizing Large Language Models (LLMs) to analyze transcripts and apply validation gates to ensure accuracy and privacy.

Sources: src/curate.rs:1-12, src/worker.rs:313-350

Curation Architecture

The curation system operates as a phase within the oboete worker. It processes records in segments called windows, which are defined by token limits and session boundaries.

The diagram shows the lifecycle of a curation run, from window selection to final claim storage.
Sources: src/worker.rs:188-210, src/curate.rs:651-700

Windowing Logic

The next_window function calculates the boundaries for the next segment of records to curate. It respects a token budget (typically 5,000 tokens) and avoids splitting logical turns unless the record exceeds the budget.

• Token Budget: Windows are bounded by WINDOW_TOKENS to fit within provider ceilings.
• Turn Boundaries: The system attempts to end windows before a new user prompt.
• Record Limits: A single window covers at most 2,000 records to prevent unbounded processing of empty records (tombstones).

Sources: src/curate.rs:24-42, src/curate.rs:149-165

Curation Process Flow

Curation involves data preparation, model interaction, and post-processing.

1. Data Preparation

The system collects three types of information for the LLM prompt:

1. Window Text: The raw lines from the current session segment.
2. Candidates: Up to 20 existing claims that are relevant to the current window text, found via trigram search.
3. Carried Context: Session goals and recently proposed claims from previous windows.

Sources: src/curate.rs:739-780, src/curate.rs:1056-1100

2. Provider Interaction

The curate module uses a Provider::Chain to attempt curation. If the primary LLM provider fails (e.g., rate limits, timeout), the system falls back to the next configured provider.

Failure Type	Behavior
Rate Limit (429)	Wait for reset if short, otherwise cooldown for 45s.
Outage (5xx/Timeout)	Cooldown for 600s before retrying.
Budget Spent	Skip until next day or month reset.
Invalid Schema	Count as failure and move to fallback provider.

Sources: src/provider.rs:24-45, src/provider.rs:114-200

3. Verification and Storage

Once a provider returns drafts, the system applies "Code Gates" to validate the claims without LLM intervention.

The sequence shows the validation steps required before a claim is permanently stored.
Sources: src/curate.rs:788-820, src/gates.rs:76-150

Validation Gates

The gates module enforces rules to prevent "hallucinated" or unauthorized claims. It ensures that decisions are actually made by the user.

• Speaker Verification: Claims quoting a user prompt are attributed to the User. Claims quoting a tool or assistant are attributed to assistant proposal or tool result.
• Promotion to Decided: A proposed claim is only promoted to decided if the user explicitly accepts it (e.g., "Yes", "OK") or uses their own words to state it.
• Taint Check: If a proposal's text is heavily derived from tool output (paraphrased or pasted), it is "tainted" and cannot be promoted to decided without user confirmation.
• Supersedes Validation: A claim can only supersede existing claims in its own repository.

Sources: src/gates.rs:22-60, src/gates.rs:172-195, src/gates.rs:250-280

Data Structures

Window Op and Claims

Curation results are stored as operations in the raw.db op log.

Operation	Purpose	Key Fields
Window Op	Marks a range as curated	from_seq, to_seq, outcome, summary
Claim Op	Represents a structured fact	kind, status, body, quote, evidence

Sources: src/curate.rs:56-78, src/curate.rs:796-815

Claim Kinds

The system categorizes knowledge into specific types.

• decision: Explicit technical choices.
• bugfix: Identification and resolution of errors.
• feature: New capabilities implemented.
• discovery: Learned facts about the codebase.
• preference: Developer habits or project styles.
• change: Recorded modifications to the code.

Sources: src/curate.rs:641-648

Error Handling and Retries

The system maintains a pending table to track windows that failed curation.

1. Attempts: A window is tried up to 3 times.
2. Skipping: If all providers fail after 3 attempts, the window is marked skipped and the system moves on.
3. Recuration: Users can trigger manual recuration of skipped spans using oboete recurate --skipped.

Sources: src/curate.rs:636-640, src/curate.rs:680-705, src/main.rs:94-102

Summary

Knowledge Base Curation is a critical background process that distills developer-agent interactions into a persistent, structured memory. It uses a combination of window-based processing, LLM analysis, and rule-based validation gates to ensure that the resulting knowledge base is accurate, relevant, and respects the developer's authority. By integrating curation into the worker loop and providing fallback mechanisms, the system maintains a consistent memory state across diverse development environments.