feat(learn): attribute lesson recurrence from session corrections

Merged
AN
2d ago
+752-9

Issue for this PR

None (feature).

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

learn delivers approved lessons but has never checked whether a lesson that was shown still drew a correction on the same topic. This adds that check. It is deterministic and read-only, with no model calls.

Was → now:

  • Correction signals: had no turn timing → now carry optional correctedAt (the correcting message's creation time) and turnStartedAt, the start of the agent's response to the request being corrected (the first assistant message after the previous user message, read from session history). Synthetic continuations and compaction markers are not request boundaries; attachment-only turns are. A replayed request after compaction has unknown timing. The field is omitted whenever it can't be determined. Old records still parse. The capture eligibility shortcut is unchanged.
  • learn status (text and --json): now has a recurrence block, scoped to the --name store. For each lesson it shows sessions shown, matching corrections after exposure, mid-turn, unknown order, ambiguous, and corrections where only an earlier lesson text was shown. Lessons first shown after a correction are ignored for it. It also lists topics corrected in two or more sessions with no matching current lesson. Text output lists only lessons with correction evidence and summarizes the rest as "no observed correction" (it never says "followed"). If the analysis fails, status reports recurrence_error instead of failing.
  • Reflection prompt: a bullet gets r:N only when its current text was shown before a matching correction in two or more distinct sessions. The prompt asks for an EDIT only when the wording is unclear, and to keep a correct rule the agent simply didn't follow. Results are still staged candidates, so the approval and auto-promotion gates are unchanged.
  • HELPFUL votes (all reflections, not only r:N): before, feedback asking for exactly what a lesson says was voted HELPFUL even when the agent had ignored the lesson. Now HELPFUL requires the digest to show the agent followed it; a correct but missed lesson keeps its counters. Ignored lessons no longer climb the core ranking or offset harmful votes.

Attribution is conservative on purpose:

  • A lesson matches a correction on a full backtick anchor plus another content token, or on at least 3 lesson tokens at 60% coverage. Affix anchors (_at, stg_) never match alone.
  • A correction that matches several lessons the session was shown is ambiguous and counts toward nothing. A matching lesson the session never saw doesn't compete.
  • Evidence is keyed by store, lesson id and normalized text hash. An edited lesson, or the same id in another store, never borrows another's evidence.
  • Only user_correction signals are attributed. Analysis reads the newest 200 session snapshots and reports how much it covered.

No change to delivery, selection, promotion or signal consumption.

How did you verify your code works?

  • bun test test/altimate/learn: 2009 pass, 0 fail. bun run typecheck: clean.
  • New labeled tests cover: exposure before vs. during the turn (tool step → file lesson → final step); synthetic continuations; compaction markers and replayed requests; corrections that share an identifier but are about something else; ambiguity across stores, and lessons the session never saw; earlier versions; the two-session threshold; the history-to-boundary mapping from real part shapes; one history read per correction, with a retry after a failed write; an analyzer that never writes; status surviving a symlinked .sessions.
  • Docs: docs/docs/configure/learn.md (status section).
  • Marker check: no upstream-shared files touched.
  • Reviews: the independent review (2 medium, 3 low) and three bot rounds (Codex, Kilo, CodeRabbit, cubic) are fixed with regression tests through 86cc88fdc0. Fixes cover store scoping, ordering exposures against the correction message time, replay, attachment and compaction boundaries, exposures after the correction, earlier_version semantics, fail-closed snapshot reads, analysis outside the learn lock, prompt wording and docs. Three cost findings were declined with reasons in their threads: one history read per correction, one lstat per snapshot, and the full signals.jsonl parse that reflection already does. A local Codex review of head 86cc88fdc0 found no P0/P1; the follow-up prompt change in 2803138059 touches only prompt.txt and its test.
  • End-to-end on a GCP VM (PR head 86cc88fdc0 run from source; real altimate-code run sessions on the Altimate Base model; capture on):
    • Lesson shown before the turn, corrected in two sessions. Approved lesson L-0001 ("Convert *_at timestamp columns with to_utc in staging models."). Each session was shown it as core 17–19 ms before the agent's turn started. Each was corrected with No, convert the created_at timestamps with to_utc. Both signals carry real turnStartedAt/correctedAt. learn status --json reports L-0001: sessions_shown 2, recurred 2, during 0, unknown 0, ambiguous 0.
    • The hint reaches the model. learn reflect sent one request to the gateway (/v1/chat/completions, altimate-base). The recorded request body contains [L-0001] (h:0 x:0 r:2) Convert ... and the new r instruction. The model kept the rule (a HELPFUL vote); no edit was proposed.
    • Control, lesson only reaches the session mid-turn. With core, retrieved and request tiers at 0, the file hook delivered L-0002 23 s after the turn started. A correction then gives during 1, recurred 0.
    • HELPFUL vote A/B (fresh VM, head 2803138059). Two real sessions: one where the agent saw the lesson and didn't apply it, one where it applied it. Each was reflected twice with the old prompt and twice with the new one, with explicit feedback. Ignored lesson: old prompt HELPFUL, HELPFUL; new prompt EDIT (made the lesson more specific), no change. Followed lesson: HELPFUL on all four runs, old and new.
    • Found while testing and fixed separately: run re-quoted space-containing prompt arguments, so corrections sent as run "No, ..." were never captured (#1560, fixed in #1561).
  • Not exercised: whether r:N leads to better lesson edits over many sessions. That needs the learn benchmark, not a single run.

Cost side effects

None. No model calls, warehouse queries or cloud resources are added. The reflection prompt gains at most a few characters per lesson that has two or more recurrences. Analysis is local file reads, capped at 200 session snapshots.

Screenshots / recordings

N/A (CLI text output).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Changes correction signal shape, reflection voting semantics, and status output; behavior is local and read-only for recurrence, but it can alter which lessons gain helpful votes and how reflection proposes edits.

Overview
Adds lesson recurrence: a read-only, deterministic check of whether a delivered lesson still drew a user correction on the same topic, without extra model calls.

Capture now records optional turnStartedAt and correctedAt on user_correction signals by resolving turn boundaries from session history (synthetic continuations, compaction, and replay are handled conservatively).

learn status (text and --json) reports recurrence per lesson—recurred, during, unknown, ambiguous, earlier_version—from the newest 200 session snapshots plus local signals, plus up to five “unmatched” identifiers corrected in multiple sessions. Failures surface as recurrence_error without breaking the rest of status.

Reflection runs the same analysis before the model call: lessons with matching post-exposure corrections in ≥2 sessions get an r:N hint in the playbook. The reflector prompt is tightened so feedback that repeats an ignored rule no longer earns HELPFUL (missed-but-correct rules keep their counters; unclear wording can still drive EDIT).

New readShownSnapshots / attribute logic keys evidence by store, lesson id, and normalized text hash so edited lessons do not inherit another version’s counts.

Reviewed by Cursor Bugbot for commit e0b04c3d6d3cce6699517da5359596c4e8d6c993. Bugbot is set up for automated code reviews on this repo. Configure here.