Skip to content

Just-in-time serving: tool-call triggers, a tighter prompt gate, and a measured useful rate - #134

Merged
emp3thy merged 16 commits into
mainfrom
feat/just-in-time-serving
Oct 6, 2026
Merged

emp3thy merged 16 commits into
mainfrom
feat/just-in-time-serving

Conversation

@emp3thy

@emp3thy emp3thy commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Summary

Serve memories at the moment they apply instead of dumping them at session start, and make the useful rate measurable per channel.

Spec: docs/superpowers/specs/2026-10-05-just-in-time-serving-design.md (measurements, design, assumptions). Plan: docs/superpowers/plans/2026-10-05-just-in-time-serving.md.

Why. The live ledger showed 12 useful ratings out of 63 rated serves (19%). Replaying the real prompts and tool calls of 11 rated sessions showed that every per-prompt or per-tool keyword gate saturates over a long session, that the session-start dump of general-scope rules accounted for 29 of the 51 ignored ratings, and that every useful rating came from a memory matching the activity of the moment. The prompt hook was also firing on peer-agent messages (571 firings against 261 typed prompts plus 296 peer messages).

What changes.

  • Trigger channel (PreToolUse). Memories carry triggers (tool:<Name>, skill:<name|prefix*>, bash:<text>, path:<text>, write:large), stored in a JSON column on both memory tables (migration 0019). The hook fires on every tool call, matches triggers with a pure matcher (services/triggers.py), renders hits with a Triggered by: line and logs them with exposure source trigger (migration 0020 widens the ledger CHECK). The once-per-session latch is gone: the full path measured 184 ms against 175 ms latched. An O_CREAT|O_EXCL claim per (session, kind, id) stops parallel hook processes serving the same memory twice. On non-sqlite backends the path returns before any backend work.
  • Prompt channel (UserPromptSubmit). Non-human prompts (prefix < or Another Claude session sent a message) are skipped. One evidence gate for both memory kinds: at least BETTER_MEMORY_CONTEXT_MIN_HITS (default 2) distinct prompt tokens that are not ubiquitous in the pool (DF_CAP 0.5, applied only when the pool has 4+ memories). Replay: 43% useful at 75% recall against 19% today. Agentcore's relevance_ranks gate is unchanged.
  • Bootstrap. Deferred mode renders the index line only and is now the default; legacy is opt-in.
  • Editing triggers. Management UI drawers for both kinds (gated by caps.supports_triggers), and a triggers parameter on memory.semantic_observe / memory.semantic_update.
  • Measurement. Diagnostics page: useful rate by channel, all time and last 15 rated sessions, with the useful-per-session guard. rated = useful + ignored + misled; overlooked shown separately.
  • Docs updated: website architecture, configuration, MCP tools, README.

Deployment note. Hooks run from the checkout, so they pick up the new code at once; the MCP server applies migrations 0019/0020 on its next restart. Until then bootstrap and the prompt channel degrade (logged to hook_errors) rather than crash.

Test plan

  • Full suite: 1807 passed, 19 skipped (browser tests need a Playwright Chromium install; test_installs_in_worktree needs a global git identity; both pre-existing on this machine).
  • Pyright clean on every changed module; ruff clean on touched files.
  • Fresh whole-branch review: 0 Critical, 3 Important (all fixed with RED→GREEN tests), Minors deferred and listed in the commit history.
  • Live verification on the owner's database: a Bash heredoc serves the heredoc pitfall with its trigger named; the planning skill serves the planning rules; a repeated call is deduplicated; exposures land with source trigger.

Deferred minors (from review)

  • Content is saved before an invalid trigger returns 400 in the UI and MCP update paths.
  • Malformed trigger cells are skipped silently rather than logged.
  • Migration 0020's table rebuild is not wrapped in a transaction (same as earlier rebuilds).
  • Spec's replayed-prompt regression fixture and clean-slate E2E trigger scenario not built.

🤖 Generated with Claude Code

emp3thy and others added 14 commits October 5, 2026 22:52
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e default

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…for both memory kinds

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… latch

Adds migration 0020 widening the exposure source CHECK to 'trigger'.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deferred is the default now and exposes nothing at session start; these
tests describe the legacy dump's exposure rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…act useful-rate denominator

- SeenStore.claim: O_CREAT|O_EXCL sentinel per (session, kind, id) so parallel
  hook processes cannot render the same memory twice
- PreToolUse returns before connect/build_backend on non-sqlite backends
- useful_rate_by_channel: rated = useful + ignored + misled; overlooked shown
  in its own column, outside the rate
- contextual_suppressed_floor is bumped by the prompt channel only

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Claude BugBot Analysis

Found 1 potential bug in this PR.

medium: 1

One real defect found: the new atomic claim sentinels are permanent, so memory re-injection after N turns (BETTER_MEMORY_CONTEXT_REINJECT_TURNS > 0) silently stops working. The migrations, trigger matcher, relevance gate and UI changes checked out. Part of the 4313-line diff was only skimmed: the tail is mostly tests and docs, and I read just the code and migration hunks closely.

@@ -1,25 +1,27 @@
"""UserPromptSubmit / PreToolUse hook: inject curated memories relevant to the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM: Permanent per-memory claim sentinel defeats BETTER_MEMORY_CONTEXT_REINJECT_TURNS

The hook now calls seen.claim(...) after filter_unseen. claim creates a context_seen_<session>.<kind>-<id>.claim file with O_CREAT|O_EXCL. The sentinel is never removed or expired, and prune_stale only deletes it after 7 days. filter_unseen still lets a memory through again once (turn - last) > reinject_turns, which is the documented BETTER_MEMORY_CONTEXT_REINJECT_TURNS behaviour. On that second pass claim hits FileExistsError, the memory is counted as lost, and it is dropped from items. So when context_reinject_turns > 0, a memory is served at most once per session and re-injection never happens. Fix: make the claim turn-aware, for example by including the injection epoch (turn // max(reinject_turns, 1)) in the sentinel name, or by removing the stale sentinel when filter_unseen re-admits the memory.

⚠️ Could not locate exact line in the diff. The bug is at better_memory/hooks/contextual_inject.py:342.

Additional Locations

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Claude BugBot Analysis

The previously reported claim-sentinel bug is still present: SeenStore.claim is not turn-aware, so reinjection never happens when context_reinject_turns > 0. No new bugs were found.

No bugs were detected in this PR.

…njection works

BugBot (PR #134): claims were permanent, so with
BETTER_MEMORY_CONTEXT_REINJECT_TURNS > 0 a memory re-admitted by
filter_unseen could never win a second claim. The sentinel name now
carries the turn the memory was last marked seen at in the caller's
snapshot; parallel callers in one window still share a filename.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@emp3thy

emp3thy commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Addressed the BugBot finding in
c05cb4f: claim sentinels are now keyed to the last-seen epoch (context_seen_<session>.<kind>-<id>.<epoch>.claim), so BETTER_MEMORY_CONTEXT_REINJECT_TURNS > 0 re-serves again after the window while parallel callers in one window still get a single winner. Tests: test_claim_reopens_after_reinject_window and test_trigger_reinjects_after_window.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Claude BugBot Analysis

The earlier claim-sentinel bug is fixed. SeenStore.claim now puts the memory's last-seen turn (the epoch) in the sentinel filename, so a memory re-admitted by BETTER_MEMORY_CONTEXT_REINJECT_TURNS gets a fresh sentinel and can win the claim again. I read only the first 868 of the 4,363 diff lines plus context_seen.py, so the rest of the diff was not reviewed for new bugs.

No bugs were detected in this PR.

@emp3thy
emp3thy merged commit 9d5be26 into main Oct 6, 2026
3 checks passed
@emp3thy
emp3thy deleted the feat/just-in-time-serving branch October 6, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant