Skip to content

Catch writes from other connections and watermark syncs before trusting decimal provenance - #46

Merged
dbccpage merged 3 commits into
mainfrom
claude/project-thread-lc05y5
Nov 11, 2026
Merged

dbccpage merged 3 commits into
mainfrom
claude/project-thread-lc05y5

Conversation

@dbccpage

Copy link
Copy Markdown
Owner

🤖 Generated with Claude Code

This is a follow-up to #45, from the third external review of that PR (at 188213c). Two of its four findings survived the merge. Each reproduces on main, and the new tests fail there.

# Finding On main Fix here
P1 A write from another SQLite connection is invisible to total_changes, so a 0.2 committed elsewhere beside an admitted 0.1 summed to Decimal('0.3') Reproduced The audit also compares PRAGMA data_version, which moves on any other connection's commit. Exact-decimal eligibility is revoked for good, because the lost source types cannot be recovered, only refused
P1 set_watermark() synced the trusted counter without auditing first, so it absorbed an unobserved write. create_and_seed_table() did the same Reproduced Both public calls audit before they sync. apply_write_event syncs its own write, then writes its watermark through an internal path that does not audit
P1 Cold MIN/MAX converted floats through Decimal(str()) Fixed by #45's fourth commit One gap closed: SortedMultisetExtremumCache.uncacheable_reason is now set on the UNSUPPORTED outcome, as for AggregateCache
P2 An unregistered float GROUP BY escaped as a bare TypeError Fixed by #45's fourth commit Test added; it passes on main

Ownership contract. The adapter owns the substrate's writes.

  • SQLite: a write the adapter did not make is detected, from this connection or any other, and revokes decimal certification.
  • DuckDB: it has neither counter, so the guarantee covers adapter writes only. The substrate says so in its own comments.

Evidence (local, Python 3.13)

  • Full suite: .venv/bin/python -m pytest -q -p no:cacheprovider --ignore=tests/production_native.py exits 0.
  • Types and formatting: mypy smplcache reports no issues, and black is clean.
  • New tests: 8 tests in tests/test_source_provenance.py.
    • On main, 4 fail: the other connection, the watermark, the seed and the extremum mark.
    • On the reviewed commit 188213c, 7 of the 8 fail. The eighth, watermarked adapter writes staying exact, is a positive control.

Branch note. This branch reused #45's name after the merge. Its diff against main is only these 3 files, and the extra commits in the list are #45's pre-squash history.

https://claude.ai/code/session_0194kAmYv4pLMyrspzKEWExZ


Generated by Claude Code

… missed

A third external review found that provenance could still be lost. All four
findings reproduced (7 of the 8 new tests fail on 188213c; the eighth is a
positive control).

- Writes from another SQLite connection are invisible to total_changes. The
  audit now also compares PRAGMA data_version, so a 0.2 committed by another
  connection beside an admitted 0.1 is refused, not summed to 0.3.
- set_watermark() and create_and_seed_table() audit before they sync the
  trusted counters, so a public adapter call can no longer absorb an
  unobserved write. apply_write_event syncs its own write before writing
  its watermark through an internal, non-auditing path.
- Cold MIN/MAX no longer converts floats through Decimal(str()); like the
  row path it refuses them as float_not_exact (bool as invalid_evidence_type),
  and process_or_rebuild returns the UNSUPPORTED uncacheable outcome for
  extremum and top-k rebuild refusals too.
- An unregistered float GROUP BY value raises FloatNotExact, so it reaches
  the typed outcome instead of escaping as a bare TypeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194kAmYv4pLMyrspzKEWExZ
The third external review of the float PR (#45) found that two provenance
bypasses survived its merge. Both reproduce on main; the tests fail there.

- Writes from another SQLite connection are invisible to total_changes. The
  audit now also compares PRAGMA data_version, so a 0.2 committed by another
  connection beside an admitted 0.1 is refused, not summed to 0.3.
- set_watermark() and create_and_seed_table() audit before they sync the
  trusted counters, so a public adapter call can no longer absorb an
  unobserved write. apply_write_event syncs its own write first, then
  writes its watermark through an internal, non-auditing path.
- SortedMultisetExtremumCache carries uncacheable_reason, set when
  process_or_rebuild returns the UNSUPPORTED outcome for an extremum
  refusal and cleared by a clean rebuild, as AggregateCache does.

The review's MIN/MAX and float GROUP BY findings were already fixed on main
by #45's fourth commit; their tests here pass on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194kAmYv4pLMyrspzKEWExZ

Copy link
Copy Markdown
Owner Author

Live Database Containers is red, but the failure is not from this PR.

  • The failure: 1 failed, 56 passed. The failing test is tests/production_native.py::test_production_read_contention_during_publication[bounded], with "native source did not catch up within 30 seconds".
  • It is timing-flaky on main. It failed on 49a3a5c and d52e174 and on every Admit floats as values; refuse fake exactness as float_not_exact #45 head, and passed on 9e55b2f and 3a838a2.
  • Cause: the bounded worker drains about 13 generations/s against the writer's 20 commits/s, so the backlog takes 29 to 41 s against a 30 s limit. This PR touches only the cold-rebuild substrate (substrate.py, normal_forms.py), not the production worker.
  • Fix: none exists yet. It changes the publication protocol and waits on the maintainer's choice.
  • Not re-run: the repository's Actions minutes are limited.

Generated by Claude Code

@dbccpage
dbccpage marked this pull request as ready for review November 11, 2026 08:44
@dbccpage
dbccpage merged commit 6b8feae into main Nov 11, 2026
4 of 5 checks passed
@dbccpage
dbccpage deleted the claude/project-thread-lc05y5 branch November 11, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants