Repository navigation
lab: D-GLOBAL-WHERE P1/P2 — criteria-aware WHERE, DR composition, Lance selective read, three parity verdicts - #1950
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…position (lab) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Routing arms A/B/C, working-set extension, presence-only updates, EXO-46 shared selection, multi-domain composition laws. F12 made non-vacuous (cross edges only); F26 fixture fits 255+1 ids per tile. Disable runs: dupcheck->F4, framecheck->F6/F7/F9, bindguard->F9 turn red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…aft, lab) where_k compiles; not yet run at 1M/10M. bench_where/compare_where smoke-tested at N=20000 against an independent 3VL oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ompare_big Same rule as the P0 fix (#1946): coverage is enforced in both directions, DuckDB results must cover exactly threads {1,4}, and any missing comparison exits 1 instead of becoming a note under an all-agree verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…anes, lab) Draft, NOT yet compiled or run: U/P/S layouts, packed 4x8 and sparse u16 encodings, refusals R1-R3, scalar oracle. Compile and measurement follow after the 10M where_k run finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Composes D working sets with the existing DR algebra (no paging engine, no corpus-wide adjacency). Stages charged separately. At N=1M: every composed answer equals the oracle; refusals TooMany, id-in-two-sets, checked_sum overflow and stale binding all fire with paired controls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…10M (lab) All answers equal the independent oracles at both sizes; all refusals fire. Warm values >= 1000 us are reported as FAILED-GATE, cold stages as COLD. DuckDB comparison follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
… crashing DuckDB answers some predicates (a tautology over min/max stats, e.g. c0<1000 OR c3<5) from table statistics: the plan is a constant Column Data Scan, no operator runs and no profile is written. Engine time is recorded as None with stats_folded=true; answers are still checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Corrected parity gate passes: every DuckDB count/EXISTS equals the oracle, coverage complete in both directions at threads 1 and 4. Analysis is parked pending the Lance 13 selective-read trace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…memory corrections probe (lab) ENGINEERING-PARITY.md: four DuckDB publications (research prototype vs shipped marked) mapped onto DR/DT64/ClassView/Quack/mask-risc/ndarray/Lance 13, plus the WHEN-vs-WHETHER recall comparison. EXISTS no-early-exit verified in code (mask-risc exec.rs:1633,1793-1794); production change not made. corrections.rs: route-once vs per-query, ndarray words vs hand-written packed compares (ordered u8-on-u32 = NO-EXISTING-KERNEL), borrowed vs gather_lane. Compiles; not yet run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…piled) Adds crates/crepe-plan-probe/examples/where_lance_arm.rs, which compares three reads on a permuted 16-property fixture: a full scan, a masked prefilter, and the #1633 attested range read. It also resolves the projection through ClassView, with refusal tests. The example now uses the addressed-read path instead of private copies. To allow that, addressed.rs makes `ranges` and `range_mask` public and adds `range_tree` (used to measure ingress bytes) and `check_fresh`, which `current_lane` now calls. Attest checks that `_rowid` equals `_rowaddr`. The D4 boundary prints a fixed BLOCKED_D4 message; no DR ordinal is composed with `_rowid`. This commit has not been compiled. A build is running; fixes follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
where_lance_arm now builds and runs at 1M. Every oracle check and every refusal check passes. Two fixes: - Restore the RowAddrTreeMap import, which the B arms need. - Write the fixture with max_rows_per_file = u32::MAX. Lance 13 compares that limit as `as u32` (write.rs:1077), so 1 << 40 truncated to 0. The 10M table then had one fragment per batch, and attest refused it as NotContiguous. That log is kept. The in-memory corrections (route once, ndarray kernels, borrowed reads) match the oracle at 1M and 10M. Results and logs are committed. Adds REPORT.md (three parity verdicts, the cost columns, the verdict list) and LANCE-TRACE.md (the source-backed trace). The 10M Lance rows are pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…CKED_D4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ed demand At 10M, every oracle check and every refusal check passes. The 1M pattern holds and grows: - a clustered 0.1% read is 114x faster than a full scan; - a scattered 50% read at k=16 is 11x slower; - ingress for 3.26M ranges is a 4.8 MB tree. New finding: ReadPolicy::DEFAULT decides on covered rows alone (addressed.rs plan_read). It therefore sends a single contiguous 10% range to a full scan: 3.8x slower at 1M, 4.9x at 10M. This is recorded as a probe item for the #1633 owner; the policy is not changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The crepe-plan-probe CI job runs `cargo fmt --check` and `cargo clippy --all-targets -- -D warnings`. This commit applies rustfmt to both files and removes the two warnings: the unused Env::n field, and an initial value of `read` that was overwritten before being read. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bb2c91b1e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
entries_index --check failed with DUPLICATE ID: D-GLOBAL-WHERE. The index takes the first D-id token in an entry's heading, so "D-GLOBAL-WHERE P2" and P0's "D-GLOBAL-WHERE P0" read as the same ID. The P2 heading is now D-GLOBAL-WHERE-P2 and its index row has been regenerated. Both checks pass: the entries index and the supersession index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The C arms timed two tree builds in the ingress stage: range_tree() to measure the serialized size, then range_mask(), which builds the tree again. That inflated the reported S-INGRESS cost. Both the B and C arms now build exactly one tree inside the timer. They measure its serialized size outside the timer (that is reporting, not ingress) and hand the same tree to RowAddrMask::from_allowed / from_block. The committed Lance-arm results predate this fix. They are being regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The 1M and 10M Lance arms were re-run on 90eb4cc. Every oracle check and every refusal check passes at both sizes. C-arm ingress roughly halves (1.8-2.9x at 1M); for 3.26M ranges it is now 123,162 us, down from 267,905 us. B-arm ingress is unchanged within noise, and mask byte counts are identical. REPORT.md tables, prose ratios and the cost table are regenerated from the logs. The board entry is updated to match. Figures previously written in ms are now in us. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ost)
The measurement data headed for main is reduced without dropping any
number that is cited or used:
- where_duckdb.json (1M, 10M): one record per line instead of one field
per line, 2,072 -> 116 lines each. The JSON round-trip is asserted
equal, and compare_where.py regenerates both compare_where.md files
byte for byte from the compacted files.
- Deleted the raw console logs:
- wk1m/wk10m/bw: their per-case data is in where_ours.json and
where_duckdb.json.
- cw: it is exactly the two compare_where.md files concatenated.
- proj1m/proj10m: their data is in project.json.
- results/run_summary.log keeps the one thing only those logs held: the
FAILED-GATE totals and the oracle verdict lines (353/648 and 23/41).
- Deleted the 10M EXPLAIN plans, which repeat the 1M ones. None was
cited.
Citations of cw.log now point at the same lines of compare_where.md.
The remaining ms figures in ENGINEERING-PARITY.md and LANCE-TRACE.md
are converted to us.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The merge auto-combined README.md, but the index was stale against the entries that main gained (D-DRAM-0, D-GRAMMAR-WHEN-1). Regenerated with entries_index.py --write: 569 rows, structurally sound. The supersession index is current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The in-memory probe (probe/), the DuckDB scripts (duckdb/) and all measurements (results/) now live only on research/d-global-where-p2 (commit 1422b2b); main does not get them. main keeps REPORT.md, ENGINEERING-PARITY.md, LANCE-TRACE.md, SPEC.md, the board entry and the crepe-plan-probe changes. Each doc says that its probe/, duckdb/ and results/ paths refer to the research branch, and links it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…G-MODEL Merges the #1950 P2 findings, the #1949 integration, the P3-1 and P3-3 results and the ndarray Abakus probe results (#389, #391, #402, microtile) into one findings / high-signal / fruits / open-ends entry. The operator's Abakus direction (repeated bit-plane folding as an ISA register contract) is recorded as WORKING-MODEL; P3-5 (k-of-n on the two shipped carry words) is its falsifier and is added to IDEAS as a high-signal fruit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Lab only. No production crate changes. Please don't merge automatically. S-INGRESS and D4 are still open, and nothing here decides them.
Start with
.claude/probes/2026-11-03-d-global-where-p1/REPORT.md. Two documents back it up:ENGINEERING-PARITY.md: the literature-first matrix.LANCE-TRACE.md: the caller path, traced against Probes: Lance 13 source pruning (D-TELEPORT-2) and 64 resident populations (D-PHOTO-0) #1489, D-HOLO H14c: thought-first positional reads, Hamming presence and Lance masked I/O #1573, D-HOLO H14d: logic → address mask → masked read → terminal (C only), measured on v4 #1575, D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin #1633, docs(D-DQM): DR64 ↔ DuckDB ↔ Lance 13 capability crosswalk (follow-up to #1787) #1788 and probe(D-DQM P1): chunk-touched predictor vs H14c/H14d Lance bytes — FAIL #1789 with source line references.What goes to
main(2,518 lines)REPORT.md,ENGINEERING-PARITY.md,LANCE-TRACE.md,SPEC.md, and the board entryD-GLOBAL-WHERE-P2.crates/crepe-plan-probe/examples/where_lance_arm.rs. It compares a full scan, an allow/block prefilter and the D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin #1633 attested range read, each with ClassView-driven column projection.crepe-plan-probe/src/addressed.rs):rangesandrange_maskare nowpub.range_tree. The example builds exactly one mask tree per read with it, and measures the tree's bytes outside the timer.check_fresh;current_lanenow calls it, so there is one freshness rule.What stays off
mainThe research branch
research/d-global-where-p2(commit1422b2b92) holds:probe/:falsify_p1,where_k,project,compose,corrections);duckdb/);results/).Each doc on
mainsays that itsprobe/,duckdb/andresults/paths refer to that branch.Results
Terminal::Anyhas no per-tile early exit.Lance findings:
ReadPolicy::DEFAULTdecides on covered rows alone. It sends a single contiguous 10% range to a full scan, which is 3.8× slower at 1M and 4.6× at 10M. Recorded only; the policy is not changed here.predicate_iri. It refuses AbsentField, Unauthorized, UnknownProperty, AmbiguousProperty and StaleSchema, each paired with a success control. An ordinal-mapping trap and a label-mapping trap prove a shortcut would be caught.BLOCKED_D4: arm C takes its ids as given. It composes no DR ordinal with_rowidand builds no third coordinate map.max_rows_per_filewithas u32(write.rs:1077), so1 << 40becomes 0. The D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin #1633NotContiguousguard caught this at 10M.Each verdict item in REPORT.md §8 is written as producer → missing edge → consumer → test → measured cost → owner.
Relation to #1949 (D-DRAM-0, documentation only)
This PR measures what #1949 plans:
replace_tile, and turning the guard off makes the stale binding pass.ENGINEERING-PARITY.md.Testing
falsify_p137/37.where_k,project,composeandcorrectionsmatch the oracle at 1M and 10M. The DuckDB parity scripts pass at 1M and 10M, threads 1 and 4.where_lance_armpasses every oracle and refusal check at 1M and 10M, re-run after the one-tree fix.crepe-plan-probe:cargo fmt --check,cargo clippy --all-targets -- -D warningsandcargo testpass locally (59 tests).main.🤖 Generated with Claude Code
https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF