Skip to content

lab: D-GLOBAL-WHERE P1/P2 — criteria-aware WHERE, DR composition, Lance selective read, three parity verdicts - #1950

Merged
AdaWorldAPI merged 24 commits into
mainfrom
ccr-85acaff0-oszq40-global-where-p1
Nov 3, 2026
Merged

AdaWorldAPI merged 24 commits into
mainfrom
ccr-85acaff0-oszq40-global-where-p1

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Lab only. No production crate changes. Please don't merge automatically. S-INGRESS and D4 are still open, and nothing here decides them.

Start with .claude/probes/2026-11-03-d-global-where-p1/REPORT.md. Two documents back it up:

What goes to main (2,518 lines)

  • Docs: REPORT.md, ENGINEERING-PARITY.md, LANCE-TRACE.md, SPEC.md, and the board entry D-GLOBAL-WHERE-P2.
  • Lance arm: crates/crepe-plan-probe/examples/where_lance_arm.rs. It compares a full scan, an allow/block prefilter and the D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin #1633 attested range read, each with ClassView-driven column projection.
  • Lab crate edit (crepe-plan-probe/src/addressed.rs):
    • ranges and range_mask are now pub.
    • New range_tree. The example builds exactly one mask tree per read with it, and measures the tree's bytes outside the timer.
    • New check_fresh; current_lane now calls it, so there is one freshness rule.
    • The example reuses these helpers instead of copying them. No new read API.

What stays off main

The research branch research/d-global-where-p2 (commit 1422b2b92) holds:

  • the in-memory probe (probe/: falsify_p1, where_k, project, compose, corrections);
  • the DuckDB 2.0.0 harness (duckdb/);
  • all raw measurements (results/).

Each doc on main says that its probe/, duckdb/ and results/ paths refer to that branch.

Results

verdict finding
Semantic parity: reached Every answer equals the oracle and DuckDB at 1M and 10M. Every Lance arm equals the oracle and the full-projection oracle at 1M and 10M.
Performance parity: split Count wins, from 14.2× down to 1.96×. EXISTS loses at 10M, 0.43× and 0.16×, because Terminal::Any has no per-tile early exit.
Engineering parity: reached on representation, not on strategy Missing wiring: EXISTS early exit, and routing once per frame. Missing capability: choice of evaluation direction, and an ordered u8-in-u32 compare word.

Lance findings:

  • Bytes follow page locality, not selectivity.
    • A clustered 0.1% read is 112× faster than a full scan at 10M.
    • A scattered selection reads 33–100% of the bytes, and at 10–50% it is 1.4–11× slower than a full scan.
  • ReadPolicy::DEFAULT decides on covered rows alone. It sends a single contiguous 10% range to a full scan, which is 3.8× slower at 1M and 4.6× at 10M. Recorded only; the policy is not changed here.
  • ClassView → column resolution goes through predicate_iri. It refuses AbsentField, Unauthorized, UnknownProperty, AmbiguousProperty and StaleSchema, each paired with a success control. An ordinal-mapping trap and a label-mapping trap prove a shortcut would be caught.
  • BLOCKED_D4: arm C takes its ids as given. It composes no DR ordinal with _rowid and builds no third coordinate map.
  • RowAddrMask ingress is a measured copy, never zero-copy.
    • It costs 0.4–5.4 µs for one range and 123,162 µs for 3.26M ranges.
    • These are the figures after the Codex fix. The first run timed two tree builds per C read and doubled them.
  • Lance 13 truncates max_rows_per_file with as u32 (write.rs:1077), so 1 << 40 becomes 0. The D-SRAM-2 spike: masked Lance 13 reads and the addressed semijoin #1633 NotContiguous guard caught this at 10M.

Each verdict item in REPORT.md §8 is written as producer → missing edge → consumer → test → measured cost → owner.

Relation to #1949 (D-DRAM-0, documentation only)

This PR measures what #1949 plans:

  • Family L (locator reuse): C1 shows that routing once pays from Q ≥ 2. The stale guard refuses a binding after replace_tile, and turning the guard off makes the stale binding pass.
  • Family A (Abakus thresholds): C2 shows there is no ordered u8-in-u32 compare. Composing the existing ternary words costs one pass per 2-bit prefix, up to 5× slower than a hand loop. This bears on docs(D-DRAM-0): DR locator memoization and adaptive Abakus mask lens (architecture gates) #1949's unverified kernel signatures.
  • Family C (DuckDB engineering parity): see ENGINEERING-PARITY.md.

Testing

  • On the research branch: falsify_p1 37/37. where_k, project, compose and corrections match the oracle at 1M and 10M. The DuckDB parity scripts pass at 1M and 10M, threads 1 and 4.
  • where_lance_arm passes every oracle and refusal check at 1M and 10M, re-run after the one-tree fix.
  • crepe-plan-probe: cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test pass locally (59 tests).
  • Board checks pass locally: the entries index (569 rows), the supersession index, and citation decay (no new decay since base).
  • Warm timings of 1000 µs or more are printed as FAILED-GATE. Storage, attest and prepare times are labelled COLD/IO. Every Lance byte count came from the page cache.
  • Not run: the full workspace test suite. Only the lab crate and the docs change on main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF

claude added 16 commits November 3, 2026 21:23
…position (lab)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Routing arms A/B/C, working-set extension, presence-only updates, EXO-46
shared selection, multi-domain composition laws. F12 made non-vacuous (cross
edges only); F26 fixture fits 255+1 ids per tile. Disable runs: dupcheck->F4,
framecheck->F6/F7/F9, bindguard->F9 turn red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…aft, lab)

where_k compiles; not yet run at 1M/10M. bench_where/compare_where smoke-tested
at N=20000 against an independent 3VL oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ompare_big

Same rule as the P0 fix (#1946): coverage is enforced in both directions,
DuckDB results must cover exactly threads {1,4}, and any missing comparison
exits 1 instead of becoming a note under an all-agree verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…anes, lab)

Draft, NOT yet compiled or run: U/P/S layouts, packed 4x8 and sparse u16
encodings, refusals R1-R3, scalar oracle. Compile and measurement follow
after the 10M where_k run finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Composes D working sets with the existing DR algebra (no paging engine, no
corpus-wide adjacency). Stages charged separately. At N=1M: every composed
answer equals the oracle; refusals TooMany, id-in-two-sets, checked_sum
overflow and stale binding all fire with paired controls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…10M (lab)

All answers equal the independent oracles at both sizes; all refusals fire.
Warm values >= 1000 us are reported as FAILED-GATE, cold stages as COLD.
DuckDB comparison follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
… crashing

DuckDB answers some predicates (a tautology over min/max stats, e.g.
c0<1000 OR c3<5) from table statistics: the plan is a constant Column Data
Scan, no operator runs and no profile is written. Engine time is recorded as
None with stats_folded=true; answers are still checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Corrected parity gate passes: every DuckDB count/EXISTS equals the oracle,
coverage complete in both directions at threads 1 and 4. Analysis is parked
pending the Lance 13 selective-read trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…memory corrections probe (lab)

ENGINEERING-PARITY.md: four DuckDB publications (research prototype vs shipped
marked) mapped onto DR/DT64/ClassView/Quack/mask-risc/ndarray/Lance 13, plus
the WHEN-vs-WHETHER recall comparison. EXISTS no-early-exit verified in code
(mask-risc exec.rs:1633,1793-1794); production change not made.
corrections.rs: route-once vs per-query, ndarray words vs hand-written packed
compares (ordered u8-on-u32 = NO-EXISTING-KERNEL), borrowed vs gather_lane.
Compiles; not yet run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…piled)

Adds crates/crepe-plan-probe/examples/where_lance_arm.rs, which compares
three reads on a permuted 16-property fixture: a full scan, a masked
prefilter, and the #1633 attested range read. It also resolves the
projection through ClassView, with refusal tests.

The example now uses the addressed-read path instead of private copies.
To allow that, addressed.rs makes `ranges` and `range_mask` public and adds
`range_tree` (used to measure ingress bytes) and `check_fresh`, which
`current_lane` now calls. Attest checks that `_rowid` equals `_rowaddr`.
The D4 boundary prints a fixed BLOCKED_D4 message; no DR ordinal is
composed with `_rowid`.

This commit has not been compiled. A build is running; fixes follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
where_lance_arm now builds and runs at 1M. Every oracle check and every
refusal check passes. Two fixes:
- Restore the RowAddrTreeMap import, which the B arms need.
- Write the fixture with max_rows_per_file = u32::MAX. Lance 13 compares
  that limit as `as u32` (write.rs:1077), so 1 << 40 truncated to 0. The
  10M table then had one fragment per batch, and attest refused it as
  NotContiguous. That log is kept.

The in-memory corrections (route once, ndarray kernels, borrowed reads)
match the oracle at 1M and 10M. Results and logs are committed.

Adds REPORT.md (three parity verdicts, the cost columns, the verdict
list) and LANCE-TRACE.md (the source-backed trace). The 10M Lance rows
are pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…CKED_D4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ed demand

At 10M, every oracle check and every refusal check passes. The 1M
pattern holds and grows:
- a clustered 0.1% read is 114x faster than a full scan;
- a scattered 50% read at k=16 is 11x slower;
- ingress for 3.26M ranges is a 4.8 MB tree.

New finding: ReadPolicy::DEFAULT decides on covered rows alone
(addressed.rs plan_read). It therefore sends a single contiguous 10%
range to a full scan: 3.8x slower at 1M, 4.9x at 10M. This is recorded as
a probe item for the #1633 owner; the policy is not changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e95e7b43-d06e-486c-89e0-f6b0fec6a8f9
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review November 3, 2026 22:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T22:30:28.948277Z 9bb2c91 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The crepe-plan-probe CI job runs `cargo fmt --check` and
`cargo clippy --all-targets -- -D warnings`. This commit applies rustfmt
to both files and removes the two warnings: the unused Env::n field, and
an initial value of `read` that was overwritten before being read. No
behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bb2c91b1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

entries_index --check failed with DUPLICATE ID: D-GLOBAL-WHERE. The
index takes the first D-id token in an entry's heading, so
"D-GLOBAL-WHERE P2" and P0's "D-GLOBAL-WHERE P0" read as the same ID.
The P2 heading is now D-GLOBAL-WHERE-P2 and its index row has been
regenerated. Both checks pass: the entries index and the supersession
index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The C arms timed two tree builds in the ingress stage: range_tree() to
measure the serialized size, then range_mask(), which builds the tree
again. That inflated the reported S-INGRESS cost. Both the B and C arms
now build exactly one tree inside the timer. They measure its serialized
size outside the timer (that is reporting, not ingress) and hand the
same tree to RowAddrMask::from_allowed / from_block.

The committed Lance-arm results predate this fix. They are being
regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The 1M and 10M Lance arms were re-run on 90eb4cc. Every oracle check
and every refusal check passes at both sizes.

C-arm ingress roughly halves (1.8-2.9x at 1M); for 3.26M ranges it is
now 123,162 us, down from 267,905 us. B-arm ingress is unchanged within
noise, and mask byte counts are identical.

REPORT.md tables, prose ratios and the cost table are regenerated from
the logs. The board entry is updated to match. Figures previously
written in ms are now in us.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
…ost)

The measurement data headed for main is reduced without dropping any
number that is cited or used:
- where_duckdb.json (1M, 10M): one record per line instead of one field
  per line, 2,072 -> 116 lines each. The JSON round-trip is asserted
  equal, and compare_where.py regenerates both compare_where.md files
  byte for byte from the compacted files.
- Deleted the raw console logs:
  - wk1m/wk10m/bw: their per-case data is in where_ours.json and
    where_duckdb.json.
  - cw: it is exactly the two compare_where.md files concatenated.
  - proj1m/proj10m: their data is in project.json.
- results/run_summary.log keeps the one thing only those logs held: the
  FAILED-GATE totals and the oracle verdict lines (353/648 and 23/41).
- Deleted the 10M EXPLAIN plans, which repeat the 1M ones. None was
  cited.

Citations of cw.log now point at the same lines of compare_where.md.
The remaining ms figures in ENGINEERING-PARITY.md and LANCE-TRACE.md
are converted to us.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The merge auto-combined README.md, but the index was stale against the
entries that main gained (D-DRAM-0, D-GRAMMAR-WHEN-1). Regenerated with
entries_index.py --write: 569 rows, structurally sound. The supersession
index is current.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
The in-memory probe (probe/), the DuckDB scripts (duckdb/) and all
measurements (results/) now live only on research/d-global-where-p2
(commit 1422b2b); main does not get them. main keeps REPORT.md,
ENGINEERING-PARITY.md, LANCE-TRACE.md, SPEC.md, the board entry and the
crepe-plan-probe changes. Each doc says that its probe/, duckdb/ and
results/ paths refer to the research branch, and links it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
@AdaWorldAPI
AdaWorldAPI merged commit 58e01cf into main Nov 3, 2026
12 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Nov 3, 2026
…G-MODEL

Merges the #1950 P2 findings, the #1949 integration, the P3-1 and P3-3
results and the ndarray Abakus probe results (#389, #391, #402, microtile)
into one findings / high-signal / fruits / open-ends entry. The operator's
Abakus direction (repeated bit-plane folding as an ISA register contract)
is recorded as WORKING-MODEL; P3-5 (k-of-n on the two shipped carry words)
is its falsifier and is added to IDEAS as a high-signal fruit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAqKMNZqKXSaeb9g2ypywF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants