Repository navigation
[weftdb] Stop pipeline dictionaries doubling on every run; release 0.3.2 - #612
Open
physics515 wants to merge 3 commits into
Open
physics515 wants to merge 3 commits into
physics515 wants to merge 3 commits into
Conversation
physics515
added a commit
that referenced
this pull request
Nov 7, 2026
The recovered commit was written by hand and never run through `cargo +nightly fmt`, so #612's `rustfmt` job went red on its first roll — the only red on the roll, and the one gate a `[skip ci]` commit cannot pass by existing. rustfmt.toml sets `max_width = 10000` with hard tabs, so what the author wrapped across five lines the style gate wants on one: the three `conn.as_ref().execute(...)` chains in the new `replace_dictionary_patterns` write-back, and the `range.map(...).collect()` in the regression test's fixture. `cargo +nightly fmt -p weftdb -p weft-orchestration -- --check` is clean now; no code changed beyond line breaks. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
physics515
marked this pull request as ready for review
November 7, 2026 07:58
physics515
added a commit
that referenced
this pull request
Nov 7, 2026
The recovered commit was written by hand and never run through `cargo +nightly fmt`, so #612's `rustfmt` job went red on its first roll — the only red on the roll, and the one gate a `[skip ci]` commit cannot pass by existing. rustfmt.toml sets `max_width = 10000` with hard tabs, so what the author wrapped across five lines the style gate wants on one: the three `conn.as_ref().execute(...)` chains in the new `replace_dictionary_patterns` write-back, and the `range.map(...).collect()` in the regression test's fixture. `cargo +nightly fmt -p weftdb -p weft-orchestration -- --check` is clean now; no code changed beyond line breaks. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
physics515
force-pushed
the
w8/dict-bloat
branch
from
November 7, 2026 08:01
acf0d01 to
255319e
Compare
physics515
added a commit
that referenced
this pull request
Nov 7, 2026
The recovered commit was written by hand and never run through `cargo +nightly fmt`, so #612's `rustfmt` job went red on its first roll — the only red on the roll, and the one gate a `[skip ci]` commit cannot pass by existing. rustfmt.toml sets `max_width = 10000` with hard tabs, so what the author wrapped across five lines the style gate wants on one: the three `conn.as_ref().execute(...)` chains in the new `replace_dictionary_patterns` write-back, and the `range.map(...).collect()` in the regression test's fixture. `cargo +nightly fmt -p weftdb -p weft-orchestration -- --check` is clean now; no code changed beyond line breaks. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
physics515
force-pushed
the
w8/dict-bloat
branch
from
November 7, 2026 13:07
255319e to
265ccd5
Compare
physics515
added a commit
that referenced
this pull request
Nov 7, 2026
…t, and gates behind a red gate are untested (#614) publish.yml has failed twice in the same step with "please provide a non-empty token" (publish.yml:123 reads secrets.CARGO_REGISTRY_TOKEN), so 0.3.1 is not on the registry even though its GitHub release and five archives are drafted. That is a human action, and the follow-on choice — publish v0.3.1 first or let 0.3.2 be the first release out — has to be made deliberately, because #612 already bumped four crates to 0.3.2 and silence is now the gap. Recorded beside it: host-install gates 6 and 7 had never run anywhere until this week, because every roll stopped at gate 4 and then gate 5, and gate 6 failed on its first-ever execution for a reason that had nothing to do with the package. A gate you have never reached is not a gate that passed. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
…e 0.3.2 load_dictionary started on top of the in-memory patterns and extraction appended the whole dictionary to the store again (patterns.id is not unique), so each Pipeline::run roughly doubled preamble.db and RAM. Fan Control hit 3.5 GB / 112 GB RSS in minutes. - load_dictionary starts from the store - new atomic Database::replace_dictionary_patterns for write-back - import_pattern skips a pattern id it already holds - dictionary-list cache key includes the aspect - regression test: repeated runs store each pattern once - ROADMAP: lossless occurrence footprint + learned constraints Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
The recovered commit was written by hand and never run through `cargo +nightly fmt`, so #612's `rustfmt` job went red on its first roll — the only red on the roll, and the one gate a `[skip ci]` commit cannot pass by existing. rustfmt.toml sets `max_width = 10000` with hard tabs, so what the author wrapped across five lines the style gate wants on one: the three `conn.as_ref().execute(...)` chains in the new `replace_dictionary_patterns` write-back, and the `range.map(...).collect()` in the regression test's fixture. `cargo +nightly fmt -p weftdb -p weft-orchestration -- --check` is clean now; no code changed beyond line breaks. Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>
physics515
force-pushed
the
w8/dict-bloat
branch
from
November 7, 2026 15:30
4a57bfe to
357285b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A
Pipeline::runon a store that already holds a dictionary wrote the whole dictionary again.load_dictionaryseeded the in-memory dictionary on top of the patterns the store alreadyheld, and extraction then appended every dictionary entry back to the store — and
patterns.idis not unique, so nothing refused the duplicates. Each run roughly doubled
preamble.dband theprocess's RSS: the Fan Control ingest reached 3.5 GB of a 112 GB store and 112 GB RSS within
minutes of starting.
load_dictionarynow starts from the store, not from a re-import of what it loaded.Database::replace_dictionary_patternswrites the dictionary back atomically, so ahalf-written write-back cannot leave the store holding half a dictionary.
import_patternskips a pattern id it already holds instead of adding a second row.cached list.
weft-orchestration/tests/dictionary_bounded.rsruns the pipeline twice over the same storeand asserts each pattern is stored once — the regression the field failure produced.
weftdb,weft-server,weft-arrow-store,weft-orchestration) anda CHANGELOG section, so this is also the release-prep commit.
Provenance
This is the one commit that existed only locally:
4ee2629onweftdb/dictionary-occurrence-log,written 2026-11-06 and never pushed. Rebased onto
origin/main8c5fea9 (clean, no conflicts),[skip ci]dropped from the subject andSigned-off-byadded, so the CI gate actually runs —the original message would have suppressed every workflow on this PR.
Verification status — this PR is a draft on purpose
No local build: the machine is running two concurrent lanes (
w8/fre-9a,w8/soa-15) and its RAMdoes not fit a third cargo invocation. CI on this PR is the verification, and it is the first
execution of
dictionary_bounded.rsanywhere. Draft until that roll is green.The release half of the commit (four version bumps, 0.3.2) additionally needs the release owner
to look at it: 0.3.1 was tagged 2026-11-06 and
release.ymlaccepts only a push/dispatched/scheduledrun of CI on
mainas evidence, so the tag step is a human action regardless of this PR.Refs: the dictionary footprint work on the ROADMAP ("lossless occurrence footprint + learned
constraints" lines this commit also ticks in).