Repository navigation
I-17: book decomposition in the ingestion pipeline, CI consolidation, and fixes - #119
Merged
Merged
Conversation
…oop close Passing story_id as a keyword to anyio's run_sync raised TypeError, which the handler swallowed, so memory hygiene never ran. Shutdown also crashed when cancelling an ingest future whose event loop had already closed. Adds regression tests for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renames variables that reused a name with another type, adds type parameters, and removes stale ignores. No behavior change, apart from returning the canon entity ID as a string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Uses the quay.io/minio/minio image in place of the Docker Hub one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o local CI now runs lint, layer checks and strict mypy as one required job, plus one hermetic test job with pytest-xdist. Contract, property and behavior tests are already in that run, so their workflows go. Mutation testing was too costly for CI and stays available locally. Integration tests move from nightly to weekly. Docs and badges follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds book, division, paragraph and sentence nodes, checks that no text is lost, and derives IDs from content so reprocessing keeps them. Fixes repeated text in parent sections and rejoins sentences broken by a font change. Prototype: not used by IngestionPipeline yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every sentence ends cited, skipped or reported. An append-only log lets runs resume after a crash and send only missing sentences, and a guard handles quota pauses, backoff and budget caps. Prototype: not used by IngestionPipeline yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verifies claims against their sources, sets examples aside using layout, refuses merges that combine separate rules, links claims from one paragraph, and checks paragraphs for what their claims leave out. Every claim gets a recorded outcome. Adds a local sheet for hand-labeling. Prototype: results don't reach canon. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…prototype script Every relationship must cite claims that name both of its ends. Kimi runs through its command line with tools off, in an empty folder, within a call limit. In a 16-question test, plain keyword search beat the graph 15 to 12, so the graph isn't yet proven to help retrieval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records the design, measurements on Dune, what was tried and removed, and the open decisions. States that this is a prototype not connected to the pipeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decides what to keep, replace or reuse from a first read of the existing stages. Deduplication and contradiction detection work across packs and canon, so they're reused rather than duplicated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Orders the work to move the prototype into the ingestion pipeline, one file per step, with checks, done criteria, and the decisions that need a person. No step calls a live model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Dune, the current reader's sections hold 5.01 times the book's text, because parent sections repeat their children. The repo's only PDF fixture can't show this, so later steps need generated PDFs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each bookmark with its own text becomes one section. Text before the first bookmark gets a section of its own, and a book with no bookmarks splits at its headings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old reader copied page ranges into each bookmark, so parents repeated children and text before the first bookmark was lost. On Dune, section text falls from 5.01x the book to 0.995x (the rest is running headers), chunks from 3,730 to 889, and sections from 732 to 700. All 19,630 lines are accounted for, including 102 the old reader dropped. The cache wrapper and public function are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a nested fixture, the old reader lost the foreword and repeated the nested text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scrolls stopped at 512 chunks in hashed point ID order, so the Analyzer read an arbitrary part of a book. It now reads all matches, then takes evenly spaced chunks in reading order, and logs when it truncates. The number of chunks reaching the model is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…comment The audit said PDF structure wasn't cached, but it is. The timeout comment said 30 minutes and tighter than the job timeout, while the value is 180 and the 45-minute job timeout fires first on the UI route. Only the comment changes; the value is a separate decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fact Nothing but tests imported loops/ingestion_loop.py, and triage, mechanics_architect and vision were used only by it. Also removes mypy_output.txt, a committed tool output. Full suite passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each record's _id comes from its document, type and key, so repeated writes are harmless. A lease that expires stops two workers running one document, without a crash blocking it forever. Not exported from mongodb_tools, which keeps bulk book data off the MCP surface. Adds mongomock for tests; nothing here has run against a real MongoDB yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gress The extraction loop and review take any backend through small protocols. Interrupted runs resume without resending finished sentences, as with the file logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… jobs Normal ingest jobs don't list the new stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rules go in game_system_data.rules, and every item cites its source sentences. Pack fields silently trim long text, so a claim over the limit is held back for a person and never cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Off unless MONITOR_BOOK_DECOMPOSITION is set. It gets its own job record, can pause and resume without repeating work, respects cancellation, and creates one review-ready pack per document. Tested end to end with scripted models on mongomock; not yet run with live models or a real MongoDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uter
Runs on its own worker, outside the ingest slot and the 45-minute timeout, so a long book doesn't block other uploads. Adds POST /ingest/jobs/{id}/resume, and paused jobs can be cancelled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decomposition jobs show sentence and claim counts, a pause reason, and Resume in place of Retry, which would re-run the whole ingest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruled tables become one table with rows intact, such as the complication table on page 163. On Dune, fragment paragraphs fall from 1,055 to 766, tables rise from 64 to 161, and empty divisions fall from 33 to 27, with no text lost. The rest have bookmark titles missing from the page text. Table detection adds about 30 seconds on a 339-page book. Books with no bookmarks already split at font-detected headings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b false alarms Examples can resume after a boxed sidebar. Kept claims naming a character used mainly in examples go to a person, and claims that describe the book are rejected, including gaps. Capitalized verbs such as Keeps no longer count as unknown names. Thresholds were set on Dune only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raphs Each claim gets a domain, and each document gets a rules tree and a lore tree following the book's headings. Graphs run through the review model behind MONITOR_BOOK_GRAPH, resume from saved batches, and pause on quota. Graph search is removed, because plain keywords beat it 15 to 12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deleting a source now removes its stored records and lease. Also updates the stage count pin for the new decompose stage, and pins that normal jobs leave it out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spuentesp
changed the title
Off by default behind MONITOR_BOOK_NEO4J. Only CanonKeeper may call the new tools, so the existing rule holds and its guards pass, with BookNode records staying off canon labels. Real Neo4j showed MERGE needed a uniqueness constraint at full-book size, so the loader creates one first. Adds optional tests against real MongoDB and Neo4j. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the UI route, the 45-minute job limit always fired before the 180-minute stage limit, so a stuck provider never got its clear error. The route now passes the job limit minus 5 minutes. Analyses taking 40 to 45 minutes used to pass if everything else was quick, and now fail. Callers with no job limit keep 180 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neo4j writes go through CanonKeeper, which replaces the earlier recommendation to widen the rule. Nothing has run with live models, because LLM work is paused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y jobs Nothing ever set stage_last_progress_at, so the stale-job watchdog failed every running job 45 minutes after it started, however well it was progressing. Every job update now sets it. Tested with the real watchdog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…retry Decomposition jobs send a heartbeat through quiet stretches like review, which stops if the process dies, so the watchdog can still catch a dead worker. Retrying a failed index now clears the source's old chunks, so they can't mix with chunks from the new reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settles the plan's open checks. Documents indexed before the reader change keep their old chunks until deleted and uploaded again. Live runs stay outside this plan while LLM work is paused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first live run showed models write types like Asset, device and Ranged Weapon, while the mapper accepted four exact values, so all 153 entities became plain facts. Types now map to the schema's six values, and the original wording is kept as sub_type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One chapter ran on Mistral and real MongoDB and recovered from a real interruption. Claim quality still needs a person's labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…boxes as tables Labels on the first live run exposed repeated phrases: this PDF draws some text twice at one spot as a shadow. The reader now keeps it once and counts what it removes, so the no-text-lost check stays honest. The same check found ruled boxes around prose becoming tables, which skipped sentence splitting, so a table now needs multi-cell rows. On Dune: fragments 896, tables 96, no repeated runs, nothing lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27 of 30 sampled kept claims were labeled canon as written. Small sample, one chapter, kept claims only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner decided book text shouldn't stay in the public repo. Tests and help text now use invented examples, and the two fixtures that held book text are gone from this branch's history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spuentesp
force-pushed
the
feat/book-decomposition-i17
branch
from
November 5, 2026 00:19
99fb8d1 to
6fd09dc
Compare
Chapter 5 ran on the fixed reader. The earlier sample covered pages 198-215, not the whole chapter, and showed 5 of its 30 claims in pre-review wording, so its result is provisional until they are labeled again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
53 claims from Chapter 5, 10 labeled by a person and 43 by Claude, so this is a rough check. It found a correct re-roll rule rejected during contradiction settling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The book states both versions of the rule, so this is a conflict settled without a person, not a plain rejection of the book's wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Chapter 5, review kept 'entire dice pool' and rejected 'any number of dice' as refuted, though the book states both. A claim that closely matches its own source can no longer be rejected as a misreading, and the prompt now says so. Replaying the saved runs, the one settled conflict goes to a person. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A person's reason for marking Specialist not canon stands, because the final wording keeps the short name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers becoming headings On Dune, 17 large first letters became headings and left paragraphs starting 'he best way', and 245 stat-block numbers became headings that named sections. Letters now rejoin their paragraph, and a heading needs a letter. Headings 1,176 to 914, nothing lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claims review sends to a person were only written to the internal ledger, so nothing in the app showed them. The job now counts them and the jobs list says how many claims need review. A review screen is still to come. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claims review sent to a person had no way to reach the pack. Jobs now list them with their reason, source sentences and the other side of any contradiction. Accepting adds the claim to the book's pack; decisions are stored apart from the ledger, so a resumed run or a rebuilt pack keeps them. Applied packs and books mid-run refuse decisions. The ledger now keeps the reviewed wording, entity and relationship fields, and conflicting claims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A claim a person accepts now joins its domain tree as soon as it's decided, and every later run keeps it there. A claim whose sentences aren't in the stored book is refused rather than added without a citation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lorebook fixture accepts include_inactive kwarg - chat_loops end-scene tests use a mock SceneLoop instead of None - isolate test_import_errors in a subprocess so its reload doesn't poison other tests' module references
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes for every upload
stage_last_progress_at, so it failed every job 45 minutes after it started. Every job update now sets it. The 45-minute job timeout on the UI route still applies to normal ingests.What's behind a flag
Book decomposition (I-17) is off unless
MONITOR_BOOK_DECOMPOSITION=1. It runs as a separate background job after a PDF ingest, with pause, resume and cancel, and creates one review-ready pack with source sentences as evidence.MONITOR_BOOK_COVERAGE,MONITOR_BOOK_GRAPHandMONITOR_BOOK_NEO4Jadd optional steps. Neo4j writes go through CanonKeeper, so the existing rule holds.CI changes to review
Not proven
Full Dune did run through real MongoDB 7 and Neo4j 5 with scripted models: 10,173 sentences, one pack, and a rerun that made no extraction or verification calls.
Testing
8,721 passed and 231 skipped on the full suite, with ruff, strict mypy, layer checks and the sealed-suite guard clean, and 466 frontend tests passing. No sealed suite files changed.
Plan and results:
docs/architecture/BOOK_DECOMPOSITION_IMPLEMENTATION_PLAN.mdBook text removed: tests and help text now use invented examples, and the two fixtures that held Dune text are gone. This rewrote the branch's history, so older commit links from this PR point at commits no longer on the branch.
🤖 Generated with Claude Code