feat(cli): configurable insert_mode (fail-fast / fail-at-end) for strict compile-failure handling - #241
Open
sebastianbraun25 wants to merge 4 commits into
Open
Conversation
…ict compile-failure handling
- Add `insert_mode` config key (`.openkb/config.yaml`) with three values:
- "normal" (default): unchanged behavior — a concept/entity generation
failure during compile is logged as a warning and the file is still
reported "added".
- "fail-fast": the first concept/entity generation failure cancels every
other still-pending generation in the batch and immediately raises
`ConceptCompilationError` — nothing from the batch is written.
- "fail-at-end": every planned concept/entity generation is attempted (so
every failure for the document is logged in one pass) before
`ConceptCompilationError` is raised if anything failed.
- Both strict modes rely entirely on the existing mutation-snapshot rollback
(`openkb.add_coordinator`/`openkb.mutation`) to discard the add and report
it "failed" — no new rollback path needed. The existing "keep raw/ on
failed" and "keep the debug log on a non-'added' outcome" behaviors already
cover the raw-file and log-preservation requirements for strict mode.
- `_compile_concepts`'s three early-return paths (unparseable plan, scalar
plan, all-items-filtered-as-malformed) now also raise under a strict
insert_mode, not just individual concept/entity generation failures — a
genuinely empty plan (nothing was ever planned) still counts as complete
success in every mode.
- `compile_short_doc`/`compile_long_doc` resolve `insert_mode` from the
already-loaded KB config, so no CLI-level plumbing is needed.
Resolves VectifyAI#239
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sebastianbraun25
pushed a commit
to sebastianbraun25/OpenKB
that referenced
this pull request
Sep 4, 2026
_llm_call/_llm_call_async now wrap the stream fetch/consume/merge step in a fixed 3-attempt loop. A dropped connection or other transient error during the now-streamed completion (VectifyAI#236) previously failed the whole concept/entity generation immediately; it now gets 2 automatic retries before giving up. Not a tunable knob, just a resilience floor sitting below the end-of-batch sweep retry added for insert_mode (VectifyAI#241).
added 3 commits
September 4, 2026 11:14
…ot the whole document _compile_concepts now runs a deferred, end-of-first-pass sweep for concepts/ entities that failed their first attempt, under "normal" and "fail-at-end" insert_mode (not "fail-fast", which already aborts before the rest of the batch runs). This gives transient failures a real second chance, with the prompt cache still warm, before insert_mode's strict modes decide the document is incomplete. _run_compile_with_retry (cli.py) no longer retries the whole document when ConceptCompilationError is raised: insert_mode already exhausted the per-call and per-item retries, so a full recompile would just repeat the same failures. Other exceptions keep their existing 2-attempt retry.
_flaky_once_acompletion returned a bare MagicMock instead of a [mock_resp] list; MagicMock auto-supports __aiter__ (empty by default), so _llm_call_async took the async-stream branch and saw 0 chunks instead of treating it as an already-collected 1-chunk list like _selective_acompletion does. Also made the injected failure survive all 3 of _llm_call_async's own retry attempts in the first pass, so it only clears up at the sweep tier -- previously it cleared up on the low-level retry's 2nd attempt, which incidentally also hit the same 0-chunks bug and masked the intended sweep-tier test.
…fails Adds the missing case symmetric to test_normal_mode_sweep_recovers_ transient_failure: a transient failure the sweep clears up must not raise ConceptCompilationError under insert_mode="fail-at-end" -- the completeness check runs after the sweep, not before it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.
Problem
openkb addcurrently reports a whole-file"added"outcome even when one or more individualconcepts/entities failed to generate during compilation.
_compile_conceptsinopenkb/agent/compiler.pycollects exceptions from the per-concept/per-entity generation tasksvia
asyncio.gather(..., return_exceptions=True), logs a[WARN] ... planned but only N writtenline, and continues — but the file's hash is still registered and the source is still eligible for
auto_delete_added_files, so a partially-compiled document looks identical to a fully successfulone from the CLI/API caller's point of view.
Solution / Changes
insert_modeconfig key (.openkb/config.yaml) with three values:"normal"(default): unchanged behavior — a concept/entity generation failure during compileis logged as a warning and the file is still reported
"added"."fail-fast": the first concept/entity generation failure cancels every other still-pendinggeneration in the batch and immediately raises
ConceptCompilationError— nothing from thebatch is written.
"fail-at-end": every planned concept/entity generation is attempted (so every failure for thedocument is logged in one pass) before
ConceptCompilationErroris raised if anything failed.(
openkb.add_coordinator/openkb.mutation) to discard the add and report it"failed"— no newrollback path needed. The existing "keep
raw/on failed" and "keep the debug log on anon-
'added'outcome" behaviors already cover raw-file and log-preservation for strict mode._compile_concepts's three early-return paths (unparseable plan, scalar plan,all-items-filtered-as-malformed) now also raise under a strict
insert_mode, not just individualconcept/entity generation failures — a genuinely empty plan (nothing was ever planned) still
counts as complete success in every mode.
compile_short_doc/compile_long_docresolveinsert_modefrom the already-loaded KB config, sono CLI-level plumbing is needed.
"normal"behavior is unchanged.Issues