Add a Rebuild Index button to the Album Manager - #402
Merged
Merged
Conversation
A red "Rebuild Index" under "Update Index", confirmed before it does anything, shown only for albums that already have an index and hidden while one is being built. It exists because Update Index cannot do this job. An update is a set difference on paths — it adds files that appeared and drops files that vanished — and never re-reads a file already in the index, whatever its mtime. So anything derived at index time is fixed at that moment, and the only way to revisit it was to delete the .npz by hand. The flow reuses the path that already recovers from a corrupted index: remove the index, then index the album. Its flag is renamed from `isCorrupted` to `removeExistingIndex`, since a deliberate rebuild is now the second caller, and the messages along it no longer say "corrupted". Two bugs found while verifying this in a browser rather than only in jsdom: - The shared confirmation dialog was z-index 9999 against the Album Management overlay's 99999, so asked from inside the album manager it rendered *behind* it: the prompt painted, its buttons unclickable, the flow dead. Raised above the overlay and the file tree, still below the spinner and toasts. A test now pins the ordering, since jsdom does no layout and could not have caught it. - rebuildIndex captured the card element before awaiting the confirmation, which is an await of unbounded length, and loadAlbums() rebuilds the card list wholesale — so progress could be painted onto a detached node while the visible card sat frozen. It re-resolves through _liveCardFor, the helper that already exists for exactly this. Also: remove_index now clears the npz lru_cache. Every other clear sits in a write path, which was enough while the endpoint only ran as a prelude to re-indexing — the rebuild's own write cleared it. A button makes that rebuild skippable, so the app could go on serving an index that is no longer on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four buttons did not form a grid. .index-controls centred its row against the status text, which wraps to two or three lines, so Update Index floated down while Edit sat at the top of the next cell — and Rebuild Index, in a row of its own below, landed 9px off Delete. The near-alignment of the first pair was coincidence, not layout. The index buttons are now a column that mirrors .action-buttons.vertical: same direction, same 0.5em gap, right-aligned, with the row switched to align-items: flex-start so it starts where the grid's align-items: start puts the Edit/Delete column. Only one of Update/Cancel is ever visible, so the column is two buttons deep in practice, exactly matching its neighbour. Measured in the browser across every card: Update Index is now 0px from Edit and Rebuild Index 0px from Delete, and the two index buttons share a width (stretch rather than flex-end) the way Edit and Delete already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lstein
enabled auto-merge (squash)
September 20, 2026 18:25
lstein
added a commit
that referenced
this pull request
Sep 20, 2026
The note shipped in #400 said an update "only re-reads files whose modification time has changed", which implies Update Index might pick up newly-readable metadata. It never will: _get_new_and_missing_images is a set difference on paths, and modification_times is stored and sorted but never consulted to decide re-processing, so a file already in the index is not re-read whatever its mtime. Confirmed on a real album: 363 indexed videos, 4 carrying a record, 89 more recovered only after rebuilding. Points at the Rebuild Index button from #402 rather than telling people to delete the index file by hand, and links to that button's own section, which covers what a rebuild costs and what it leaves alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lstein
added a commit
that referenced
this pull request
Sep 20, 2026
* feat(video): read the JSON sidecar for videos generated before InvokeAI 7
Pre-7 releases kept a generated video's record in a sidecar under
{outputs}/videos/sidecars/, mirroring the video's own subfolder, rather than
inside the MP4. InvokeAI 7 still writes one when the embedding remux fails,
so this is a live fallback and not only a legacy path.
The extractor now reads the MP4's keyed metadata first and the sidecar
second, which is InvokeAI's own order and means a v7 video costs no sidecar
lookup at all.
Finding the sidecar is the interesting part: PhotoMapAI indexes absolute
paths and never learns where an `outputs` directory begins, so each ancestor
of the video is tried as the videos root, nearest first, mirroring the
video's relative path under `sidecars/`. Bounded, and required to carry an
`invokeai_metadata` key, so an unrelated directory called `sidecars` cannot
be mistaken for InvokeAI's.
Measured against a real install (668 videos, 662 sidecars): every sidecar
resolved at ancestor depth 1, 132 carried a record and 530 carried a null
one (a workflow but no parameters), 3 videos had the record embedded, and
the whole scan cost 1 ms per video including the MP4 walk. All 132 recovered
records parse, are recognised as video generations, and render a panel with
no undeclared fields — which also validates the v5 video profile against
real data rather than only against the synthetic fixture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(video): close the gaps an adversarial review found in the sidecar reader
Six defects, each reproduced first, and the fixes mutation-tested after.
- RecursionError escaped the reader. json.loads raises it on deeply nested
JSON; it is a RuntimeError, so neither `except OSError` nor
`except ValueError` caught it, and ~120 KB of brackets is three orders of
magnitude under the size cap. The damage was not a lost record but a lost
*video*: _load_video catches it, returns None, and the file is recorded as
bad and left out of the album — exactly what the docstring promises cannot
happen. Both json.loads sites were exposed; they now share a helper.
- The `invokeai_metadata` key gate was weak and untested: it handed back
whatever object another tool stored under that name, and removing the gate
entirely left all 25 tests passing. The record must now also satisfy
looks_like_invoke_metadata — the same test the drawer routes on, and one
all 132 records in the reference install pass.
- MAX_SUBFOLDER_DEPTH cut from 3 to 1. The bound had no real test (the old
one imported the constant it was checking, so it passed at 1 and at 9) and
no evidence: 661 of 661 real sidecars resolve at depth 1. The extra levels
only bought reach *outside* the configured album — at depth 3, a path at
the filesystem root. That also removes the `..` escape the review found,
which needed depth 2.
- The reader resolves the video path first, so a symlinked video finds the
sidecar beside its target. sidecar_candidates stays pure and now documents
that it expects a resolved path.
- Size cap 32 MiB -> 8 MiB, matching mp4_metadata.MAX_TAG_BYTES, so the same
record is not rejected from one source and accepted from the other.
- UnicodeDecodeError was caught (it is a ValueError) but logged as a JSON
error; split so the message names the right stage.
Also pinned the empty-embedded-record fall-through, which the `or` cannot
distinguish from "no tag", in both a test and the docstring.
Re-measured against the reference install after the fixes: the same 135
records recovered from 668 videos, at 0.2 ms per video.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: an existing album needs a full re-index, not an Update Index
The note shipped in #400 said an update "only re-reads files whose
modification time has changed", which implies Update Index might pick up
newly-readable metadata. It never will: _get_new_and_missing_images is a
set difference on paths, and modification_times is stored and sorted but
never consulted to decide re-processing, so a file already in the index is
not re-read whatever its mtime.
Confirmed on a real album: 363 indexed videos, 4 carrying a record, 89 more
recovered only after rebuilding.
Points at the Rebuild Index button from #402 rather than telling people to
delete the index file by hand, and links to that button's own section,
which covers what a rebuild costs and what it leaves alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A red Rebuild Index under Update Index, confirmed before it does anything, shown only for albums that already have an index and hidden while one is being built.
It exists because Update Index cannot do this job. An update is a set difference on paths — it adds files that appeared and drops files that vanished — and never re-reads a file already in the index, whatever its mtime. So anything derived at index time is fixed at that moment, and until now the only way to revisit it was to delete the
.npzby hand. (That is what made #401's new sidecar metadata invisible on an existing album.)The flow reuses the path that already recovers from a corrupted index: remove the index, then index the album. Its flag is renamed
isCorrupted→removeExistingIndex, since a deliberate rebuild is now the second caller, and the messages along it no longer say "corrupted".Two bugs found by driving a real browser, not jsdom
The confirmation dialog rendered behind the Album Manager.
#confirmModalwasz-index: 9999against the overlay's99999, so the prompt painted butelementFromPointover its Cancel button returned the album card underneath — unclickable, flow dead. It is now above the overlay and the file tree, still below the spinner and toasts. A test pins the ordering by parsing the CSS, because jsdom does no layout and could never have caught it.#deleteConfirmModalis the same shape but is only ever opened from the main UI, so it is left alone and noted in place.rebuildIndexcaptured the card element before awaiting the confirmation — an await of unbounded length — whileloadAlbums()rebuilds the card list wholesale. Progress could be painted onto a detached node with the visible card frozen. It now re-resolves through_liveCardFor, the helper that exists for exactly this.Neither is visible in a screenshot; the first only surfaced because Playwright could not click Cancel, which was easy to dismiss as a test artefact.
Backend
remove_indexnow clears the npzlru_cache. Every other clear sits in a write path, which was sufficient while the endpoint only ran as a prelude to re-indexing — the rebuild's own write cleared it moments later. A button makes that rebuild skippable, cancellable and failable, so the app could go on serving an index that is no longer on disk. A test fails without the fix.Only
embeddings.npzis removed. The semantic map, cluster labels and thumbnails are derived and self-invalidate by mtime —umap.npzagainst the index, the labels againstumap.npz— so they regenerate, and deleting them would only throw away reusable work. Pinned by a test.Layout
The four buttons did not form a grid:
.index-controlscentred its row against a status message that wraps to three lines, so Update Index floated down while Edit sat at the top of the next cell, and Rebuild landed 9px off Delete. The near-alignment of the first pair was coincidence. The index buttons are now a column mirroring.action-buttons.vertical. Measured across every card afterwards: 0px offset for both pairs, and the two index buttons share a width the way Edit and Delete do.Testing
1097 backend + 906 frontend tests pass; ruff, eslint and prettier clean. Verified against a real 13-album install: placement, colour, the dialog's wording and buttons, and hide/show across the indexing lifecycle. The confirmation was exercised with Cancel only — no index was destroyed in the making of this PR.
Docs: a new "Rebuilding an index from scratch" section in
docs/user-guide/albums.md, which explains why an update cannot substitute. #401 still tells people to delete the index file by hand; whichever of the two lands second should point at this button instead.🤖 Generated with Claude Code