[damogallagher]

What this adds

Folio already performs on-device OCR on scanned PDF pages to make their text selectable. This PR builds on that existing recognition pipeline so the recognized text is useful throughout the reader and beyond the current session:

• Search scanned documents: recognized text becomes available to document-wide search, with results refreshed as recognition progresses.
• Ask questions about scans: Ask AI can use recognized text alongside embedded PDF text, with a caveat that OCR can contain errors.
• Keep and export recognized text: restore locally saved recognition when a document is reopened, and add a searchable text layer when saving a copy where it is safe to do so. Page reordering preserves the correct mapping.
• Improve coverage and recovery: recognize scan-like pages that also contain sparse native text, avoid duplicating overlapping text, and offer a manual recognition action and retry after failures.
• Control saved data: clear recognized text from Settings or when removing documents from Recent. Password-protected documents keep recognized text in memory only.
• Respect scan languages: reuse main's language selection and installed language packs, and keep saved recognition separate for each language selection.

Recognition remains on-device. English is available by default; other supported languages use the scan language packs provided by main. Ask AI is optional and uses its selected provider. Signed or encrypted files are not rewritten to add a text layer; the original OCR behavior and its limitations still apply.

Validation

• Earlier revisions passed local unit tests, production builds, security checks and macOS application smoke tests; see the PR discussion for the specific commits and results.
• Cross-platform GitHub Actions are required for the current head. The upstream workflow is awaiting repository administrator approval.
• Settings screenshots were verified locally and are not stored in GitHub.

This change was prepared with OpenAI Codex using the Codex desktop harness.

[mberman84]

Thanks for the follow-ups. The latest head (e71576c) passes the existing tests, the build, and the Windows smoke test, including recognizing a scanned page and reopening the saved copy. Two bugs still need fixing before this can merge, plus some cleanup.

1. Stopping and restarting can stop the new run

cancel() sets the phase back to offer right away, so start() can run again while the first run is still starting its workers. Both runs share #cancelled and #workers, so when the old run finishes it calls #stopWorkers() and terminates the new run's worker (and start() clears #cancelled, so the old run carries on as if it hadn't been stopped). See OcrSession.ts start() and cancel().

Each run needs its own cancellation and its own workers (for example, a run token or generation counter that cancel() bumps, with each run only stopping the workers it created).

This test fails on the current head:

it("restarting during startup does not let an old run stop the new worker", async () => {
  const oldStartup = deferred();
  const newRecognition = deferred();
  const oldWorker = { recognize: vi.fn(async () => nothing), terminate: vi.fn(async () => {}) };
  const newWorker = { recognize: vi.fn(async () => { await newRecognition.promise; return nothing; }), terminate: vi.fn(async () => {}) };
  mocks.createWorker
    .mockImplementationOnce(async () => { await oldStartup.promise; return oldWorker; })
    .mockResolvedValueOnce(newWorker);
  const session = new OcrSession(makePdf([makePage({ image: true })]), "k");
  const oldRun = session.start(() => 1);
  await vi.waitFor(() => expect(mocks.createWorker).toHaveBeenCalledTimes(1));
  session.cancel();
  const newRun = session.start(() => 1);
  await vi.waitFor(() => expect(newWorker.recognize).toHaveBeenCalledTimes(1));
  oldStartup.resolve();
  await oldRun;
  const terminatedByOldRun = newWorker.terminate.mock.calls.length;
  const phaseDuringNewRun = session.state.phase;
  newRecognition.resolve();
  await newRun;
  expect(terminatedByOldRun).toBe(0);
  expect(phaseDuringNewRun).toBe("running");
});

2. A failed scan is reported as success in mixed documents

The run ends with failures === total ? error : done, but total counts every page, including pages that already have text and are skipped. In a document with one scanned page and one typed page, if the scanned page fails, failures is 1 and total is 2, so the phase becomes done and the user never sees the error or the retry button. Compare failures against the pages that were actually sent for recognition (and consider reporting a partial failure rather than done).

This test fails on the current head:

it("a failed scanned page in a mixed document reports failure", async () => {
  const worker = makeWorker();
  worker.recognize.mockRejectedValue(new Error("bad scanned page"));
  const session = new OcrSession(makePdf([makePage({ image: true }), makePage({ items: TYPED })]), "k");
  await session.start(() => 1);
  expect(session.pages.size).toBe(0);
  expect(session.state.phase).toBe("error");
});

Please add both tests with the fixes.

3. Cleanup

• Screenshots: drop the Add screenshots of text recognition commit. CONTRIBUTING.md asks for screenshots to be attached to the pull request, not committed, so please attach them here instead.
• Unrelated changes: the contribution and architecture rule edits should come out of this pull request; one pull request fixes one problem.
• Scope: issue #15 asked about OCRmyPDF and choosing a language. Bundled, English-only recognition is fine as a first step from our side, but please say so in the description and note that other languages are a follow-up.

If we haven't heard back by Monday, November 2, we'll make these changes on your branch ourselves so this can ship. Thanks again for the work here.

[damogallagher]

@mberman84 thanks, all of it is done on f83fa82 (also merged with the latest main, no conflicts).

1. Stop and restart. cancel() now bumps a run number, and each run checks its own number and stops only the workers it created, so an old run can no longer terminate the new run's worker or carry on after being stopped. Your test is in OcrSession.test.ts and passes.

2. Failed scan in a mixed document. Failures are now counted against the pages actually sent to the recognizer. Your test is included and passes. I also took your partial-failure suggestion: if any scanned page fails the run ends as an error ("Couldn't read 1 of 3 scanned pages.") with the retry button; pages that were read keep their text and a retry reads only the others.

3. Cleanup.

• Screenshots: no longer in the change. The docs/ocr files are removed (the net diff has none). I did this with a follow-up commit rather than rewriting the branch history; since PRs are squash-merged the result is the same. The description still links the images from the fork's history, and I can't attach files from the command line, so please drag them into the description if you want them hosted here.
• Unrelated changes: AGENTS.md and CONTRIBUTING.md are back to exactly what is on main (no diff).
• Scope: the description now says recognition is bundled and English-only as a first step for Feature request: Integrate OCRmyPDF to make scanned PDFs searchable #15, and that other languages (and choosing one) are a follow-up.

Tests: 301 pass, the OCR code stays at 100% coverage, and CI (this repo's ci.yml) passes on Ubuntu, Windows and macOS: https://github.com/damogallagher/folio/actions/runs/41118344165 . The runs on this PR are waiting for a maintainer's approval.

[mberman84]

Thanks for fixing the earlier stop/restart and mixed-document failure cases; the regression tests for both pass. I reviewed the current head a17da1d and found three remaining issues detailed inline: the lockfile blocks clean installs, search snippets can retain pre-OCR text, and a worker can leak when another worker fails startup.

Validation on Windows in isolated snapshots: a fresh npm ci fails with the missing Mammoth dependency tree. After repairing dependencies only in the temporary snapshot, all 301 existing tests, the build, and the full Windows smoke test pass, including recognition and reopening a searchable saved copy. Two additional focused regression probes reproduce the stale-snippet and late-worker cleanup bugs. No source changes were made to the project checkout.

Please fix these three issues and add the regression tests before merge. Cross-platform CI must also pass at the final head.

Reviewed with gpt-6.1-sol through the Codex harness in T3 Code.

[damogallagher]

@mberman84 Could you please re-review this PR? Your three findings were addressed in d07c694, and the full CI suite passed on Windows, macOS and Linux: https://github.com/damogallagher/folio/actions/runs/41202132162. Thank you!

[mberman84]

I verified the three fixes from the previous review. On macOS, a clean install, all 303 tests and the build passed at d07c694. I also confirmed that the fork CI run passed on all three platforms at that commit.

I tested a local merge with current main (c72f8cf). The document-open path no longer compiles, and an existing Folio profile silently drops recognized text when it tries to save a page. Ask AI also moved its prompt construction to the main process. Please update the branch, fix the three integration points below, and include the v4 upgrade regression. The updated head needs upstream CI on Windows, macOS and Linux before merge.

[joswalt]

Two things I found while test-merging this branch (at d07c694) into current main:

1. Existing users won't get the ocr store. This branch raises the IndexedDB version from 3 to 4 and creates the
ocr store in onupgradeneeded. Since then, main has also moved to version 4 (in 1.10.9, for the Recent migration).
After merging, the two version-4 upgrades combine cleanly, but anyone who has already run 1.10.9 is at version 4, so
onupgradeneeded never runs for them and the ocr store is never created. getOcrPages and saveOcrPage catch the
resulting error silently, so for those users recognized text is never saved and every scan is read again on each
open. Raising the version to 5 should fix it. The existing contains(...) checks already handle each step.

2. It no longer merges cleanly. Merging into current main conflicts in src/AiPanel.tsx. The rest merges
automatically.

A smaller point for the maintainers: the per-page ocr records join the notes and ai stores as data that's never
removed (Clear all only clears Recent). #37 is about letting people remove stored data, which would need to cover this
store too.

[damogallagher]

@joswalt Thanks for catching these. Commit 56d8cea merges current main, resolves the AI panel conflict, and upgrades the database to v5. Regression tests cover upgrades from v3 and shipping v4, preserving notes and v4 Recent entries while saving and restoring OCR pages. The OCR store should also be included when the stored-data controls in #37 are implemented; this PR leaves Clear all’s existing behavior unchanged.

@mberman84 Could you please re-review and test the updated PR? It also fixes the aborted document-open lifecycle and carries a validated OCR flag through both AI providers. The full suite passed on Windows, macOS and Linux in the fork: https://github.com/damogallagher/folio/actions/runs/41242880567. The upstream PR workflow still requires maintainer approval: https://github.com/Forward-Future/folio/actions/runs/41242885738. Thank you both!

[mberman84]

The updated branch addresses the earlier document-open, database migration, and Ask AI issues. I also verified all three platform jobs passed in the fork on 56d8cea.

Please rebase onto current main and build on the page OCR shipped in 1.12.0 so the same page is recognized only once. The inline findings cover starting OCR on a mixed document, recovering from a worker startup failure, and packaging the CPU fallback.

Please also test saving a reordered two-page scan with a different word on each page. The exporter currently indexes the output PDF by the original page number. Map both the word positions and the recognized-page metadata to the output order so each word stays on its own page. Once the changes are ready, we can run upstream CI on all three platforms.

[mberman84]

P2 — Refresh the text layer after OCR on image-only pages. At b72ab07, this path appends word spans directly. Find can report a recognized word while PDF.js still has an empty map of the text spans. I reproduced the exception in PDF.js’s match conversion with that empty map. Please rebuild the visible text layer from the recognized items and test that searching a newly recognized word highlights it on the page without reopening the PDF. Upstream CI for this revision is awaiting approval. OCR code · CI

[damogallagher]

Fixed in 4a0a26f. Image-only pages now rebuild their text layer through PDF.js after recognition, so its search highlighter uses the same text spans as Find. I removed the old manual span rendering and added an app regression that searches for and highlights a newly recognized word before reopening the PDF.

Locally on macOS, the build, 250 unit tests, and full app smoke test passed, including selection on a rotated scan and mixed-document OCR. The updated cross-platform checks are pending.

[mberman84]

Please make two changes before merge.

1. When someone searches while a scan is still being recognized, zero matches can look like a final result. Please show that results are incomplete until recognition finishes, or refresh matches as pages become ready. Check this by searching for a word before its page is recognized and leaving the search open.
2. Please make the Clear all confirmation say “Remove all recent files and all saved recognized text from this device?” It also clears recognized text for PDFs that are no longer in Recent. Choosing Cancel should preserve both the recent files and their saved text.

[damogallagher]

Thanks, @mberman84. I've addressed both points in 1d92925:

• Search now shows that results may be incomplete while text recognition is running. The desktop test searches before recognition, leaves the search open, and checks that the recognized word appears and is highlighted afterward.
• Clear all now uses the requested confirmation text. The test checks that Cancel preserves recent files and saved recognized text both for recent files and for files no longer in Recent; confirming clears them.

The fork CI run passed on Windows, macOS and Linux at this commit, including the existing OCR regressions. The upstream PR CI is still awaiting maintainer approval.

Could you please re-review when you have a chance?

[damogallagher]

Hi @mberman84, I've merged the latest main into the PR and resolved the package.json conflict in 0cf15d7. The resolution keeps both the OCR unit-test command and the new thumbnail-test command, along with main's 1.15.2 package and lockfile versions.

The fork CI run passed on Windows, macOS and Linux, including the OCR and thumbnail tests and desktop smoke checks. GitHub now reports the PR as mergeable. The upstream PR CI is still awaiting maintainer approval.

Could you please re-review when you have a chance? Thanks!

[mberman84]

Re-reviewed at a18613a. Both points from 1d92925 are fixed:

• Search shows "Results may be incomplete while text is being recognized" while recognition runs, and the desktop test checks that the newly recognized word gets highlighted afterward.
• Clear all now asks before removing recent files and saved recognized text, and Cancel leaves both alone.

The PR conflicts with main again, in 14 files. Could you merge the latest main in once more? This one is more than mechanical, because scan languages landed in #66:

• recognizeScannedPages on main now takes the chosen scan languages and returns a cleanup function, which the app calls when the language changes. It also reports a missing language pack, reads packs from a read-only cache, and encodes the page before sending it to the recognizer (21f0c6d). Please keep that behaviour.
• Saved recognized text is keyed by document and page only. With scan languages, text saved under one language would be reused after switching to another. Please put the scan languages in the key, or clear the document's saved text when the language changes, and add a test for it.
• SettingsDialog.tsx and recent.ts also changed in Choose scan languages and download them once for offline reading #66, Clear saved annotations and chats separately in Settings #67 and Fix ten problems found in a code review of links, Ask AI, shortcuts and search #80, so the saved-text clearing needs to fit in with the separate clearing of annotations and chats.

Once it's conflict-free I'll approve the CI run and do a final pass.

---

Generated by Claude Code

[mberman84]

The earlier review requests are addressed. Changing languages reloads the matching OCR cache. Clearing saved scan text preserves annotations and chats. Search refreshes when recognition finishes. The export test verifies that pages whose text cannot be embedded remain eligible for recognition.

I checked the recognized-text warning through both AI providers after resolving the merge conflicts. I also reproduced an invisible OCR status box intercepting clicks on the PDF and verified the fix in the reader. The retry button still works. Windows, macOS, and Linux passed on 846bd16.