Skip to content

PT-4911 + PT-4921: Hidden-view clipboard filters and lossless paste - #43

Open
tjcouch-sil wants to merge 157 commits into
mainfrom
pt-4921-lossless-paste
Open

tjcouch-sil wants to merge 157 commits into
mainfrom
pt-4921-lossless-paste

Conversation

@tjcouch-sil

@tjcouch-sil tjcouch-sil commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

TL;DR

This PR combines PT-4911 and PT-4921, the two clipboard tickets for the views that hide USFM markers (Simple mode's paragraph-structure view, the formatted view, block-verse). It replaces #41 (closed as superseded). Companion: paranext/paranext-core#2976, which likewise absorbs #2969. In these views:

  • Copy gives prose. No gutter paragraph markers (\q1 ) and no hidden figure or unknown-construct text; a stored no-break space is written as ~ in the plain text.
  • Paste, drop and typing never refuse for \ or //. Each \ and each // is removed from the inserted text and the host is told through onUnstorableTextRemoved. A single / ("and/or", 1/2) is kept. A delete, cut, transpose or drag-move that would join two / into // is refused instead.
  • Ctrl+V and menu Paste within the editor are lossless. Character attributes, footnote details (\cat), milestones, figures and other unknown constructs survive, because the copy carries the editor's own format in an invisible data-* attribute of its text/html. A Select All copy keeps one book line and one chapter line, and a copy from Standard view or the Footnotes pane loses its marker tokens instead of pasting marker names as words.
  • Line breaks and drags behave. Shift+Enter is Enter (as in Paratext 9), a line break inside inserted text goes in as a pasted one, and the editor performs a within-editor drag-move itself so the text lands where it was dropped and never loses a /.
  • Native paste in Standard and the Footnotes pane pastes as Ctrl+V does (no more thrown or unsaved pastes on macOS's Edit > Paste).

Standard and Markers views keep their own \ and // handling.

Bundled on purpose (review item 13): three fixes that are not clipboard changes ride in this PR: the loader fix (usxStringToUsj reads USX by XML 1.0 line ends, so a stored U+2028, U+2029 or U+0085 is no longer lost), entered control characters becoming spaces in every view, and the U+000B settle break (a Standard-view marker settle no longer drops a U+2028). Each is small, has its own tests and its own commit; they came out of the U+2028 work. A squash-merge revert of the clipboard work would take them with it, which is accepted.

"Catalog" labels (CB-6, T-18, EV-5, F-9, S5, ...) are row ids in paranext-core's .context/standards/Scripture-Data-Fidelity-Catalog.md (CB = clipboard, T = typing, EV = entered values, F = file/load fidelity) and docs/clipboard-semantics.md (S5).

Public API

@eten-tech-foundation/platform-editor:

  • EditorOptions.onUnstorableTextRemoved: new optional handler, called synchronously from the editor command or update that made the edit, once per paste, drop or blocked keystroke that removed something (and once per refused delete or drag-move).
  • UnstorableTextRemoved: the payload. source is "paste" | "drop" | "typing" | "deletion" | "move"; removed is a readonly array of UnstorableCharacter.
  • UnstorableTextSource: new exported alias for the source union, named in the CHANGELOG's Hosts note.
  • UnstorableCharacter ("\\" | "//") and UnstorableTextRemovedHandler: the other two types.
  • Everything else changes behavior only, with the same signatures. The fixes beyond the clipboard change no public type.
  • Internal (shared-react) optional props: ClipboardPlugin.pastesAsClipboardRead/logger, ContextMenuPlugin.logger, StructureKeyboardPlugin.nodeCopyNamespace/removesMarkerTokens/logger.

Host action and merge order

  1. Merge this PR (scripture-editors#43) first.
  2. In paranext-core, run npm run move-platform-yalc.
  3. Merge PT-4911 + PT-4921: Hidden-view clipboard notices, e2e and catalog paranext-core#2976. Its CI is red until step 2: the new editor option and types aren't in the pinned editor.

The host shows the notices (paranext-core#2976).

Decisions for the reviewer

PT-4911:

  • Locked structure + a stray marker: under protected (Simple's default), the locked html path's DOMPurify drops the <unmatched> tag, so a copied stray \* pastes as * (with the notice). Accepted by TJ for now. A two-line DOMPurify allowlist would keep it; unlocked and Ctrl+V-through-the-editor-format routes already keep it.
  • Plain-text-only paste of a copied stray marker (\*) gives * with the notice, the same rule as \v 9 x → v 9 x.
  • IME-composed \ or // is not filtered (documented in the plugin's TSDoc).
  • A refused delete during a drag-move leaves the text in both places: the editor now makes the move itself, but its insertion is judged separately from the removal, so a refused removal turns the move into a copy, with the "not deleted" notice. Nothing is lost.
  • A cut over hidden content that would join two /: HiddenContentDeleteGuardPlugin copies before it asks, so such a cut used to leave the copy on the clipboard and remove nothing. Since 3e0000d8 the guard does not ask before a cut that UnstorableTextPlugin refuses, so that cut is refused without a question and copies nothing (CHANGELOG and S5 updated in 37b7a491).
  • Pre-existing, not changed here: under protected, a glyph-only html paste over a selection removes the selection (StructureKeyboardPlugin inserts an empty list).
  • Formerly tracked in PT-4921, now fixed here: the html route lost attributes and note details, and a caption-less figure was left out of the editor's own clipboard format, so cut → paste back of a range holding one lost it.

PT-4921:

  • Why an html attribute (TJ, short term): a host-driven native paste needed a main-process command a browser page on the unauthenticated PAPI WebSocket could reach, and web custom formats need a second async write that fails when unfocused. The attribute holds the whole node copy, hidden constructs included, for any program reading the html. Probed in Electron 39 on Linux and Windows up to 5 MB; macOS untested. Long-term direction: USFM on the clipboard using Standard view's tokenizer (PT-4984).
  • Standard view keeps its tokenizing paste (TJ), so a fix there also helps USFM from other programs; its PT-4388 pins stay.
  • Html from another program stays lossy (the html export carries no attributes/figures; TJ moved this to PT-4984). Pinned as "accepted" in the corpus.
  • Accepted: a plain-text paste that starts a paragraph with a space drops it; ParatextData drops a paragraph's plain leading spaces on save anyway (TJ).
  • Pre-existing, Standard only: a paste the browser runs itself (a host menu's own Paste, a drop) stores a pasted tab as a literal tab character.
  • No paragraph- or chapter-attribute rows: no fixture has a non-derived paragraph attribute, and a chapter line is never in a node copy by design.
  • A paste with neither sign (paste as plain text, or a marker-showing copy that lost its html's mark going through another program) still leaves marker names as words, with the notice. A Ctrl+V of a Standard or Markers copy carries the mark, so its tokens go silently even with no node copy. A book line and chapter tokens now go whole, with their code, title or number, and a line left empty (a bare \p) goes too, matching Standard view's own paste (stripPastedChapterAndBookId, shared by both views); a verse's number still stays as text, and so does a space the user wrote. A \ inside an attribute value counts as a marker (src="c:\pics\a.jpg" → src="c:.jpg"): rare, and fixing it needs real tokenizing.
  • The editor performs a within-editor drag-move itself in hidden-marker views, rather than letting the browser remove and insert. It is the only way found to put the text where it was dropped, and it keeps every guard and filter by dispatching the same commands. A refused move gets its own notice (source: "move"), since nothing was removed.
  • Footnote popover and Footnotes pane editors are not covered by the chapter-load pause (out of scope).

Beyond the clipboard:

  • U+000B is reserved for the settle's display-only line break (TJ, 2026-11-11). It is safe because no text can hold one. Entered control characters become spaces and loaded text is never changed (paranext-core ADR adr-editor-control-characters-entered-as-spaces).

Known gaps

  • macOS: unverified. If the html attribute didn't survive the clipboard there, a paste would fall back to the lossy html. The macOS Ctrl+T transpose refusal is modelled in jsdom only.
  • jsdom-only models for Selection.modify (word and line deletes) and caretRangeFromPoint (the drop point).
  • beforeinput target-range handling (insertClaimedBeforeInputText applying the event's first target range) is unverified in a real browser.
  • IME composition and Android Chrome's browser-applied Backspace are not filtered or judged.
  • A crafted node copy can still hold a char whose marker is a paragraph marker (p), which saves as \p …\p*. The paste check tests a marker's shape, not its kind, since real copies hold markers outside the node classes' lists (\cp and \zunk paragraphs, a \cat char in a note).
  • Hand QA in the app: done by TJ on Linux (WSL), 2026-11-18, both interface modes. It found two bugs, both fixed in this PR: a cut of a range lying inside a collapsed footnote copied part of the note and removed all of it (catalog CB-25, scripture-editors ee572e4d; TJ re-tested it), and with structure locked a drag-move did nothing, because the drop went in at the selection rather than the drop point (CB-26, 85c447a4; unit tests and the drag-move e2e, not yet re-tested by hand).

Testing at HEAD

After merging main (#55):

  • nx typecheck and lint (9 projects): green.
  • vitest: platform 8027 passed, 99 expected-fail, 165 skipped; shared-react 2737 passed; shared 951 passed; utilities 112 passed.
  • paranext-core e2e in-editor-paste, hidden-view-clipboard and hidden-view-drag-move against this tip staged: passed (3/3).
  • Committed dist verified current (node scripts/verify-committed-dist.mjs).

After the two hand-QA fixes (ee572e4d, 85c447a4):

  • nx typecheck and lint: green.
  • vitest: platform 8039 passed, 99 expected-fail, 165 skipped; shared-react 2737 passed.
  • paranext-core e2e hidden-view-drag-move against this tip staged: passed.
  • Committed dist verified current.

After review round 5 (62736de4..e159a5cb):

  • nx typecheck and lint (9 projects): green.
  • vitest: platform 8128 passed, 99 expected-fail, 165 skipped; shared-react 2799 passed; shared 951 passed; utilities 112 passed.
  • Committed dist verified current.
  • paranext-core e2e hidden-view-clipboard, in-editor-paste, hidden-view-drag-move and clipboard-usfm-round-trip against this tip staged: passed (4/4; the cold first test needed one retry).

After review round 4 (3f126398..87b1befb):

  • nx typecheck and lint (9 projects): green.
  • vitest: platform 8106 passed, 99 expected-fail, 165 skipped (one marker-menu test timed out at 5 s under load and passed alone); shared-react 2741 passed; shared 951 passed; utilities 112 passed.
  • paranext-core e2e hidden-view-clipboard, in-editor-paste, hidden-view-drag-move and clipboard-usfm-round-trip against this tip staged: passed (4/4; the cold first test needed one retry).
  • Committed dist verified current.

Not done:

  • macOS: not checked on a Mac.
  • Hand QA of this round's fixes.

AI Involvement

AI-assisted (Claude Code): implemented task by task by AI subagents from written plans (including Electron transport probes on Linux and Windows), each task spec- and quality-reviewed by a separate AI reviewer, then a whole-branch AI review and fix wave per ticket, an automated /review-paratext run per ticket and a /code-review high round per ticket. TJ ruled on the open product and scope questions.

Risk Level

Medium — changes every clipboard route, merges this branch's copy writer into PT-4126's, and changes the html of every copy in Simple mode's editable view and the formatted view, plus drag-move and line-break handling. The fixes beyond the clipboard change what an entered control character becomes in every view, and how usxStringToUsj reads USX for every consumer (narrowly: only line-end normalization). Covered by the corpus suite in all three structure-protection modes, and falls back to today's html path when the attribute is missing or unreadable.

History

Review round 5 (Katherine, 2026-11-19)

  • 1 (crafted node copies, High): 11979192. Every field each allowed type reads or saves is checked: attribute names may not be saved fields, attributeOrder is a list of names, node state only the editor's own keys, unmatched markers end in *, text is a string, element format fields are defaults. One unit test per check (nodeCopy.utils.test.ts), each mutation-verified.
  • 2 (copy inside a collapsed note set up its deletion, Medium): 62736de4. The selection goes back after the copy.
  • 3 (glyph copy on a text-led line, Medium): 488cae84; a glyph-only cut now writes and removes nothing too.
  • Optional items: docs, CHANGELOG and test narration (dbdded4a, 62736de4); text: null refused and the 1000-level case removed (11979192).

Review round 4 (Katherine, 2026-11-17)

  • 1 (crafted node copy, High): 3f126398. nodeCopyProblem checks each entry's placement, marker, numbers, attributes, text fields and nesting depth (32 levels) before any node is made. immutable-typed-text and typed-mark leave the allowlist. Each probe is a crafted-payload test, with a positive control.
  • 2 (protected paste skipped the question, Medium): d18947d6, acefb5a9. Under protected structure, $predictInsertion judges the sanitized nodes, and a paste whose paragraphs are all filtered out inserts nothing (the sanitizer's separator spaces don't count), as off and guarded do, so nothing is asked.
  • Optional items: backward-selection test (cd5331ea); doc names and module (cd5331ea); namespace doc and test comment (3f126398); \ca/\cp left out with their chapter (12fa0a4c), then reverted (7bdd023b): they are markers of their own and keep their content; reconsideration asked of the reviewer. CodeQL PT-4730: Limit how much text can be copied from the editor #23–PT-4703: Ask before an edit removes content the view does not show #27 are dismissed as false positives; the node-copy carrier is built in an inert document.
  • Gutter-glyph copy (comment, 2026-11-18): ce419fb6. A copy of only gutter glyphs writes nothing.

Review round 3 (Katherine, 2026-11-13)

Findings 1-26, all answered in the PR thread. Where the code changed:

  • 1 (crafted node copy): 33e9459f. $nodesFromNodeCopy checks every entry's type, at any depth, against the types a hidden-marker view's document holds before it makes any node (RootNode.importJSON returns the live root, so a later check is too late); an already-attached made node is rejected too. Either case falls back to the html with a warn. Six new htmlClipboardRoute.test.tsx cases.
  • 2 (false // refusal across a chapter line): 7d68e042. The judge uses the ranges the removal uses ($rangesRemovedKeepingBookAndChapterLines).
  • 3 (Standard copy's \id/\c): 6f126f49. withoutMarkerTokens drops book lines and chapter tokens with their payload and a line it leaves empty.
  • 4 (guard asks before an html paste that inserts nothing): cfa219ff. The guard asks $predictInsertion (hiddenViewInsertion.utils.ts), the prediction a drop's filter uses, so it judges every route.
  • 6 ($runEnter fakes a keydown): documented in 2a3ba42e; no shipping path reaches the wrong case.
  • 8, 9 (parse the paste once; split UnstorableTextPlugin): moved to PT-4984, which rewrites this routing for USFM on the clipboard. The route decision and the insertion prediction already live in hiddenViewInsertion.utils.ts.
  • 10 (two join rules): cfa219ff, d923189b. $canSplitAt and joiningText are shared by LineBreakPlugin and the plain-text paste.
  • 5 (target range dropped): 2b5e1e2f. Both claims go through insertClaimedBeforeInputText.
  • 7 (mount order): af96d243, 2b5e1e2f.
  • 11, 12, 16-18, 20, 22 (docs, CHANGELOG): af96d243, 2a3ba42e (UnstorableTextSource exported).
  • 13: kept bundled (see TL;DR).
  • 26 (node-copy format split across packages): d0f64479; writer, readers and namespaces in nodeCopy.utils.ts.
  • 15, 23: f860044d (a dead selection restore removed; transferWith exported and a plainTextTransfer added for the test transfers). 24: 72a13793. 25: 7d68e042.
  • 14 (hand QA): done by TJ, 2026-11-18 (see Known gaps). 21: this description restructured to the reviewer's skeleton.

Behavior details by ticket

PT-4911: Copy prose, and filter \ and // instead of refusing

  • Copy no longer writes gutter paragraph markers (\q1 ) or hidden figure/unknown-construct text. A collapsed footnote's text stays in the plain-text copy, by design (catalog CB-6). A stored no-break space is written as ~ in the plain text unless allowInvisibleCharacters is on (hiddenViewCopySpelling, the one spelling path for these views); the html and node copies keep U+00A0 (catalog EV-5). The copy is written by main's HiddenMarkersViewCopyPlugin, which this branch's own copy writer was merged into (see "Merged with PT-4126").
  • Paste and drop are never refused any more. Every \, and every // scanning left to right (/// leaves /), is removed from the inserted text, on every insertion route (Lexical's node route, its plain-text route, drops, and the locked-structure html path). A / next to an existing / counts as forming //. A single / ("and/or", 1/2) is kept. Stray-marker (unmatched) nodes are left alone. Since main's StructureKeyboardPlugin inserts a protected paste through SELECTION_INSERT_CLIPBOARD_NODES_COMMAND, UnstorableTextPlugin filters that paste even without the sanitizer's own filter; the editor still passes the sanitizer's filter so a drop is reported as a drop.
  • Typing a single / works; \, and a / beside a /, are blocked. A / beside a / is judged across a comment or annotation mark's edge too.
  • An optional line break counts as the // the file stores for it. The view shows nothing there, but a / against one saves as ///, and with a / on its other side as ////, two breaks, which hides both / after the next load. So a / typed or pasted against a break is refused or removed, and a delete that would bring a / against one is refused, as for any other /. TJ confirmed this rule: a lone / typed right beside a hidden break is refused even with no / on the break's other side.
  • A macOS transpose (Ctrl+T) never joins two / into //. Chromium raises it as a beforeinput insertTranspose, which Lexical applies past the filters, so /x/ could become //x. UnstorableTextPlugin refuses a transpose that would put a / against a / (an optional line break counts as //), judged as the Backspace and Delete at that caret ($transposeJoinsSlashes). It keeps the caret in place and reports source: "typing". Modelled in jsdom only; unverified on a real Mac.
  • Deleting never joins two / into //. Deleting b from a/b/c saved a//c, which the next load reads as an optional line break (the view then showed ac). A delete whose result would put a / against a / it was not already touching is refused, and reported once through onUnstorableTextRemoved with the new source: "deletion". It covers Backspace and Delete with their word and line chords, keydown-less deletes, selection deletes, a selected paragraph marker, a paragraph join (guarded structure's first press), cut and a drag-move's removal. Under protected structure a join stays protection's own refusal. An authored zero-width space between two / counts as text (b7647eca), and a range running out of an expanded note is judged in its paragraph (85ed24d9), so neither is wrongly refused. Known gaps (S5): IME composition, and Android Chrome's browser-applied Backspace.
  • "Delete hidden content?" is not asked for nothing (3e0000d8). HiddenContentDeleteGuardPlugin no longer asks before a cut that UnstorableTextPlugin then refuses for joining two /, nor before a paste that inserts nothing; pasteInsertsNothing now judges the paste against the characters around the selection.
  • The host is told through a new EditorOptions.onUnstorableTextRemoved, called once per paste, drop or blocked keystroke that removed something (and, with PT-4921, once per refused drag-move). Hosts show the notice (paranext-core#2976).
  • Locked structure: a plain-text paste of several lines joins them with one space instead of starting a new paragraph (PT-4921 refines the join so it never joins two words).
  • Marker glyphs in pasted or dropped html (a drag across a paragraph break, an older copy) are dropped in these views, as the locked path already did, so a marker name never arrives as text.

UnstorableTextPlugin is mounted in every view and acts only where markers are hidden (isEnabled), so an editor that switches into a hidden-marker view keeps its delete handlers ahead of the other plugins' at the same priority. Mounted only in hidden views, it registered behind ParaMarkerSelectionPlugin's after a switch, and deleting a selected paragraph marker could join a/ and /c. Standard and Markers rows now pin that every handler stands aside there when disabled.

PT-4921: Lossless Ctrl+V in hidden-marker views

Copy and paste within the editor is now lossless in these views: character attributes (incl. the "not closed" state), footnote details (\cat, attributes), milestones, alternate and published verse numbers, figures and other unknown constructs all survive Ctrl+V, the context menu's Paste and the Edit menu's Paste.

  • Editor format in the html: Ctrl+V and menu Paste read the clipboard through the browser's async read, which never returns a custom flavor. So hidden-marker views' copies also carry the node copy in an invisible data-* attribute of text/html, written during the copy event, and paste reads it back. A native paste (the macOS app menu's Paste) sees the flavor as well. No host command is needed. Standard and Markers views keep their route.
  • Tagged editor format: hidden-marker views write the node copy under the namespace platformEditor/hidden-markers, without gutter glyphs and keeping content-less constructs (a caption-less figure), and paste it losslessly. A Standard-view copy pasted here takes its html/plain text, and a hidden-view copy pasted into Standard view is foreign to it, so neither view family rebuilds the other's node shapes. The locked-structure path flattens the tagged copy like html and keeps everything else.
  • Marker-showing copies: a copy from Standard view or the Footnotes pane's row editor pasted into a hidden-marker view loses its marker tokens instead of leaving marker names as words (1 \ft TR and → 1 TR and ). Verse numbers, a note's + caller and its reference stay as text; book lines and chapter tokens go whole, and so does a line left empty.
  • Select All keeps one book and one chapter line, and chapter 1's front matter (hidden-marker views). Under off/guarded, Select All includes the \id and chapter lines (and in chapter 1, \h/\mt1/the introduction between them); the copy used to drop its node copy for them, so Ctrl+V read the html and added a second chapter (every later save fails), or lost the chapter line, the title and every note's \cat. Now the copy writes its node copy; no paste route (node copy, html, the locked sanitizer) inserts a book or chapter line; a copy holding a chapter line splits around the document's (front matter before it, Scripture text after) only when pasted over a selection holding it or from a caret right beside it (the start of the line after it, where a cut leaves the caret, or the end of the line before it), and anywhere else goes in at the caret like any paste; a cut, or a paste over a selection, keeps each book and chapter line, drops each run of lines it covers whole without leaving an empty paragraph, and leaves the caret in the Scripture text. Select All → cut → paste back restores chapter 1 exactly, and pasting a Select All copy over Select All changes nothing. A whole chapter-1 copy pasted at the end goes in there, front matter and all; pasted into any other chapter, its front matter goes after that chapter's line with the rest (only chapter 1 has lines before its chapter line). Plain text of several lines pasted over Select All keeps its lines (b8687c24); only a paste that lands in the \id line joins them. Protected structure still refuses the cut and the paste-over. A delete (Backspace, Delete, word/line chords), typed text, Enter or a drag-move over such a selection used to remove both lines and type into the \id line (older than this PR); the new BookAndChapterLinesPlugin (hidden-marker views, LOW priority, after every guard) removes the selection the way the cut does, so typing over Select All goes in after the chapter line. Pinned by hiddenViewSelectAllClipboard.test.tsx and hiddenViewSelectAllEdit.test.tsx (each case on a chapter with front matter and one without, plus a chapter-1 copy pasted into chapter 2, a drop, a line delete and a line break, and carets near the chapter line that must not split).
  • A chapter line a selection holds alone is kept, and a paste after a chapter line with no paragraph after it goes after it. A selection holding only the chapter glyph (both ends on the root) used to let a paste, typing, Backspace or Delete remove the chapter line. A paste over a selection ending on a chapter line followed by a table used to go into the \id line.
  • A copy started partway into a front-matter line rejoins it across the chapter line. Such a copy marks its first paragraph with the node state startsMidParagraph, carried in the node copy and so in the html route too. When it is pasted around the chapter line, that paragraph joins the line that a paste without the split would have joined. So a mid-line cut pasted back restores \ip An introduction to the book. exactly, instead of splitting it in two. Whole-line copies, older copies without the mark, and Standard view are unchanged. The mark is stripped on paste, so no node in the document keeps it.
  • A Standard-view copy ending on the editor root copies the last paragraph. The copy walker took a boundary node's offset from Lexical's $getCharacterOffsets, which answers 0 for an element point short of its element's end, so a selection ending on the root between two paragraphs copied none of the last one's text (and a stray line break), while the cut removed it. It now places each end by position. No gesture in Standard view is known to make that selection; fixed as low effort and low risk. Pinned by standardCopyElementPoints.test.tsx.
  • A native paste pastes as Ctrl+V in the views that show markers. macOS's Edit > Paste and Shift+Insert deliver the editor's application/x-lexical-editor flavor, and Standard view and the Footnotes pane's row editor rebuilt nodes from it: a cut collapsed footnote pasted back threw, a paste over a note's \ft landed outside the note, and a pasted paragraph marker showed but wasn't saved (PT-4872 cases 2, 4, 5, 8). ClipboardPlugin's new pastesAsClipboardRead (set by Editor for every markerMode but "hidden") claims such a paste ahead of every other paste handler and re-dispatches it without that flavor, so it takes exactly what Ctrl+V delivers; Standard keeps its tokenizing paste. Trade-off: a ref's wrapper and loc are lost on a native paste, as on Ctrl+V. In Paratext 10 that changes no saved byte: its data provider never makes a ref node from a project's USFM, and Paratext does not write a ref wrapper to USFM either (paranext-core catalog F-2). Pins: nativePasteFlavor.test.tsx, ClipboardPlugin.test.tsx; unknownClipboardFidelity.test.tsx's full shape now pastes as plain+html (changed on purpose); the corpus runs native-paste rows in Standard and the pane row.
  • A pasted optional line break against a / is left out. The file stores a break as a bare //, so a break pasted after a/, before /b, or between a pasted x/ and y saved ///. $removeUnstorableTextFromNodes removes a pasted break that a lone / touches, in the pasted text or in the document's text at the insertion point, and reports it as // through onUnstorableTextRemoved. A break beside text or another break (////) goes in as before. Covers the node copy, Ctrl+V's html, drops and locked structure's sanitizer; $textAroundSelection now reads a neighbouring break as //.
  • A drag-move never loses a /, and lands where it was dropped (catalog CB-23, CB-24). With structure off or guarded, the browser removes the dragged text before it inserts it, and the drop's filter then left out a / that would land against a / (or an optional line break), after its source was already gone: dragging the lone / out of a/b to just after c/ saved ab c/. UnstorableTextPlugin now judges the drop at DROP_COMMAND, before anything is removed, and refuses a move that would leave anything out, reported once with the new source: "move" (12ff8fce). Separately, a move within one line went in at the start of the line, because Lexical rewrites a changed text node whole (setTextContent), which moved the browser's drop point to the node's start. So the editor now performs every within-editor drag-move in these views itself, in one update and one undo step, through the same REMOVE_TEXT_COMMAND and CONTROLLED_TEXT_INSERTION_COMMAND the browser's move dispatches, so every guard and filter still applies (9ff56908). A drop at or inside what was dragged changes nothing. Copy-drops and drops from other programs are unchanged. Under protected structure a drop goes in at its drop point too (85c447a4): StructureKeyboardPlugin inserted it at the selection, still the dragged text, and now leaves a move within the editor to this move.
  • Several pasted lines in a line with no marker keep their breaks (pre-existing on main; catalog CB-21). Plain text of several lines pasted, dropped or typed with Enter into the blank line after a bare chapter line, or into text with no paragraph marker, used to save run together as onetwo.
  • Enter in a line with no paragraph marker starts a \p paragraph. This covers the blank line after a bare chapter line, and unmarked text in Standard view too. Enter used to make a second unmarked line, which saved run together (\c 2 onetwo). ImpliedParaNode.insertNewAfter now creates a \p, and a paste's line breaks use the same split. The Enter menu's marker choice now applies there too. Protected structure still refuses Enter.
  • Shift+Enter is Enter. Paratext 9 parity: its key handler never checks Shift. A line break has no USFM form and used to save as nothing, joining the text either side. Every route that makes a line break now runs an unshifted Enter, with Enter's own refusals: Shift+Enter, macOS Ctrl+O, beforeinput insertLineBreak, and a typed newline. Pasted or dropped <br> and a node copy's line breaks split the paragraph. They become a space instead under protected structure, in a note, or on the \id line. A lone pasted \r counts as a line break. New LineBreakPlugin is mounted in every view.
  • A line break inside inserted text goes in as a pasted one. This is text inserted as one string, from dictation, autocorrect, a yank or an extension. Its line breaks used to stay as characters and saved as a space (\v 1 first x yverse text). Now each \n, \r\n and \r splits the paragraph, or becomes a space, exactly as a pasted line break does. In Standard it goes in as Standard's plain-text paste, whose \id-line paste now joins lines with a space instead of running them together. PT-4911's filter still strips \ and // from the whole string, with one notice. IME composition is left alone.
  • U+2028 and U+2029 are characters, not line breaks (f2857c60, catalog T-18). As in Paratext 9, the line and paragraph separators stay in the text as characters wherever they are typed, pasted or dropped. One the loaded text holds is kept even when typed text holding a line break lands in the same text node; that used to split it (load \p a<U+2028>b, type x\ny after b: saved a / bx / y). Only \n, \r\n and \r split. A U+2028 also now survives Standard view's marker settle and loading from disk; see "Data-fidelity fixes beyond the clipboard".
  • A joined plain-text paste joins no two words. This applies where a paste's lines are joined: protected structure, or the \id line in any mode. Each run of line breaks now becomes one space, or nothing next to whitespace. Before, a trailing line break joined the paste to the next word (x\n before verse saved xverse), and a line break next to a space doubled it. A trailing line break at a paragraph's very end still leaves a trailing space, as typed text does.
  • A failed clipboard read is logged. Ctrl+V, Ctrl+Shift+V, the right-click menu's Paste items and EditorRef.paste/pastePlainText now log a failed read to the editor's logger instead of leaving an unhandled rejection. They paste nothing from an empty clipboard, and nothing into an editor that became read-only during the read. The permission-denied alert(), which could never show, is removed, and Paste as Plain Text reads the clipboard once. ClipboardPlugin and ContextMenuPlugin take an optional logger.
  • Robustness: an unreadable node copy (any program can write that clipboard type) falls back to html/plain; a pasted footnote's caller gets the host's click handler; a pasted tab node saves as a space (in every view, since ClipboardPlugin registers it); plain prose pasted under protected structure stays inside the char span it lands in (main's SELECTION_INSERT_CLIPBOARD_NODES_COMMAND route does this; this branch pins it with a test). StructureKeyboardPlugin reads its paste options through refs, so a view switch no longer re-registers its handlers behind others at the same priority (3ad9745c).

Commits and their tickets (oldest first)

  • PT-4911: the first 33 commits, db86453f through 100e8bbe. That is everything PT-4911: Hidden-marker views copy prose and filter \ and // on paste #41 held (up to e7a9373c), plus its /code-review high fixes 3e0000d8, b7647eca, 85ed24d9 and a dist rebuild.
  • PT-4921: everything after, 25c7e77a through 36789846. The first commits add a host-driven native paste, which the html-attribute route replaces; ae302ac3 removes it, so it leaves no net diff. This range also holds fixes where PT-4921's paste meets PT-4911's filter (a pasted optional line break against a /, the drag-move fixes).
  • Merge of main: eca22e85 merges origin/main, bringing in PT-4126 (PT-4126: Copy hidden-marker views' notes as prose, without layout spacers #49, "Copy hidden-marker views' notes as prose, without layout spacers"); 37b7a491 (CHANGELOG) and f191b099 (dist) follow it. The branch is up to date with main. See "Merged with PT-4126" below for how the two copy writers became one.
  • Data-fidelity fixes beyond the clipboard (after the merge): 4e389fe7 (entered control characters go in as spaces), f46ffc9f (a marker settle keeps a U+2028), 4c144a28 (dist), 0c42ac47 (USX read keeps U+2028, U+2029 and U+0085) and 36789846 (utilities dist). They grew out of the U+2028 work in PT-4921 and touch typing and loading, not only the clipboard; see "Data-fidelity fixes beyond the clipboard" below.

Since the /code-review high rounds

One round ran on each ticket's pair of PRs. Its fixes are folded into the bullets above; for reviewers who read the earlier descriptions, they are:

  • PT-4911: 3e0000d8 (no hidden-content question before a refused cut or an empty paste), b7647eca (an authored zero-width space counts as text when judging a / join; $isCaretHost instead of $isCursorPlaceholderOnlyText), 85ed24d9 (a range running out of an expanded note is judged in its paragraph).
  • PT-4921: b8687c24 (multi-line plain text pasted over Select All keeps its lines), 3ad9745c (StructureKeyboardPlugin paste options through refs), 88f8e72a (removed BookAndChapterLinesPlugin's unreachable line-break handler: LineBreakPlugin claims it first and runs Enter), 100ee439 (doc fixes).
  • New since then: the drag-move fixes 12ff8fce and 9ff56908 (with 2f0eb805 moving the drop point's DOM range into dropPoint.utils), and U+2028/U+2029 kept as characters f2857c60.
  • No finding from these rounds is left open.
  • 37b7a491: the CHANGELOG now records the drag-move changes, including the "move" source as a public API change, and that a cut over hidden content that would join two / is not asked about and copies nothing.

Merged with PT-4126 (#49)

main's PT-4126 copies hidden-marker views' notes as prose, without layout spacers, through its own HiddenMarkersViewCopyPlugin. This branch's HiddenViewCopyPlugin did the same job, so the merge (eca22e85) keeps one plugin, main's HiddenMarkersViewCopyPlugin (at COMMAND_PRIORITY_LOW), and deletes HiddenViewCopyPlugin.tsx; its tests moved to hiddenViewCopy.test.tsx. A hidden-marker view's copy now writes:

  • text/html: the view's html, carrying PT-4921's node-copy attribute.
  • application/x-lexical-editor: PT-4921's tagged node copy (platformEditor/hidden-markers).
  • text/plain: PT-4126: Copy hidden-marker views' notes as prose, without layout spacers #49's $selectionPlainText (no note spacers or empty-span placeholders, one space at a note's edges, ~ spelling), filtered by $isLeftOutOfHiddenViewCopy (no gutter glyphs, no hidden captions).

Other parts of the resolution:

  • registerViewCopyWriter gained an optional $removeRange, so a cut keeps the book and chapter lines.
  • PT-4126: Copy hidden-marker views' notes as prose, without layout spacers #49's two declines are kept: a range with no text to write, and a copy whose page selection is outside the editor. Read-only views' test suites now clear the page selection before copying, since the second decline would otherwise apply in jsdom.
  • InvisibleCharactersPlugin's copy handler is removed (as on main); hiddenViewCopySpelling is the one spelling path.
  • CopyLimitPlugin combines main's $overLimitText and fit with this branch's hidden-text predicate.
  • PT-4126: Copy hidden-marker views' notes as prose, without layout spacers #49's "keeps Lexical's own html and internal flavors" tests now expect the view's own html and node copy (spacers kept), and its CHANGELOG line says they "still hold the spacers".
  • A fix the merge exposed: $selectionShownText read an element point at a range's end as offset 0, so after the merge a range ending right before an optional break copied the wrong text. $offsetInText and $liesBeforeElementPoint moved to copyLimit.utils.ts and are shared. This is the same class of bug as CB-19.

Data-fidelity fixes beyond the clipboard

These came out of the U+2028 work above. They are not clipboard changes: they fix how characters are typed in and loaded, in every view.

  • U+2028 kept through Standard view's marker settle (f46ffc9f, paranext-core catalog T-19). The tier-2 rebuild spelled its display-only line break as U+2028 in the fragment it re-reads, and removed every U+2028 before tokenizing, so a real one in a rebuilt paragraph was lost (in Standard view, typing aa<U+2028>bb on a chapter line saved aabb, and a typed marker settled a paragraph holding one without it). The display-only break (DISPLAY_LINE_BREAK in tier2Rebuild.utils.ts) is now U+000B. No text can hold it: ParatextData reads it as a space, and the editor takes an entered one in as a space (next item). It is still whitespace to every fragment reader that counts positions, and it never leaves the fragment. TJ ruled this on 2026-11-11. The former it.fails in chapterLineEditing.test.tsx now passes, and a new test checks that a stored U+2028 and U+2029 survive a typed marker's settle in Standard and Unformatted view, with no U+000B reaching the text nodes, the page or the saved document.
  • Control characters entered go in as spaces, in every view and on every route (4e389fe7, paranext-core catalog EV-6). That covers typing, CONTROLLED_TEXT_INSERTION_COMMAND and beforeinput insertText, plain-text, html and node-copy paste, and drop. Every C0 control character but tab, line feed and carriage return (U+0000–U+001F) becomes a space, as ParatextData's reader treats them. InvisibleCharactersPlugin, which already dropped an entered U+FEFF, now does both through normalizeEnteredText, and Standard view's typing and paste use the same rule. Standard view already did this, except for U+0000; the other views took these characters in as they were. A space rather than nothing keeps the text on either side apart. Stored and loaded text is never changed, following the U+FEFF precedent. This also keeps the editor's own uses collision-free: U+001A is Standard view's no-break-space stand-in, and U+000B is the settle's display-only break. 51 new tests failed before the fix.
  • A stored U+2028, U+2029 or U+0085 is no longer lost on load (0c42ac47, pre-existing on main; paranext-core catalog F-9, T-18 updated). usxStringToUsj (packages/utilities) read USX with xmldom's default XML 1.1 line-end rules, which turn these characters into \n in text and a space in attribute values; a text node holding only one was dropped as formatting whitespace. The next save then wrote a space. It now uses XML 1.0 rules (only \r\n and \r become \n), matching how ParatextData writes USX, and Paratext 9 keeps these characters. This fixes the editor, the finder and every other usxStringToUsj consumer. Entry in packages/utilities/CHANGELOG.md.
  • Docs: docs/standard-view-invariants.md documents the reserved U+000B. It also corrects an old claim that ParatextData keeps U+000B and U+000C: its reader turns every C0 control character but tab, LF and CR into a space. The platform CHANGELOG covers the first two items.

Second /code-review high round

  • fff9ee32: a paste the hidden-marker views insert themselves (plain text filtered for \ or //, joined into one line under protected structure, or a Markers view copy without its marker tokens) over a range across a span's edge no longer moves the text after the range into the span. UnstorableTextPlugin dispatches the new BEFORE_PLAIN_TEXT_INSERTION_COMMAND before inserting plain text itself, and InlineRangeEditPlugin removes the range there, as it already did before Lexical's own plain-text paste. 10 new pasteOverInlineRange tests failed before the fix.
  • 72884b27: doc fixes. LineBreakPlugin's clipboard rule applies in a view without spacing too, and a refused "move" can be for a \ as well as a /.
  • 2be13cda: dist.
  • 944437ec: htmlWithNodeCopy builds its carrier span with the DOM (setAttribute, then outerHTML) instead of escaping the node copy by hand; the value read back is the same, and the round-trip tests (quotes, &, <, an attribute-closing attack string) still pass and fail when the escaping is removed. CodeQL did not recognize the hand-written escaping. 7259f000: dist.
  • Discarded after checking: a word-delete probe before a cheap / check, a large paste parsed more than once, the denied clipboard-read alert replaced by a log, and a line break over a range on a chapter line (it runs as Enter, which does the same).

Why review this

Part of the Standard-view epic's data-fidelity work: catalog rows CB-1 and CB-5 (PT-4911) and CB-3 / EV-1 (PT-4921), found by the PT-4197 audit. In Simple view, copying across a paragraph break couldn't be pasted, a cut text was lost until Undo, and any paste with \ or / (a URL, "and/or") was silently refused. Ctrl+V in Simple and formatted views lost attributes, footnote details and figures from the pasted copy. The fixes beyond the clipboard (catalog T-19, EV-6, F-9) stop U+2028, U+2029 and U+0085 from being lost on load or in a marker settle, and keep entered control characters out of the text.

Changes

PT-4911:

  • hiddenViewCopy.utils.ts + the copy/cut writer for hidden-marker views (plain text without gutter glyphs or unknown-construct text; html without glyph spans; the editor's own flavor, which PT-4921 replaces with the tagged node copy). After the merge of main, the writer is main's HiddenMarkersViewCopyPlugin (see "Merged with PT-4126"), and its tests are in hiddenViewCopy.test.tsx. CopyLimitPlugin's over-limit copy combines main's $overLimitText and fit with the same hidden-text predicate.
  • UnstorableTextPlugin + unstorableText.utils.ts + unstorable-text.model.ts (shared-react): the filter, its paste/drop/typing routes, the delete check and the notice. Mounted in every view, acting only in hidden-marker ones, where it replaces CommandMenuPlugin (the Markers view keeps that). Word and line deletes are measured by $probeDeleteExtent (hiddenContentDelete.utils.ts), shared with HiddenContentDeleteGuardPlugin. StructureKeyboardPlugin takes $filterPastedNodes for its locked html path (hidden views only). HiddenContentDeleteGuardPlugin asks before a paste only when it will insert something, and not before a cut UnstorableTextPlugin refuses.
  • shared-react now depends on @lexical/clipboard (same version as packages/platform).
  • Clipboard corpus (clipboardViewsCorpus.test.tsx): paragraph-structure rows run off, guarded and protected; rows for "and/or", 1/2, a URL, ranges holding a figure; hidden-view copies checked for no gutter text, no caption, and the collapsed note's text. The 44 PT-4911 pins are gone.

PT-4921:

  • clipboard.utils.ts (shared-react): node-copy readers and writers, the html attribute (data-platform-editor-node-copy) and the marker-showing mark (data-platform-editor-marker-text), removeMarkerTokens/withoutMarkerTokens, registerPasteAsClipboardRead, registerTabsAsSpaces; LEXICAL_CLIPBOARD_TYPE exported there.
  • bookAndChapterLines.utils.ts + BookAndChapterLinesPlugin (Select All keeps the book and chapter lines), LineBreakPlugin + lineBreakText.utils.ts (Shift+Enter and inserted line breaks), dropPoint.utils.ts (where a drop lands, shared with ParaMarkerSelectionPlugin), ImpliedParaNode.insertNewAfter (shared).
  • New optional props: ClipboardPlugin.pastesAsClipboardRead/logger, ContextMenuPlugin.logger, StructureKeyboardPlugin.nodeCopyNamespace/removesMarkerTokens/logger. ENGINE_MARKER_NAME_BYTES moved to shared and is re-exported under its old name.
  • Tests: hiddenViewSelectAllClipboard, hiddenViewSelectAllEdit, bookAndChapterLines.utils (47), nativePasteFlavor, standardCopyElementPoints, pastedOptionalBreak (91), lineBreak, hiddenViewDragMove; the corpus models Ctrl+V through the html attribute and turns its PT-4921 pins into passing rows.

Beyond the clipboard:

  • tier2Rebuild.utils.ts (DISPLAY_LINE_BREAK is U+000B), InvisibleCharactersPlugin.tsx and whitespaceDisplay.utils.ts / whitespaceDisplay.plugin.utils.ts (normalizeEnteredText), packages/utilities usx-to-usj.ts (XML 1.0 line ends), with tests in invisibleCharacters.test.tsx, whitespaceDisplay.utils.test.ts, lineBreak.test.tsx, chapterLineEditing.test.tsx and usx-to-usj.test.ts.

Both:

  • docs/clipboard-semantics.md (S5 rewritten; the html route; Select All; drag-move); docs/standard-view-invariants.md; the platform and utilities CHANGELOGs; committed dist (platform and utilities) and etc/platform-editor.api.md rebuilt.
  • Public API (@eten-tech-foundation/platform-editor): EditorOptions.onUnstorableTextRemoved and three types, UnstorableCharacter ("\\" | "//"), UnstorableTextRemoved and UnstorableTextRemovedHandler. UnstorableTextRemoved.source is "paste" | "drop" | "typing" | "deletion" | "move", and removed is a readonly array. The handler is called synchronously from the editor command or update that made the edit. Everything else changes behavior only, with the same signatures; the fixes beyond the clipboard change no public type.

Earlier testing rounds

At 2be13cda (after the second /code-review high round):

  • nx typecheck and lint: clean.
  • packages/platform: 7921 passed, 99 expected-fail. shared-react: 2717 passed.
  • pasteOverInlineRange with SE_TEST_LIBRARY_VIEWS=1: 96 passed.
  • node scripts/verify-committed-dist.mjs: current.
  • paranext-core e2e (under xvfb) with this editor staged: hidden-view-drag-move, hidden-view-clipboard and in-editor-paste passed.
  • At 7259f000 (the DOM-built carrier): shared-react 2717, packages/platform 7921 passed (99 expected-fail); e2e hidden-view-clipboard and in-editor-paste passed, clipboard-usfm-round-trip passed on retry (first attempt: Electron "execution context was destroyed").

Earlier, at 36789846 (after the fixes beyond the clipboard):

  • nx typecheck and lint: clean.
  • packages/platform: 7911 passed, 99 expected-fail. shared-react: 2717 passed. utilities: 112 passed.
  • With SE_TEST_LIBRARY_VIEWS=1, the touched and related suites: 4172 passed.
  • node scripts/verify-committed-dist.mjs: current.
  • paranext-core e2e (under xvfb) with this editor staged: the four no-unasked-edits-invisible-characters-* specs (which now seed, type and paste U+2028, U+2029 and U+0085, each saved as itself), no-unasked-edits-power-standard and no-unasked-edits-simple-paragraph-structure passed. Against 4c144a28, earlier in the same round, hidden-view-clipboard and in-editor-paste passed too.

Earlier rounds, kept for the coverage they show:

  • At f191b099, after the merge of main with PT-4126: nx typecheck and lint 0 errors; shared-react 2717 passed; packages/platform 7874 passed, 100 expected-fail; with SE_TEST_LIBRARY_VIEWS=1, the clipboard corpus plus the copy, gate, invisibleCharacters and copyLimit suites 2327 passed, 127 expected-fail; e2e hidden-view-drag-move, hidden-view-clipboard, in-editor-paste and clipboard-usfm-round-trip all passed on the first try.
  • At 93f6f465, before the merge (after the /code-review high rounds, the drag-move fixes and the U+2028 change): shared-react 2717 passed; packages/platform 7849 passed, 100 expected-fail; lineBreak and chapterLineEditing with SE_TEST_LIBRARY_VIEWS=1 703 passed, 1 expected-fail (the U+2028 settle gap, since fixed); the same four e2e specs passed (hidden-view-drag-move on retry after the known cold-start timeout; in-editor-paste flaked once at the Edit flyout's Paste step, then passed on retry and in two more clean runs).
  • Clipboard corpus: 1533 passed, 96 expected-fail of 1629 rows (80 under CB-14, 2 under CB-16, 14 under PT-4388); with library views, both typing oracles green (formatted-view oracle 23 passed).
  • unstorableText.utils.test.ts ($deletionJoinsSlashes, $transposeJoinsSlashes rows), unstorableTextDeletion (101), unstorableTextTyping with transpose rows (39), pastedOptionalBreak (91), bookAndChapterLines.utils (47), lineBreak (149 when added, 111 red before the fix). shared: 951 passed.
  • Most fixes were written red first and mutation-checked: each guard of the transpose refusal, the optional-break removal, the chapter-line utilities (12 mutations), the line-break routes (11 mutations).
  • Before the merge of main, both branches were rebased onto main after the Scribe removal (scripture-editors#2) and PT-4982 (#2973); no Scribe-specific content is left. The PT-4911 work was also re-run against main before that rebase.
  • paranext-core e2e at earlier tips: all nine no-unasked-edits-* specs, no-unasked-edits-footnotes-pane-simple and simple-view-popups passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c


This change is Reviewable

tjcouch-sil and others added 21 commits November 5, 2026 15:41
…constructs

A copy or cut in a hidden-marker view (formatted, paragraph-structure,
block-verse, Preview) now writes the text the view shows:

- HiddenViewCopyPlugin (COPY/CUT at NORMAL, mounted right after
  CopyLimitPlugin for markerMode "hidden") writes text/plain without a
  paragraph's gutter marker glyph or the text of an UnknownNode construct
  (a figure's caption), text/html from Lexical's export, and the internal
  lexical flavor. Collapsed note text stays in the plain text, which a
  plain-text paste needs.
- ImmutableTypedTextNode exports no element for a gutter glyph, so the
  HTML flavor leaves it out. Lexical consults excludeFromCopy only on
  element nodes, so a decorator has to leave itself out through exportDOM.
- CopyLimitPlugin's over-limit plain writer and its fit leave the same
  nodes out in hidden views.
- The clipboard corpus's gutter-glyph known-failure rule is removed: its
  43 rows now pass (the 44th still fails, under the html-flavor rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add UnstorableTextPlugin, which removes every `\` and every `//` (pairing
slashes from the left, and counting a pasted `/` that touches a document
`/`) from each paste and drop route: node copies and html through
SELECTION_INSERT_CLIPBOARD_NODES_COMMAND, plain text through PASTE_COMMAND
and insertFromDrop. It reports each removal once through onRemoved and
logs it at info. Under protected structure, a plain-text paste or drop
joins its lines with a space instead of splitting the paragraph.

StructureKeyboardPlugin takes a $filterPastedNodes prop for the html it
inserts itself under protected structure. HiddenContentDeleteGuardPlugin
treats a plain paste as inserting nothing only when the filter leaves it
empty, and asks about html and node copies. EditorOptions gains
onUnstorableTextRemoved (wired into the Editor separately).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The protected-structure sanitizer lists a paragraph's children without
detaching them, so a text node the unstorable-text filter emptied was
detached but stayed listed and was inserted with its original text. The
filter now drops an emptied node from the list whenever it is listed, and
both the plugin and the sanitizer use one `$insertsNothing` predicate, so
a paste that leaves only an emptied character span no longer deletes the
selection it was pasted over.

Plain-text insertion goes through `$insertDataTransferForRichText` again
(shared-react now depends on @lexical/clipboard, already in the lockfile).
Under protected structure, each run of line breaks becomes one space and
a trailing line break is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hidden-marker views mount UnstorableTextPlugin in place of
CommandMenuPlugin, so a paste or drop holding `\` or `//` goes in without
them instead of being refused, and EditorOptions.onUnstorableTextRemoved
is told once per paste or drop. Under protected structure the html
sanitizer filters the same way through StructureKeyboardPlugin's
$filterPastedNodes, given only in hidden-marker views so Standard view's
paste keeps its markers. The Markers view keeps CommandMenuPlugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hidden-marker views now take a paste holding `\` or `//` without those,
so the corpus oracle filters the pasted text (filterFragment: strings only,
never a stray-marker node) instead of expecting the paste refused, and
checks that the host is told once whenever the filter removed something.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t stray \ and //

The paste filter strips `\` and `//` from a hidden-marker view's pasted
text, so a copy that leaked a gutter glyph or marker into text/plain would
pass the paste checks with a matching notice. Each hidden-view copy row
now checks that the text/plain holds exactly as many `\` and `/` as the
copied content does, a stray marker counted as its own text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hidden-marker views refuse a typed `\`, and a typed `/` that would touch
a `/` beside it (judged outside any selection the keystroke replaces); a
lone `/` goes in. Each refused keystroke is reported once through
`onUnstorableTextRemoved` with source "typing".

The refusal happens at KEY_DOWN (shortcut chords such as Ctrl+/ pass,
AltGr does not), with backstops for typed text that arrives without a
keydown: a CRITICAL CONTROLLED_TEXT_INSERTION_COMMAND handler (string
payloads and insertReplacementText events) and a capture-phase root
beforeinput listener for native insertText. Multi-character typed text
(autocorrect, replacements) loses only `\` and `//` and goes in, with
one notice. IME composition is not filtered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…guarded

Simple mode's paragraph-structure view now runs every clipboard corpus row
under each structure protection (off, guarded, protected — Simple mode's
default); the other views run off.

- Oracle: protected html pastes (all flavors included, since the sanitizer
  takes text/html first) unwrap paragraphs joined by one space and drop
  verse/chapter markers (protectedFragment); protected plain multi-line
  pastes join lines; a protected cut across a paragraph or over a verse is
  refused (document unchanged, nothing on the clipboard). The locked html
  route carries a stray marker as text, so its `\` is filtered with one
  notice (TJ ruling: accepted behavior).
- Cut-paste-back expectations are built from the filtered fragment, so the
  document and the notice count agree; the filter models the characters
  beside the insertion point.
- New rows: a range holding 2SA-1's figures, "cut, paste html and plain",
  and plain pastes of and/or, 1/2 and https://example.org. Every
  hidden-view copy checks text/plain for no gutter glyph, no figure
  caption, and the collapsed notes' text.
- PT-4921 rule covers the html route under protection and a caption-less
  figure left out of the editor's own clipboard format (TJ ruling);
  PT-4388 rule narrowed to the rows that fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ally

The plain-text paste rows now state what a hidden-marker view inserts and
what its notice names as removed (https://example.org -> https:example.org
with one "//" notice; and/or and 1/2 unchanged, no notice; the USFM-looking
texts lose their "\"), instead of deriving both from the filter under test.
A precondition asserts no "/" sits beside the paste point, which the
literals assume.

Also widens the PT-4921 reason to name an optbreak alongside the
caption-less figure, and fixes two comment nits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rewrite the clipboard semantics doc's S5 for the hidden-marker views:
the copy leaves out gutter markers and hidden constructs but keeps a
collapsed note's text; paste, drop and typing remove `\` and `//`
(with adjacency to a document `/`) and tell the host once through
EditorOptions.onUnstorableTextRemoved; locked plain multi-line paste
joins lines; the Markers view still refuses `\` and `/`. Close the
CommandMenuPlugin paste-gate deferral, restate the collapsed-note copy
as intended, point the remaining html-route and caption-less-figure
gaps at PT-4921, and map the new tests.

Record the user-visible changes and the new public API in the platform
package's changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dden-marker views

Outside structure protection, html pasted or dropped into a hidden-marker
view brought a marker glyph span back as text (ImmutableTypedTextNode's
html import never claims one), and the unstorable-text filter then left
it as text the user never wrote (`\q1 ` -> `q1 `) with a notice. A
drag-move across a paragraph break in the paragraph-structure view hits
this, since Lexical's DRAGSTART writes no data and the browser serializes
the next paragraph's gutter glyph; so do html pastes of a Markers-view
copy or an older copy.

UnstorableTextPlugin now removes every `[data-text-type="marker"]`
element from the html route of a paste or drop before Lexical parses it,
through removeMarkerGlyphElements, which StructureKeyboardPlugin's
protected sanitizer now shares. The node-copy and plain-text routes are
untouched; html left with nothing to insert changes nothing. Standard
view's handling of the same html is pinned unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CopyLimitPlugin: the over-limit writer runs ahead of the hidden-marker
  views' HiddenViewCopyPlugin (NORMAL, registered after it), not only of
  HIGH handlers.
- MarkerEditPlugin: the CUT arm's list of removers names
  HiddenViewCopyPlugin.
- clipboard-semantics S5 and CHANGELOG: under protection a tab becomes a
  space only in a multi-line plain-text paste.
- UnstorableTextPlugin joinLines: a trailing line break is dropped because
  under protection it would only split the paragraph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e rule

HiddenContentDeleteGuardPlugin skips asking about a paste that will
insert nothing. It judged that by UnstorableTextPlugin's rule (only `\`
and `//` are removed), but the guard also runs in the Markers view, where
CommandMenuPlugin still refuses any paste whose plain text holds a `\` or
`/`. In a Markers view made editable, a paste of "and/or" over hidden
content would be asked about, and a "delete" answer would remove the
hidden content while nothing was pasted. (The guard does nothing in a
read-only editor, so the shipped read-only Markers view is not affected.)

A new refusesCommandCharacters prop (set by Editor for markerMode
"visible") restores the command-character answer there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or precedence

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A comment or annotation mark saves as nothing, so text either side of its
edge ends up adjacent in the file. The filter read only direct siblings, so
a / typed or pasted right beside a / across a mark's edge saved as //.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nput alone

insertFromYank (macOS Ctrl+Y) reaches CONTROLLED_TEXT_INSERTION_COMMAND as a
replacement does, but the filter only read the replacement's event, so a
yanked \ or // went in unfiltered.

The beforeinput check called editor.read on every typed character, which
commits pending updates; it now tests for a \ or / first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
$deletionJoinsSlashes reads what the file stores either side of what a
delete removes (a range, one character from a caret, or a paragraph
join) and says whether a / would end up against a / it was not already
touching. It reads through annotation marks, gutter glyphs and cursor
placeholders, through a character span the delete empties, and through
two pieces of one span a join makes one again; a span edge that stays,
a verse, a note or a hidden construct keeps the two apart, and a //
already in the text is not counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5J1wq8kko2sACPaBFZ
…idden-marker views

Deleting b from a/b/c saved a//c, which USFM reads as an optional line
break, so the next load showed ac. UnstorableTextPlugin now refuses any
delete whose result would put a / against a / it was not already
touching, and tells the host once through onUnstorableTextRemoved with
the new source "deletion".

It judges every delete route: Backspace and Delete with their word and
line chords at the keystroke (CRITICAL, ahead of guarded structure's
arming press, the hidden-content question and the paragraph-marker
merge), the same deletes without a keystroke (DELETE_*_COMMAND), a
selected paragraph marker, a cut and a drag-move's removal of its source
(HIGH, after the guards that narrow them). Under protected structure a
join is left to that structure's own refusal. Word and line extents are
measured on a copy of the selection, as HiddenContentDeleteGuardPlugin
measures them; its deleteChordOf is exported for the keystroke mapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5J1wq8kko2sACPaBFZ
An optional line break is a childless unknown node in the views that
hide markers, so the filter and the delete check read it as a wall. The
file stores it as a bare `//`, though: deleting the space from `a/ `,
break, `/b` saved `a////b`, which reads back as two breaks, and the view
lost both `/`. Typing or pasting a `/` against a break had the same
effect when a `/` stood on its other side.

$characterBeside now reads a break beside a point as `/`, and
$readPieces gives a break's atom the text `//`, so $joinsSlashes sees a
`/` brought against it. A delete still takes the break whole.

Also in unstorableText.utils.ts: say where $insertsNothing and
$textAroundSelection may be called, return undefined rather than null
from $neighbourThroughMarks, build the grapheme segmenter once, and name
the deletion's start, end and caret a Boundary rather than a Marker,
which in this file otherwise means a USFM marker. New
$deletionJoinsSlashes rows: a verse or a note between two `/` (staying
and going), a range starting or ending inside a note, and a delete
inside an expanded note's text.

Docs: the CHANGELOG's protected-paste entry loses a stray line break,
and Deferred item 9 states its open half instead of narrating it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5J1wq8kko2sACPaBFZ
…s filtered

Pins two rules UnstorableTextPlugin documents but no test held: a
keydown an input method is composing is neither refused nor reported,
and a drop carrying only `text/uri-list` (as Lexical reads it when
there is no `text/plain`) loses its `//` with one notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5J1wq8kko2sACPaBFZ
…where markers hide

It was mounted only when markerMode was "hidden", so an editor that
switched into a hidden-marker view registered its CRITICAL KEY_DOWN and
delete handlers behind ParaMarkerSelectionPlugin's: deleting a selected
paragraph marker joined `a/` and `/c` into `a//c`. The plugin now takes
isEnabled, held in a ref and asked by every handler and its beforeinput
listener, and Editor mounts it always with isEnabled set for hidden
markers, so its handlers keep their place across a view switch.

mountInView gains rerenderWithView for the test.

(Carried down from the PT-4921 branch, where it was 717333f, so that
this branch holds it on its own.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5J1wq8kko2sACPaBFZ
tjcouch-sil and others added 2 commits November 17, 2026 12:39
# Conflicts:
#	packages/platform/dist/index.js
#	packages/platform/dist/index.js.map
#	packages/platform/src/editor/Editor.tsx

@katherinejensen00 katherinejensen00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just two things blocking the merge, one new and one left over from item 1. I included some optional smaller things at the end, but you can merge without them so it is totally up to you

  1. High: the node-copy check is by type only, so a crafted copy can still break the editor or save corrupt USFM
    nodeCopy.utils.ts:378 (typeOutsideHiddenView) · Confirmed (probe)

The allowlist stops root and marker entries. But it doesn't check where an allowed type sits, or what its fields hold, so any web page's copied html can still:

  • Leave the editor unusable. An immutable-note-caller outside a note: the paste goes through, then every later update throws "noteNode.getIsCollapsed is not a function" until a reload. Probe: [para[immutable-note-caller]] in the html attribute, under off and protected.
  • Save corrupt USFM. A char with marker "a b" is saved to USJ as is, and createDOM's classList.add throws. A verse number of "2\v 3" is saved and would write \v 2\v 3. An unknown entry with tag "chapter" or "book" saves a chapter or book node inside a paragraph, where nothing shows.
  • Show what isn't saved, or the reverse. A text entry's style is applied to the live span, but the style isn't saved: display:none hides text that is still saved, and background-image:url(https://…) makes a request the web view's CSP allows. immutable-typed-text holding "\v 5" shows but isn't saved.
  • Overflow the stack. Depth isn't capped: about 1000 levels pastes, and then getUsj throws RangeError, so the document can't be saved until the paste is undone. Deeper nesting throws out of the paste.

Suggested fix, all checked before any node is made, in typeOutsideHiddenView:

  • allowed parents per type (a note caller only directly in a note);
  • marker validated with the node class's isValidMarker;
  • verse numbers run through the \ and // filter;
  • text style and format rejected;
  • unknown tags limited to the content-less constructs a hidden view holds (figure, optbreak, …);
  • a depth cap of about 32.

A crafted-payload test for each would pin it. Another option is to round-trip the made nodes through USJ and the normal loader, which validates all of this already.

  1. Medium: under protected structure, a paste can now delete hidden content without asking
    hiddenViewInsertion.utils.ts:160 ($predictInsertion) · Confirmed (probe + trace)

This comes from the item 4 fix. Under protected, StructureKeyboardPlugin's sanitizer puts a " " between top-level pasted paragraphs (structureKeyboard.utils.ts:839), but $predictInsertion judges the nodes before sanitizing. So when every pasted paragraph holds only \ or //, the guard decides the paste inserts nothing and doesn't ask. The sanitizer then inserts the lone space over the selection.

Probe: formatted view, protected, a selection across a \fig, html "

</p>

</p>". onHiddenContentDelete is never called, and the figure is replaced by a space. Off and guarded leave the document unchanged.

Simple mode uses protected whenever structure is locked, so this ships, though the trigger is unusual and Undo restores the figure. Suggested fix: predict on the sanitized nodes when isProtected, plus a two-paragraph test.

Smaller things, optional:

  • Item 25: the selection-start change has no backward-selection test, so going back to anchor fails nothing.
  • docs/clipboard-semantics.md:185 still names $unstorableInDrop, which is now $predictInsertion. Line 133 puts htmlWithNodeCopy in clipboard.utils.ts; it's in nodeCopy.utils.ts now.
  • nodeCopy.utils.ts:99: "platformEditor" isn't Lexical's default namespace; Editor.tsx sets it. The {@link MARKER_TEXT_HTML_ATTRIBUTE} at :101 won't resolve.
  • htmlClipboardRoute.test.tsx:278 narrates the old bug ("a root entry crashed the editor"); say what the check prevents instead.
  • Withdrawn-marker leftovers from item 3: \cp B pastes "B" as its own line, and \ca 3\ca* pastes " 3" as text.

On the CodeQL alerts: #23–#27 are false positives like the earlier ones. Each is an inert DOMParser parse whose result only becomes Lexical nodes or an html string for another inert parse, and nothing reaches the live DOM. The editor's importDOM converters copy no href, src, on* or style. Dismissing them is right. #27 would go quiet if htmlWithNodeCopy built the carrier inside a DOMParser document. Please don't quiet it with carrier.innerHTML = html, which would be a real sink.

@katherinejensen00

Copy link
Copy Markdown
Contributor

Found while testing PT-4820 (Model Text panel with gutter markers): in a hidden-marker view, double-click a paragraph's gutter glyph, say \q1, and press Ctrl+C. The clipboard gets "\q1 ". The double-click selects only the glyph's "". HiddenMarkersViewCopyPlugin filters glyph text out, gets "", and returns undefined (if (plain === "") return undefined;), so Lexical's own copy runs and writes the glyph text. Could the writer claim a copy whose filtered text is empty instead of falling through? paranext-core has a test.fails tripwire for this in model-text-gutter.real-editor.test.tsx that will flip once it's handled.

tjcouch-sil and others added 8 commits November 18, 2026 12:35
…the whole note

A range with both ends inside one collapsed note (its content hidden, only
the caller showing) copied part of the note's shell, e.g.
`\f - \cat things\cat*`, while the cut removed the whole note through the
damaged-glyph-pair rule, so a cut and paste back lost everything after the
category. Found in hand QA of PT-4921 (Power mode, Standard view).

`$takeCollapsedNotesIntoRange` already widened a range with one end inside
a collapsed note on selection change, and deliberately leaves a range
wholly inside one note alone for the marker menu. Copy and cut now widen
that range too, at CRITICAL priority ahead of every copy writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
… dropped

With structure locked, `StructureKeyboardPlugin` took every drop and
inserted it at the editor's selection, which at a drop is still what was
dragged. A drag-move within the editor so put the text back where it was
(nothing moved), and a copy-drop or a drop from another program replaced
the selection instead of landing at the drop point. Found in hand QA.

It now inserts at the drop point (`$selectionAtDropPoint`), refuses a drop
whose drop point would replace structure before anything is removed, and
leaves a move within the editor to the move the unlocked views make
(`UnstorableTextPlugin`, catalog CB-23/CB-24). A drag that holds structure
is still refused at its dragstart. `isCopyDrop` moves to
`dropPoint.utils.ts` for both plugins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
The paste check took any entry of a type the view's document holds, so a
crafted node copy in any web page's html could still put a note caller
outside its note (every later update threw), save a char marker "a b" or
a verse number "2\v 3" as corrupt USFM, show styled or typed text the
save drops, or nest deep enough to overflow getUsj. nodeCopyProblem now
checks each entry's placement, marker, number, attribute and text fields
and caps nesting at 32 before any node is made; such a copy goes in by
its html or plain text. immutable-typed-text and typed-mark leave the
allowlist (a copy never holds them). The html carrier is built in an
inert document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…s sanitized

$predictInsertion judged a paste's nodes before StructureKeyboardPlugin's
sanitizer puts a space between top-level paragraphs, so a paste of
paragraphs holding only `\` over a figure was judged to insert nothing:
HiddenContentDeleteGuardPlugin did not ask, and the space replaced the
figure. Under protected structure the prediction now sanitizes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
In a hidden-marker view, double-clicking a paragraph's gutter glyph and
copying fell through to Lexical's copy, which wrote the glyph's text
(`\q1 `). The view's copy writer now claims a copy holding nothing but
gutter glyphs and writes nothing; a cut is still left to the
paragraph-marker rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…p out of a hidden-view paste

The chapter token went, but `\ca 4\ca*` pasted `4` as text and
`\cp B` pasted `B` as a line of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…r this round

A backward selection is judged where it starts (its focus). The clipboard
semantics doc names $predictInsertion and nodeCopy.utils.ts where it
named a removed helper and the wrong module, and describes the new
node-copy checks, the protected-structure prediction and the \ca/\cp
strip; CHANGELOG entries for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
@tjcouch-sil

tjcouch-sil commented Nov 18, 2026 •

Copy link
Copy Markdown
Member Author
  1. Crafted node copy (High): fixed in 3f126398. $nodesFromNodeCopy now runs nodeCopyProblem before any node is made. It is a per-type table recording where each type may sit (a note caller only directly in its note, a row only in a table, a cell only in a row, paragraphs, book lines, chapters and tables only at the top level or in a sidebar or periph) and which fields to check:

    • Markers must be USFM names and not id, c or v.
    • Verse and chapter numbers may not hold whitespace, \, /, |, * or ".
    • Text, tab and unmatched entries may not carry a style or format. Their textType may only be a note's layout spacer, and their text may not start with the attribute prefix.
    • A note caller must match its note's caller.
    • An unmatched marker must show its own glyph text.
    • An unknown node's tag must be figure, optbreak, ref, periph or sidebar.
    • Attribute names must be words.
    • Nesting is capped at 32 levels, and the check itself stops there.

    immutable-typed-text and typed-mark left the allowlist, since a copy never holds them. Each of your probes is a crafted-payload test in htmlClipboardRoute.test.tsx under off, guarded and protected: the paste falls back to the html, the book and chapter lines are unchanged, and nothing crafted is saved. A positive control checks that a legitimate copy (a \zcustom char, verse 2-3a, a note with its caller, a figure) still goes in by its node copy.

    I didn't use the node classes' isValidMarker. The clipboard corpus has real copies with markers outside those lists (\cp, \t-s and \zunk paragraphs, a paragraph with no marker, a \cat char inside a note, \z chars), and 63 corpus tests failed with it. So the check is the marker's shape. One gap remains: a crafted char with a known paragraph marker such as p still pastes. The round trip through USJ wasn't available here, because the adaptors live in platform, which shared-react can't import.

  2. Protected paste that skips the question (Medium): fixed in d18947d6. Under protected structure, $predictInsertion now judges the nodes as $sanitizeNodesForProtectedStructure leaves them, so the space between paragraphs counts and the guard asks. Tests in HiddenContentDeleteGuardPlugin.gate.test.tsx cover your two-paragraph probe and a one-paragraph control that still doesn't ask.

Smaller items:

  • Item 25 backward selection: cd5331ea adds hiddenViewInsertion.utils.test.ts, which covers forward and backward selections each way across a paragraph/note edge. Going back to anchor fails both backward cases.
  • docs/clipboard-semantics.md: now names $predictInsertion, and htmlWithNodeCopy is in nodeCopy.utils.ts (cd5331ea).
  • nodeCopy.utils.ts:99: now says Editor sets the namespace. MARKER_TEXT_HTML_ATTRIBUTE is named with its module rather than linked (3f126398).
  • htmlClipboardRoute.test.tsx:278: now says what the check prevents.
  • Withdrawn-marker leftovers: 12fa0a4c. A marker view's copy pasted into a hidden view drops a chapter's \ca …\ca* and \cp … along with its \c.
  • CodeQL: PT-4730: Limit how much text can be copied from the editor #23–PT-4703: Ask before an edit removes content the view does not show #27 are dismissed as false positives. htmlWithNodeCopy now builds its carrier in an inert DOMParser document, with no innerHTML, so PT-4703: Ask before an edit removes content the view does not show #27 may go quiet. If it doesn't, the alert is on the concatenation of html, which is the inert export.

@tjcouch-sil

Copy link
Copy Markdown
Member Author

Fixed in ce419fb6. A copy holding nothing but gutter marker glyphs is now claimed and writes nothing, so the clipboard keeps what it had, as a copy of nothing does. A selection with no text that does hold something (a verse number) still goes to Lexical's copy, as before. A cut of a glyph-only selection is left alone, since what it removes is up to the paragraph-marker rules. The test.fails tripwire in paranext-core's model-text-gutter.real-editor.test.tsx (on the PT-4820 branch) should flip once platform-yalc moves past this.

tjcouch-sil and others added 3 commits November 18, 2026 17:17
…o its paragraph spaces inserts nothing

The sanitizer puts a space between pasted paragraphs. Once a filter took
out all of their text (paragraphs holding only `\`), that space was still
inserted over the selection, so the paste replaced a figure with a
space, where an unprotected paste of the same html changes nothing. The
spaces it adds are now known ($insertsOnlyParagraphSeparators), and a
paste left with nothing else inserts nothing; $predictInsertion judges
it the same way, so nothing is asked either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…o a hidden view

This reverts 12fa0a4. \ca and \cp are markers of their own in USFM and
stand alone in every data format, so a paste that withdraws the chapter
line keeps them, as it keeps any other marker's content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
@tjcouch-sil

Copy link
Copy Markdown
Member Author

Two follow-ups to my reply above:

  • Item 2 (protected paste): changed further in acefb5a9. The protected paste no longer asks and then inserts the leftover separator space. If every pasted paragraph is filtered out, it now inserts nothing, the same as off and guarded, so there is nothing to ask about. A paste that keeps any text still asks. The test covers all three structure modes.
  • \ca / \cp: reverted in 7bdd023b. TJ's view is that \ca and \cp are separate markers in USFM and stand alone fine in every data format, so withdrawing the chapter line doesn't need to take them with it. Their content stays, as it does for any other marker whose token is stripped: \ca 4\ca* pastes 4, and \cp B pastes a B line. Could you reconsider? It's ultimately a UX call, but it's small enough not to hold this PR for.

JSON.stringify of the crafted 1000-level payload overflowed the stack of
CI's Windows test worker before the paste ran. The test now builds that
payload's JSON by repetition; the code under test parses and refuses it
under the same reduced stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c

@katherinejensen00 katherinejensen00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes, TJ! Three things block the merge. Optional items are at the end.

  1. High: crafted node copies still get past the check
    nodeCopy.utils.ts:652 · Confirmed (probe + code)
  • Attributes overwrite real fields. editor-usj.adaptor.ts spreads ...unknownAttributes after type, marker and number (:251, :267), so an attribute named marker, number or type replaces that field on save. Probes saved a char marked "a b", verse "2\v 3", and a chapter or book inside a paragraph, the same results the round-3 probes got.
  • Two fields still break the editor. If a milestone's attributeOrder is a string, every later update throws "attributeOrder.forEach is not a function". Deeply nested $ node state makes every later update overflow the stack, because the depth cap counts only children.
  • The * in UNMATCHED_MARKER (:415) is optional, so an unmatched "v" followed by " 5 X" saves as \v 5 X, a new verse.

Fix: reject structural keys as attribute names (or spread unknownAttributes first), require attributeOrder to be an array of names, allow only known $ keys, and make the * required. No test reaches fieldsProblem: with it forced to accept everything, all 1670 clipboard tests pass. One crafted case per check would pin them.

  1. Medium: Ctrl+C inside a collapsed note sets up its deletion
    TrailingNoteCaretGuardPlugin.tsx:342 · Confirmed (probe + code)

The new COPY handler widens the live selection to the whole note and leaves it there. Select inside a collapsed footnote, press Ctrl+C, type X: the footnote is replaced without asking, in Standard view too. Restore the selection after the copy (keep the widening for cut), and add a test that types after copying.

  1. Medium: the glyph-copy fix only works when a verse follows the glyph
    HiddenMarkersViewCopyPlugin.tsx:73 · Confirmed (probe)

On a line that starts with text (\q2 continuation, \m), getNodes() returns [glyph, text], so $selectsOnlyGutterMarkers fails and "\q2 " is copied again. The test's \q1 line starts with a verse. Letting text nodes through the every() fixes it, since the plain text is already "". The PT-4820 tripwire may not flip until then.

On \ca and \cp: I'm fine with your call. One sentence near clipboard-semantics.md:156 saying their content is kept would stop someone "fixing" it later.

Optional:

  • HiddenMarkersViewCopyPlugin.tsx:64: the two new declarations split the component from its doc comment.
  • StructureKeyboardPlugin.tsx:135: doc still names $insertsNothing.
  • No CHANGELOG entry for the protected drop (85c447a) or the collapsed-note copy/cut (ee572e4). clipboard-semantics.md:182/:185 and the PR body still describe the old protected drop. :141 misses the glyph-only copy.
  • A cut of a glyph still writes "\q1 ". Should it write nothing, like the copy?
  • A text entry with text: null passes the check and then the paste throws. Require a string.
  • collapsedNoteClipboard.test.tsx:3 narrates the old bug.
  • The 1000-level depth case adds nothing over the 40-level one, and if the cap regresses it times out and breaks the tests after it.

tjcouch-sil and others added 5 commits November 19, 2026 13:49
…n where it was

The copy widened the live selection to the whole note and left it there,
so typing next replaced the note without asking. The selection now goes
back once every copy writer has run; a cut still takes the whole note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
… any line

On a line led by text, a glyph-only selection also holds the text beside
it, so the check failed and `\q2 ` was copied. Text in a selection whose
plain text is empty contributes nothing, so it now passes. A glyph-only
cut wrote `\q1 ` too while removing nothing; it is now claimed the same
way, and still removes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…d cut, \ca/\cp kept

clipboard-semantics.md and the CHANGELOG describe the drop that lands at
its drop point under protected structure and the whole-note copy and cut,
which had no entries, and the glyph-only copy and cut; the doc says why a
chapter's \ca and \cp keep their content. StructureKeyboardPlugin's doc
names $insertsOnlyParagraphSeparators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
…or save

A crafted node copy could still get past the check: an attribute named
marker, number, type (or another field the save writes) replaced that
field on save; a milestone's attributeOrder as a string, or deeply nested
node state, made every later update throw; an unmatched marker without
its `*` saved as a new marker; a text of null threw on paste. Every
field each type's importJSON reads or the save writes is now checked:
attribute names may not be saved fields, attributeOrder is a list of
names, node state only the editor's own keys with values their own parse
keeps, unmatched markers end in `*`, text is a string, and element
format fields are their defaults. A unit test pins each check (each fails
if its check accepts everything); the 1000-level case goes, since the
40-level one and a unit test at the cap cover depth.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WCZ5GVaw3B7KJ9xc7TC6c
@tjcouch-sil

Copy link
Copy Markdown
Member Author
  1. Crafted node copies (High): fixed in 11979192. All your probes reproduced, and I audited every field each allowed type's importJSON reads and every field the save writes for it, so the check now covers the whole class:

    • Attributes: names may not be a field the save writes for that node (type, marker, content, the type's own saved fields such as number, code, caller, sid), nor __proto__, constructor or prototype. Names must be words and values strings.
    • attributeOrder: must be a list of attribute names.
    • Node state ($): only the editor's own state keys, each value kept by that state's own parse. So deep nesting there is refused too.
    • Unmatched markers: the * is required. Every unmatched marker in the corpus and the fixtures ends in one.
    • Text: must be a string (text: null is refused), with format 0, style empty and mode/detail in range. A tab's text must be a tab.
    • Element fields: format, indent, direction, textFormat and textStyle must be their defaults; none of them is saved.

    The new nodeCopy.utils.test.ts holds one crafted refusal per check, plus two positive controls: a legitimate copy holding every type, and one nested to the cap. Against the old check, 30 of its 58 tests fail. Forcing each check function to accept everything, one at a time, fails at least one test for every check, including fieldsProblem. htmlClipboardRoute.test.tsx gained 6 editor-level payloads (attribute overwrites included), and all 18 runs failed on the old code. The 1000-level case is gone, as you suggested. The clipboard corpus is unchanged.

  2. Ctrl+C inside a collapsed note (Medium): fixed in 62736de4. The copy still widens the selection to the whole note so every writer copies all of it, then puts the selection back once they have run; a cut still takes the whole note. The test compares a copy followed by typing with typing alone, in Standard and paragraph-structure: the selection is the same after the copy, and so is the document after typing. Without the fix it fails in both views.

  3. Glyph copy on a text-led line (Medium): fixed in 488cae84 as you suggested, by letting text nodes through, since the plain text is already empty. The test now covers a line led by a verse and one led by text.

On \ca and \cp: the sentence is in clipboard-semantics.md (dbdded4a).

Optional:

  • Doc split, $insertsNothing in the doc, collapsedNoteClipboard.test.tsx narration: fixed (488cae84, dbdded4a, 62736de4).
  • CHANGELOG and docs: entries for the protected drop (85c447a4) and the collapsed-note copy and cut (ee572e4d). The clipboard-semantics paragraphs on the protected drop and the glyph-only copy are corrected, and so is the PR description (dbdded4a).
  • Glyph-only cut: yes. It wrote \q1 while removing nothing (checked in the full editor, every structure mode). It is now claimed like the copy: nothing written, nothing removed (488cae84), pinned by a full-editor test in each mode.
  • text: null: refused (11979192).
  • 1000-level case: removed (11979192).

@katherinejensen00 katherinejensen00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So sorry to put you through one more round of review. I am hoping this is the last one.

Blocking:

  1. Ctrl+C inside a collapsed note in Standard view drops the closing \f* (caused by this round). TrailingNoteCaretGuardPlugin.tsx:345
    • Cause: the selection restore is queued with editor.update. That only runs after the copy writers when the copy is dispatched inside an update, which is how the tests do it. A real Ctrl+C dispatches outside one, so the restore lands before Standard view's copy writer runs.
    • Probe: a real copy event gives text ending …\ft B with no \f*. Pasting that back saves an unclosed footnote. With the restore removed, the whole note copies correctly. Simple mode wasn't affected.
    • Fix: run the restore after the whole copy has finished, and have the test fire a real copy event.
  2. A style attribute still replaces the marker (left over from blocker 1). nodeCopy.utils.ts:435
    • Cause: when the document is saved, the USJ-to-USX converter sets style from the marker and then writes every other attribute over it. paranext-core saves through that converter.
    • Probe: a pasted w char with style: "v" passes the check and saves as .
    • Fix: add style to the list of field names an attribute can't use, plus one test.

Worth fixing, not blocking:

  • Deep objects in node state: a deep object under a true/false node state (isAuthoredNbsp, isCaretHost) still gets through. Later edits are fine now, but saving the editor state to JSON throws a stack-overflow error, so copying that text would fail.
  • Possible crash instead of fallback: the check builds its warning messages before the try starts. From reading the code (I didn't run this one), a deeply nested paste would throw instead of falling back to the HTML.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants