Skip to content

Built-in engines W30: Word layout, TXT and Markdown output, left-out notices, smaller DjVu PDFs - #1727

Open
KnifeLemon wants to merge 12 commits into
builtin-enginesfrom
p4/w30
Open

KnifeLemon wants to merge 12 commits into
builtin-enginesfrom
p4/w30

Conversation

@KnifeLemon

@KnifeLemon KnifeLemon commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Wave 30 of the built-in engines: a Word layout batch, built-in TXT output fixes, left-out notices, and smaller DjVu PDFs with links, bookmarks and indirect files.

Word layout

  • compatibilityMode below 15 (and no mode given, as Word treats it) places a left-aligned table's edge one first-cell margin out (HTable.EdgeBeforeIndent, layout only); tblInd read from table styles.
  • A fixed table wider than the body runs into the margin (both margins when centred) instead of shrinking.
  • w:tblCellSpacing (space on each side of a cell, outer border around the cells), w:wBefore/w:wAfter widths for empty grid columns, RTF \trautofit1 with \trwWidth sized like Word autofit.
  • List numbers: w:lvlJc center/right, the level's w:rPr over the paragraph mark's format (HListRef.MarkerFormat, layout only); custom tab stops between a hanging first line and the left indent are used.
  • Word page borders drawn over the text unless zOrder="back".
  • A table that flows into a column of another width is laid out again for it; list counters saved and restored (ListNumbers.Save/Restore).
  • Not done: balancing columns of different widths at a continuous section break.

TXT output

  • PAGE / NUMPAGES fields write the number the file showed when saved (HField.Shown: DOCX field results and w:fldSimple, ODT page-number text, RTF \fldrslt); otherwise a count from sections and page breaks.
  • A paragraph holding only a page break writes an empty line (HBlock.BreakParagraphsBefore); layout unchanged.
  • List numbers are followed by a tab (Word's w:suff default; space or nothing when the level says so), also for HWP/HWPX and e-book TXT.
  • RTF/Word/ODT → TXT built in at cost 15, so it wins wherever LibreOffice stands; LibreOffice's RTF → TXT edge removed (its fallback stays for Word 6/95).
  • RTF/Word/ODT → Markdown in one step with MarkdownWriter (cost 15, direct only). Before, DOCX → Markdown went through the HWPX writer and the equations came back as 한글 equation script in italics; now they're TeX math (checked on DOCX to PDF conversion causes incorrect equation formatting and corrupted Visio diagrams #356's 114 equations). Pandoc no longer takes RTF → Markdown.
  • Markdown: an equation alone on its line is $$…$$ also when tabs centre it; tabs at a paragraph's start or end no longer write spaces (an indented code block).
  • Markdown tables: a cell's paragraphs are joined by <br> and its list items marked • / 1. (Word → Markdown: tables with merged cells come out as HTML, and picture links are absolute #391), as Pandoc's filter did; the two Word → Markdown: tables with merged cells come out as HTML, and picture links are absolute #391 tests in PandocMarkupTests now check the built-in output and run without Pandoc.
  • engine.hwp-text.description in 23 locales; stale Java/H2Orestart line removed from bn-BD, fa, vi engines.guidance.linux.libreoffice.

Left-out notices

  • DOCX: vertical text boxes (wps:bodyPr vert, VML layout-flow:vertical) and ActiveX controls are counted and flagged; legacy check boxes and drop-downs are drawn (☒/☐, chosen entry).
  • Slides: PPTX, PPT, ODP and ODG drawn by the built-in renderer now report what they leave out; ODP master shapes counted once.
  • A chart in a header shared by several sections is counted once.
  • DOCX to PDF conversion causes incorrect equation formatting and corrupted Visio diagrams #356's file checked end to end: the notice is right (one Visio object from its saved picture).
  • 11 new notice keys in 23 locales.

DjVu

  • Masks written as JBIG2 (symbol dictionary + text region from the page's DjVu shapes) or CCITT G4, whichever is smaller; Flate when both do badly. New CcittG4.cs (encoder and decoder) and Jbig2Encoder.cs, written from T.6 and T.88.
    • DjVu3Spec 6.04 → 1.45 MB, malliavin 13.9 → 1.51 MB, czech 4.24 → 2.57 MB.
    • Every page of the public sample files renders pixel-identical before and after (pdfium). G4 also checked with PdfPig's decoder; JBIG2 only with pdfium.
  • NAVM bookmarks → PDF outline; ANTa/ANTz hyperlinks → link annotations (http, https, ftp, mailto and page links only). DjvuLimits.MaxAnnotationNodes.
  • Indirect documents read from below the index file's folder (no absolute paths, .., :, device names, symlinks or junctions); a missing page still refuses with error.djvu_indirect.
  • Smmr (G4) masks read, whole, striped and inverted.

Merge notes

  • DocxReader.cs: the TXT branch's PAGE/NUMPAGES field numbers combined with the left-out branch's legacy form fields.
  • Two commits on top of the merge: the Markdown route and equation fix, and the Markdown table cells (found by CI with Pandoc installed).

Tests

Run locally on the merged branch:

🤖 Generated with Claude Code

KnifeLemon and others added 11 commits November 20, 2026 22:18
…-in TXT

RTF, Word and ODT to TXT on the built-in route:
- PAGE / NUMPAGES fields write the number the file showed when it was
  saved (Word's field result, ODT's text:page-number text, RTF's
  \fldrslt), kept as HField.Shown; without one, the page counted from
  the sections and page breaks. Before, the field came out empty.
- A Word paragraph that holds only a page break is an empty line, as
  Word and LibreOffice save it. The DOCX reader still leaves it out of
  the layout and counts it on the next block (BreakParagraphsBefore).
  No form feed.
- A list number or bullet is followed by a tab, or by the space or
  nothing a Word list level asks for (w:suff), as Word and LibreOffice
  write plain text. HWP/HWPX and e-book TXT use the same marker.
- The built-in RTF, Word and ODT to TXT edges cost 15, below
  LibreOffice's fallback (25) with any engine order penalty, so they
  win wherever LibreOffice stands in the engine list. LibreOffice's
  separate RTF to TXT edge (10) goes. A Word 6.0 / 95 file still goes
  to LibreOffice.
- ENGINES.md and the hwp-text engine description (all 23 languages)
  name RTF, Word and ODT to TXT.
- bn-BD, fa, vi: the Linux LibreOffice guidance no longer mentions
  Java and H2Orestart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g, list numbers, page border order

- Word 2010-and-older documents (compatibilityMode < 15, or none given) measure
  tblInd to the first cell's text: the table's edge sits a cell margin further out
  (layout-only HTable.EdgeBeforeIndent); tblInd also comes from the table style.
- A fixed table in the body wider than its column keeps its widths and runs into
  the margin (both margins when centred) instead of being narrowed.
- w:tblCellSpacing: cells inset on every side, the table's border drawn around
  them; w:wBefore / w:wAfter size the grid columns a row leaves empty; RTF rows
  with \trautofit1 (and \trwWidth) are sized as Word's autofit tables.
- List numbers: w:lvlJc center/right, the number in the paragraph mark's format
  with the level's w:rPr over it (bullets keep their mapped character's font),
  and tab stops between a hanging first line's start and the left indent.
- Page borders go over the text unless w:zOrder="back".
- In columns of their own widths, a table that goes on in a column of another
  width is laid out again for it from its next row group (list numbers restored).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…boxes, controls and slide decks

- DOCX: text boxes written top to bottom (wps:bodyPr vert, VML layout-flow)
  are named as vertical text set horizontally; ActiveX controls are named as
  form controls drawn from their saved picture or left out. Both are flagged
  by the capability check, so such files keep LibreOffice's route.
- DOCX: legacy check boxes and drop-downs (w:ffData) are drawn as Word prints
  them (a checked or empty box, the chosen entry) instead of nothing.
- DOCX: a header or footer shared by several sections counts what it leaves
  out once, not once per section.
- Slides: PPTX, PPT and ODP decks drawn by the slide renderer get the same
  note: the capability check's pass run to the end for PPTX (and what a PPT
  translation dropped), the ODP reader's own checks counted once per shape.
  ODG drawings get it through the drawing converter.
- 11 new notice texts in all 23 languages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Smmr

- PDF masks are coded the smallest way: JBIG2 (the JB2 shapes as a symbol
  dictionary and a text region, MQ-coded, written from ITU-T T.88), CCITT
  G4 (own encoder from ITU-T T.6), or Flate for dithered pictures. The
  DjVu 3 specification goes from 6.0 MB to 1.45 MB, a 115-page scanned
  book from 13.9 MB to 1.5 MB; the pages draw pixel for pixel as before.
- ANTa/ANTz hyperlinks become PDF link annotations: links into the
  document (by id, name, title, page number or +n/-n) and http, https,
  ftp and mailto addresses only. NAVM bookmarks become the PDF outline.
- Indirect documents read their component files from below the index
  file's folder only (no "..", roots, drives, ':' or device names, no
  path through a link); a missing page or included file is still refused
  with error.djvu_indirect.
- Smmr (G4) masks are read, whole or in stripes, inverted or not.
- Bound annotation parts (DjvuLimits.MaxAnnotationNodes) and nesting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page-sized shape (an Smmr mask) no longer needs a padded copy of its
pixels while it is coded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	src/Filee.Engines/Hwp/Hwpx/Docx/DocxReader.cs
…s math

DOCX, ODT and RTF reached Markdown through the HWPX writer, which gave the
equations back as 한글 equation script, written as italic text. The HWP
text engine now writes Markdown from them directly (cost 15, direct only,
like TXT), so equations come out as TeX math.

An equation alone on its line is display math also when tabs centre it,
and tabs at the start or end of a paragraph no longer write spaces that
turned the paragraph into an indented code block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 20, 2026
A cell's paragraphs are joined by <br> and its list items marked "•" or "1.", as Pandoc's filter did; before,
the built-in writer joined them with spaces. The two #391 tests no longer need Pandoc: DOCX → Markdown goes
through hwp-text wherever Pandoc stands, so they check the built-in output (pictures linked relative to the
Markdown file, merged cells as a Markdown table).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant