Skip to content

[US-1983] Add PDF/UA structural content rules to validate - #38

Open
unidoc-anom wants to merge 3 commits into
devfrom
feat/pdfua-structural-content-rules
Open

unidoc-anom wants to merge 3 commits into
devfrom
feat/pdfua-structural-content-rules

Conversation

@unidoc-anom

Copy link
Copy Markdown
Contributor

Description and links

validate --profile pdfua-1-structural only looked at the catalog (/MarkInfo, /StructTreeRoot, /Lang). On creator-tagged files it said nothing while veraPDF found an MCID with no ParentTree entry, an empty /S, and a table whose BDC/EMC pairs broke across a page. This adds five rules that read the dump structure report from #37 and check the marked content of every page and the /S of every structure element.

Severities follow veraPDF 1.30.2 on the same faults. A rule is an error (exit 1) when veraPDF fails the file for it, and a warning when veraPDF only logs it:

Rule Severity Checks ISO 32000-1
mcid-in-parent-tree error every content MCID reaches a structure element through /StructParents and the ParentTree; a BDC whose MCID can't be read 14.7.4.4
struct-elem-type error every structure element has a usable /S name 14.7.2
marked-content-balance warning BMC/BDC and EMC pair up over the page's /Contents (an array counts as one stream); an undecodable content stream is reported here 14.6.1
mcid-unique warning no MCID marks two BDCs on one page 14.7.4.2
mcid-not-nested warning no MCID sequence opens inside another MCID sequence 14.7.4.1

The Validate tab shows the new findings without a component change. While testing it on a production build I also fixed two things there: switching documents cleared the result, and the findings sat below the scope sentence and the rule list.

Jira: US-1983

Technical changes

  1. internal/pdfcore/validate_structure.go (new): the five checks. Findings within a page are in content order. File text in messages is ASCII-quoted. Nothing is truncated: a page with 2000 duplicate MCIDs gives 2000 findings, each listing every occurrence.
  2. internal/pdfcore/validate.go: registry entries for the five rules and a checkStructure field next to check. Validate builds the structure report once, before it takes pdfMu, under its own recover. A close during the walk returns the cancel error. Any other walk failure leaves only the structure rules not evaluated. runRule reads the catalog first, so an unreadable catalog gives the same message under every rule.
  3. The rules fail closed rather than pass on what they could not read. The four content rules are not evaluated when the page-tree walk stopped early. mcid-in-parent-tree is not evaluated when the structure tree root or a ParentTree number-tree node could not be read in full, because the "has no key N" verdict would then be about entries we never saw. struct-elem-type is not evaluated when the root is not a dictionary or could not be read.
  4. Exit 1 is now decided per rule, not per profile: an error finding, or an error-severity rule that could not be evaluated. ProfileGates had no other caller and is gone. pdfa-1b exit codes don't change, since every rule there is an error rule.
  5. Fixtures from testdata/generate_test.go: ua-structural-clean.pdf plus one file per rule that trips only that rule. veraPDF ua1 results for each are committed under testdata/verapdf-ua1/. tests/compliance-validation/verapdf_ua1_golden_test.go re-runs veraPDF when it is installed (and rewrites with PDFDEBUG_WRITE_VERAPDF_GOLDENS=1), otherwise it skips. CI only checks the recorded mapping.
  6. New acceptance suite tests/ua-structural-rules: exit codes per fixture, --json shape, and the 2000-finding document checked line by line in both outputs.
  7. frontend/src/components/ValidateView.tsx: each document keeps its own profile, result, in-flight run and rules-section state, keyed by tab id. A document starts on PDF/A-1b until its profile is changed. A reload mints a new tab id, so a reloaded document starts empty. The findings come first. The scope sentence and the rule list sit below in a "Rules checked (N)" section, collapsed by default, whose label also counts rules that could not be evaluated.
  8. docs/cli-usage.md, docs/cli/scripting.md, docs/cli/validate-and-diff.md and the godoc no longer say PDF/UA rules never fail the run. CHANGELOG entries under Unreleased (Added and Changed).

Considerations

veraPDF 1.30.2 passes files with unbalanced BDC/EMC, duplicate MCIDs or nested MCIDs and only logs a warning. It fails an MCID with no ParentTree entry (7.1-3) and an empty /S (7.1-5). We follow that split. One consequence: a content stream that can't be decoded is a marked-content-balance finding, so it's a warning and doesn't fail CI on its own.

Only page content is scanned. Form XObjects, annotation appearances and Type3 glyphs are not, and content outside any marked-content sequence (veraPDF 7.1-3 untagged content) is out of scope. Each rule's checks text says this.

struct-elem-type doesn't see elements below the walker's 256-level depth cap and still reports the rule as evaluated.

The per-document state in ValidateView is kept for every tab id seen this session and isn't dropped when a tab closes. Each entry is one result object.

How was this tested?

  • GOFLAGS=-count=1 scripts/test-all.sh on the final tree: go vet, golangci-lint 0 issues, the root module, every tests/*/ suite including tests/ua-structural-rules, frontend typecheck, lint and Vitest. All suites passed.
  • go test -race -count=1 ./internal/pdfcore/ passed.
  • The veraPDF golden test ran against veraPDF 1.30.2 locally and matched.
  • Unit tests cover each rule's branches, every fail-closed path (walk panic, runtime.Error, cancel, unreadable catalog, early page-tree stop, partly read root, truncated ParentTree), and confirm pdfMu is free afterwards. New guard tests were checked to fail with their guard removed.
  • Vitest covers the Validate tab: findings grouped under Errors and Warnings with click and Enter navigation, the collapsed section and its label, per-document results, profile and section state, a run that finishes on another document, and a reload starting empty.
  • Production build (task package) on macOS: ran the fixtures through the Validate tab, switched between documents, expanded and reran.

What could go wrong?

Scripts that run validate --profile pdfua-1-structural and expect exit 0 will now get exit 1 on files with unreachable MCIDs or bad /S. The CHANGELOG calls this out.

A tagged file with a huge number of MCIDs per page produces one finding per fault and very long messages for duplicates. That is deliberate; nothing is truncated.

Screenshots/videos (if appropriate)

Checklist

  • Tests pass locally
  • I targeted the dev branch, not master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants