Repository navigation
[US-1983] Add PDF/UA structural content rules to validate - #38
Open
unidoc-anom wants to merge 3 commits into
Open
unidoc-anom wants to merge 3 commits into
unidoc-anom wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description and links
validate --profile pdfua-1-structuralonly looked at the catalog (/MarkInfo,/StructTreeRoot,/Lang). On creator-tagged files it said nothing while veraPDF found an MCID with no ParentTree entry, an empty/S, and a table whose BDC/EMC pairs broke across a page. This adds five rules that read thedump structurereport from #37 and check the marked content of every page and the/Sof every structure element.Severities follow veraPDF 1.30.2 on the same faults. A rule is an error (exit 1) when veraPDF fails the file for it, and a warning when veraPDF only logs it:
mcid-in-parent-tree/StructParentsand the ParentTree; a BDC whose MCID can't be readstruct-elem-type/Snamemarked-content-balance/Contents(an array counts as one stream); an undecodable content stream is reported heremcid-uniquemcid-not-nestedThe Validate tab shows the new findings without a component change. While testing it on a production build I also fixed two things there: switching documents cleared the result, and the findings sat below the scope sentence and the rule list.
Jira: US-1983
Technical changes
internal/pdfcore/validate_structure.go(new): the five checks. Findings within a page are in content order. File text in messages is ASCII-quoted. Nothing is truncated: a page with 2000 duplicate MCIDs gives 2000 findings, each listing every occurrence.internal/pdfcore/validate.go: registry entries for the five rules and acheckStructurefield next tocheck.Validatebuilds the structure report once, before it takespdfMu, under its ownrecover. A close during the walk returns the cancel error. Any other walk failure leaves only the structure rules not evaluated.runRulereads the catalog first, so an unreadable catalog gives the same message under every rule.mcid-in-parent-treeis not evaluated when the structure tree root or a ParentTree number-tree node could not be read in full, because the "has no key N" verdict would then be about entries we never saw.struct-elem-typeis not evaluated when the root is not a dictionary or could not be read.ProfileGateshad no other caller and is gone.pdfa-1bexit codes don't change, since every rule there is an error rule.testdata/generate_test.go:ua-structural-clean.pdfplus one file per rule that trips only that rule. veraPDF ua1 results for each are committed undertestdata/verapdf-ua1/.tests/compliance-validation/verapdf_ua1_golden_test.gore-runs veraPDF when it is installed (and rewrites withPDFDEBUG_WRITE_VERAPDF_GOLDENS=1), otherwise it skips. CI only checks the recorded mapping.tests/ua-structural-rules: exit codes per fixture,--jsonshape, and the 2000-finding document checked line by line in both outputs.frontend/src/components/ValidateView.tsx: each document keeps its own profile, result, in-flight run and rules-section state, keyed by tab id. A document starts on PDF/A-1b until its profile is changed. A reload mints a new tab id, so a reloaded document starts empty. The findings come first. The scope sentence and the rule list sit below in a "Rules checked (N)" section, collapsed by default, whose label also counts rules that could not be evaluated.docs/cli-usage.md,docs/cli/scripting.md,docs/cli/validate-and-diff.mdand the godoc no longer say PDF/UA rules never fail the run. CHANGELOG entries under Unreleased (Added and Changed).Considerations
veraPDF 1.30.2 passes files with unbalanced BDC/EMC, duplicate MCIDs or nested MCIDs and only logs a warning. It fails an MCID with no ParentTree entry (7.1-3) and an empty
/S(7.1-5). We follow that split. One consequence: a content stream that can't be decoded is amarked-content-balancefinding, so it's a warning and doesn't fail CI on its own.Only page content is scanned. Form XObjects, annotation appearances and Type3 glyphs are not, and content outside any marked-content sequence (veraPDF 7.1-3 untagged content) is out of scope. Each rule's
checkstext says this.struct-elem-typedoesn't see elements below the walker's 256-level depth cap and still reports the rule as evaluated.The per-document state in
ValidateViewis kept for every tab id seen this session and isn't dropped when a tab closes. Each entry is one result object.How was this tested?
GOFLAGS=-count=1 scripts/test-all.shon the final tree: go vet, golangci-lint 0 issues, the root module, everytests/*/suite includingtests/ua-structural-rules, frontend typecheck, lint and Vitest. All suites passed.go test -race -count=1 ./internal/pdfcore/passed.runtime.Error, cancel, unreadable catalog, early page-tree stop, partly read root, truncated ParentTree), and confirmpdfMuis free afterwards. New guard tests were checked to fail with their guard removed.task package) on macOS: ran the fixtures through the Validate tab, switched between documents, expanded and reran.What could go wrong?
Scripts that run
validate --profile pdfua-1-structuraland expect exit 0 will now get exit 1 on files with unreachable MCIDs or bad/S. The CHANGELOG calls this out.A tagged file with a huge number of MCIDs per page produces one finding per fault and very long messages for duplicates. That is deliberate; nothing is truncated.
Screenshots/videos (if appropriate)
Checklist
devbranch, notmaster