[US-1983] Add PDF/UA structural content rules to validate

CHANGELOG.md

@@ -7,13 +7,17 @@ All notable changes to UniDoc PDF Debugger are recorded here. Format follows Kee
 ### Added
 
 - `dump structure [--json] <file>` prints a tagged PDF's structure tree and lines it up with the page content. Each element shows its `/S` (and the role it maps to), reference, page, `/T`, `/Lang`, `/Alt`, `/ActualText` and `/E`, with its MCID, MCR and OBJR kids underneath. Each page gets a table, one row per MCID, with how often the content marks it, the element in the page's ParentTree slot and the elements whose `/K` claims it; rows where those do not agree on one element are marked `*`. Cycles, a tree deeper than 256 levels, a `/Pg` that is not a page, an EMC with nothing open, a missing property list, a `/StructParents` key the ParentTree lacks and malformed ParentTree nodes are reported where they occur, and the command still exits 0. `--json` carries the full report. MCIDs inside Form XObjects are not aligned
+- `validate --profile pdfua-1-structural` checks the marked content of every page's `/Contents` (an array is read as one stream, as ISO 32000-1 14.6.1 says) and the structure elements, with five new rules. `marked-content-balance` reports an EMC with nothing open and sequences still open at the end of the page, and also a page content stream that cannot be decoded or a `/Contents` that cannot be read, so unreadable content is a finding rather than a silent pass. `mcid-unique` reports an MCID that marks more than one BDC on a page, listing every occurrence. `mcid-in-parent-tree` reports an MCID that does not reach a structure element through the page's `/StructParents` and ParentTree array, a BDC whose MCID cannot be read, and page content or a content stream it could not check because it cannot be read or decoded. `mcid-not-nested` reports an MCID sequence opened inside another MCID sequence (a BMC or an MCID-less BDC around one is fine). `struct-elem-type` reports a structure element whose `/S` is absent, empty or not a name, or that cannot be read. Content-rule findings name the page, plus the content stream where one is involved, and a finding about one MCID gives the line and column of its BDC; `struct-elem-type` findings name the structure element. Form XObject, annotation appearance and Type3 glyph content is not scanned
 - File > "Reload from Disk" (Cmd+R on macOS, Ctrl+R on Windows and Linux, and F5 everywhere) opens the active tab's file again in place. The tab keeps its position and its left-rail view; the tree, selection, detail panel, history and recent jumps reset as for a fresh open. A short "reloaded - view reset" note appears at the top of the window. A reload that finishes after you switched to another tab replaces that tab without taking focus. If the file is gone or no longer parses, the old tab stays as it was and the error banner says why; on Windows a file another app has locked shows "in use by another app - try again" instead
 - Open files are watched for changes on disk, background tabs included. When a file is rewritten, replaced or deleted, its tab gets a small amber dot (2 to 4 seconds after the write while the window has focus, 10 to 20 seconds otherwise, also for a file a generator keeps rewriting) and a tooltip with the time it changed. The detail panel shows "Document updated" with Reload and Compare buttons, or "File not found". Nothing reloads by itself, and the marker never moves focus or shifts the layout. Detection compares size, modification time and file identity, so an in-place rewrite that keeps all three is not seen
 - File > "Compare with File on Disk" diffs the open version against the file as it is now on disk, in the Diff panel, labelled "On disk (14:05)". Files over 500 MB ask first. Reload in the Diff header then makes the compared copy the tab's document without parsing it again
 - Plain Text and the CLI's `dump bytes` (raw and `--json`) refuse to read a file that changed since it was parsed ("file changed on disk") instead of mixing the old structure with the new bytes. Signatures and `dump signatures` keep every signature's details and show the /Contents hole check as "not checked" for such a file; they used to report a hole mismatch. On Windows a file another app has locked shows "File is in use by another app" in Plain Text and in the hole check instead
 
 ### Changed
 
+- `validate --profile pdfua-1-structural` now exits 1 when it finds an MCID that does not reach the ParentTree or cannot be read, page content whose MCIDs could not be checked because it cannot be read or decoded, or a structure element without a usable `/S`, or when `mcid-in-parent-tree` or `struct-elem-type` could not be evaluated (for example when the page tree or the structure tree root could not be read). It used to exit 0 whatever it found. These are the two faults veraPDF fails a file on (ua1 clauses 7.1-3 and 7.1-5). `marked-content-balance`, `mcid-unique` and `mcid-not-nested` report warnings, as veraPDF does, so a marked-content imbalance, a duplicate MCID or a nested MCID exits 0; the catalog rules (`marked`, `struct-tree-root`, `lang`) stay warnings. A warning rule that could not be evaluated still exits 0. `pdfa-1b` exit codes are unchanged. The profile's scope sentence now names the content rules and what it does not examine
+- The Validate tab keeps each open document's profile, result and rules section when you switch to another document and back, so the checks no longer have to be run again. A run still in progress when you switch away finishes for its own document. A document starts on PDF/A-1b until its profile is changed, and a document reloaded from disk starts with no result. Switching documents used to clear the result
+- The Validate tab shows the findings first. The profile's scope sentence and the list of rules checked moved below them into a "Rules checked (N)" section that starts collapsed and stays open once expanded for that document; its label also counts rules that could not be evaluated, as in "Rules checked (8, 1 not evaluated)"
 - Non-zero bytes after the CMS in a signature's /Contents, or after the SignedData inside the ContentInfo's `[0]` wrapper, are now reported, and the signature still decomposes in full. Bytes after the CMS show once, as `Trailing bytes: 4 bytes after the CMS at offset 2028 contain non-zero data` in plain text, as `cmsTrailingBytes` (`offset`, `count`) in `--json`, and as a "Trailing bytes" row in the Signatures panel. Bytes after the SignedData get a note with the byte count. Such bytes used to be ignored without a word. Zero padding is not reported
 - A CMS nested more than 64 levels deep is now a decompose error for DER signatures too. DER signatures used to decompose with no nesting limit
 

cmd/cli/cmd_validate.go

@@ -21,7 +21,7 @@ var validateUsage = usageLine("validate")
 //
 //	0  ran successfully, ZERO error-severity problems (warnings/info allowed)
 //	1  ran successfully AND found >=1 error-severity problem (the CI gate), or
-//	   a rule of a gating profile could not be evaluated
+//	   an error-severity rule could not be evaluated
 //	2  operational or usage error (missing/unreadable file, unknown profile, bad
 //	   args, view failure)
 //
@@ -97,10 +97,11 @@ func execValidate(filePath, profile string, jsonOut, pretty bool) (exitCode int)
 
 // renderValidate writes the result (plain text default, --json opts in) and
 // returns the compliance-gate exit code: 1 when any error-severity problem was
-// found, else 0. It ALSO returns 1 (fail closed) when a gating-profile rule
-// could not be evaluated (degraded to an info problem) - reporting exit 0 "no
-// structural errors" for a file whose required rule never actually ran would
-// silently pass a CI gate on an unverified document.
+// found, else 0. It ALSO returns 1 (fail closed) when a rule whose registry
+// severity is error could not be evaluated (degraded to an info problem) -
+// reporting exit 0 "no structural errors" for a file whose required rule never
+// actually ran would silently pass a CI gate on an unverified document. A
+// warning-severity rule that could not be evaluated does not gate.
 func renderValidate(res *pdfcore.ValidationResult, jsonOut, pretty bool) int {
 	if jsonOut {
 		if err := emit(os.Stdout, res, pretty); err != nil {
@@ -111,15 +112,23 @@ func renderValidate(res *pdfcore.ValidationResult, jsonOut, pretty bool) int {
 		writeJSONError(os.Stderr, fmt.Sprintf("failed to write output: %v", err))
 		return 2
 	}
-	if res.Summary.Errors > 0 {
-		return 1
-	}
-	if res.Summary.Info > 0 && pdfcore.ProfileGates(res.Profile) {
+	if res.Summary.Errors > 0 || errorRuleNotEvaluated(res) {
 		return 1
 	}
 	return 0
 }
 
+// errorRuleNotEvaluated reports whether a rule whose registry severity is
+// error could not be evaluated in res.
+func errorRuleNotEvaluated(res *pdfcore.ValidationResult) bool {
+	for _, r := range res.Rules {
+		if !r.Evaluated && r.Severity == "error" {
+			return true
+		}
+	}
+	return false
+}
+
 // severityOrder is the display order for the grouped plain-text problem list.
 var severityOrder = []string{"error", "warning", "info"}
 
@@ -207,9 +216,9 @@ func printValidatePlain(out io.Writer, res *pdfcore.ValidationResult) error {
 		fmt.Fprintf(&b, ", %d %s", res.Summary.Info, plural(res.Summary.Info, "info problem"))
 	}
 	b.WriteString("\n")
-	// renderValidate exits 1 on an info problem under a gating profile even with
-	// no errors; name that cause next to the summary.
-	if res.Summary.Errors == 0 && res.Summary.Info > 0 && pdfcore.ProfileGates(res.Profile) {
+	// renderValidate exits 1 when an error-severity rule could not be
+	// evaluated even with no errors; name that cause next to the summary.
+	if res.Summary.Errors == 0 && errorRuleNotEvaluated(res) {
 		b.WriteString("Exit status 1: a rule that gates the exit code could not be evaluated\n")
 	}
 	_, err := io.WriteString(out, b.String())

cmd/cli/validate_gate_test.go

@@ -9,11 +9,11 @@ import (
 )
 
 // TestRenderValidate_ExitCodeGate locks the three-way exit contract at the point
-// it is decided (renderValidate), including the fail-closed rule: an info-only
-// degraded run under a GATING profile (pdfa-1b) must exit 1, so a required rule
-// that never actually ran cannot silently report a clean exit 0. The same
-// info-only run under the non-gating pdfua-1-structural profile exits 0. The
-// plain-text exit explanation prints exactly when the exit is 1 with no errors.
+// it is decided (renderValidate), including the fail-closed rule: a run where an
+// error-severity rule degraded to info must exit 1, so a required rule that
+// never actually ran cannot silently report a clean exit 0. A run where only a
+// warning-severity rule degraded exits 0. The plain-text exit explanation
+// prints exactly when the exit is 1 with no errors.
 func TestRenderValidate_ExitCodeGate(t *testing.T) {
 	// Silence the plain-text report the function writes to os.Stdout.
 	devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
@@ -25,18 +25,22 @@ func TestRenderValidate_ExitCodeGate(t *testing.T) {
 	os.Stdout = devnull
 	defer func() { os.Stdout = old }()
 
+	degraded := func(id string, sev pdfcore.Severity) []pdfcore.RuleInfo {
+		return []pdfcore.RuleInfo{{RuleID: id, Severity: sev, Evaluated: false}}
+	}
 	tests := []struct {
 		name    string
 		profile string
 		summary pdfcore.ValidationSummary
+		rules   []pdfcore.RuleInfo
 		want    int
 	}{
-		{"clean pdfa-1b exits 0", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{}, 0},
-		{"error problem exits 1", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Errors: 1}, 1},
-		{"info-only under gating profile fails closed (exit 1)", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Info: 1}, 1},
-		{"info-only under non-gating profile exits 0", pdfcore.ProfilePDFUA1Structural, pdfcore.ValidationSummary{Info: 1}, 0},
-		{"warnings alone do not gate", pdfcore.ProfilePDFUA1Structural, pdfcore.ValidationSummary{Warnings: 3}, 0},
-		{"errors and info under gating profile exit 1", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Errors: 1, Info: 1}, 1},
+		{"clean pdfa-1b exits 0", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{}, nil, 0},
+		{"error problem exits 1", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Errors: 1}, nil, 1},
+		{"info-only from a degraded error rule fails closed (exit 1)", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Info: 1}, degraded("font-embedding", "error"), 1},
+		{"info-only from a degraded warning rule exits 0", pdfcore.ProfilePDFUA1Structural, pdfcore.ValidationSummary{Info: 1}, degraded("lang", "warning"), 0},
+		{"warnings alone do not gate", pdfcore.ProfilePDFUA1Structural, pdfcore.ValidationSummary{Warnings: 3}, nil, 0},
+		{"errors and a degraded error rule exit 1", pdfcore.ProfilePDFA1B, pdfcore.ValidationSummary{Errors: 1, Info: 1}, degraded("font-embedding", "error"), 1},
 	}
 	for _, tc := range tests {
 		t.Run(tc.name, func(t *testing.T) {
@@ -45,6 +49,7 @@ func TestRenderValidate_ExitCodeGate(t *testing.T) {
 				Summary:    tc.summary,
 				Problems:   []pdfcore.Problem{},
 				Disclaimer: pdfcore.DisclaimerText,
+				Rules:      tc.rules,
 			}
 			got := renderValidate(res, false, false)
 			if got != tc.want {

cmd/cli/validate_rule_gate_test.go

@@ -0,0 +1,98 @@
+package main
+
+import (
+	"os"
+	"strings"
+	"testing"
+
+	"unidoc-pdf-debugger/internal/pdfcore"
+)
+
+// renderValidate gates per rule: a run exits 1 on an error-severity problem or
+// when a rule whose registry severity is error could not be evaluated, under
+// either profile. A warning-severity rule that could not be evaluated does not
+// gate. The plain-text exit explanation prints exactly when the exit is 1 with
+// no errors.
+func TestRenderValidate_GatesOnUnevaluatedErrorRule(t *testing.T) {
+	devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
+	if err != nil {
+		t.Fatalf("open devnull: %v", err)
+	}
+	defer func() { _ = devnull.Close() }()
+	old := os.Stdout
+	os.Stdout = devnull
+	defer func() { os.Stdout = old }()
+
+	rule := func(id string, sev pdfcore.Severity, evaluated bool, findings int) pdfcore.RuleInfo {
+		return pdfcore.RuleInfo{RuleID: id, Severity: sev, SpecRef: "spec", Checks: "checks", Evaluated: evaluated, Findings: findings}
+	}
+	tests := []struct {
+		name    string
+		profile string
+		summary pdfcore.ValidationSummary
+		rules   []pdfcore.RuleInfo
+		want    int
+	}{
+		{
+			"UA error rule not evaluated exits 1", pdfcore.ProfilePDFUA1Structural,
+			pdfcore.ValidationSummary{Info: 1},
+			[]pdfcore.RuleInfo{rule("lang", "warning", true, 0), rule("mcid-unique", "error", false, 0)},
+			1,
+		},
+		{
+			"UA warning rule not evaluated exits 0", pdfcore.ProfilePDFUA1Structural,
+			pdfcore.ValidationSummary{Info: 1},
+			[]pdfcore.RuleInfo{rule("lang", "warning", false, 0), rule("mcid-unique", "error", true, 0)},
+			0,
+		},
+		{
+			"UA error finding exits 1", pdfcore.ProfilePDFUA1Structural,
+			pdfcore.ValidationSummary{Errors: 1},
+			[]pdfcore.RuleInfo{rule("lang", "warning", true, 0), rule("mcid-unique", "error", true, 1)},
+			1,
+		},
+		{
+			"UA warnings only exit 0", pdfcore.ProfilePDFUA1Structural,
+			pdfcore.ValidationSummary{Warnings: 2},
+			[]pdfcore.RuleInfo{rule("lang", "warning", true, 1), rule("struct-elem-type", "warning", true, 1), rule("mcid-unique", "error", true, 0)},
+			0,
+		},
+		{
+			"pdfa-1b error rule not evaluated exits 1", pdfcore.ProfilePDFA1B,
+			pdfcore.ValidationSummary{Info: 1},
+			[]pdfcore.RuleInfo{rule("font-embedding", "error", false, 0), rule("output-intent", "error", true, 0)},
+			1,
+		},
+		{
+			"pdfa-1b clean exits 0", pdfcore.ProfilePDFA1B,
+			pdfcore.ValidationSummary{},
+			[]pdfcore.RuleInfo{rule("font-embedding", "error", true, 0)},
+			0,
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			res := &pdfcore.ValidationResult{
+				Profile:    tc.profile,
+				Summary:    tc.summary,
+				Problems:   []pdfcore.Problem{},
+				Disclaimer: pdfcore.DisclaimerText,
+				Rules:      tc.rules,
+			}
+			if got := renderValidate(res, false, false); got != tc.want {
+				t.Errorf("renderValidate exit = %d, want %d", got, tc.want)
+			}
+			if got := renderValidate(res, true, false); got != tc.want {
+				t.Errorf("renderValidate --json exit = %d, want %d", got, tc.want)
+			}
+			var b strings.Builder
+			if err := printValidatePlain(&b, res); err != nil {
+				t.Fatalf("printValidatePlain: %v", err)
+			}
+			wantLine := tc.want == 1 && tc.summary.Errors == 0
+			if has := strings.Contains(b.String(), "Exit status 1: a rule that gates the exit code could not be evaluated"); has != wantLine {
+				t.Errorf("exit explanation present = %v, want %v:\n%s", has, wantLine, b.String())
+			}
+		})
+	}
+}

docs/cli-usage.md

@@ -28,9 +28,12 @@ pdfdebug validate --profile pdfua-1-structural file.pdf
 
 After the profile name and its scope sentence, the output lists the rules it
 checked with `0 found` or `N found` beside each, then the problems and a
-summary line. The profile checks catalog-level entries only, such as whether
-the document is marked as tagged. A file that passes every one of them can
-still fail PDF/UA, so a clean run is not a verdict; use veraPDF for that.
+summary line. The profile checks catalog entries such as whether the document
+is marked as tagged, that every structure element has a usable `/S`, and the
+marked content of each page: balanced BMC/BDC and EMC, MCIDs that are unique,
+not nested and reachable through the ParentTree. A file with no finding from
+any of them can still fail PDF/UA, so a clean run is not a verdict; use
+veraPDF for that.
 `pdfdebug --help` lists the current rules of each profile.
 
 ### Compare two versions of a file
@@ -69,9 +72,11 @@ or `--ref` for any stream object. An attachment comes out with
 pdfdebug validate --json --profile pdfua-1-structural file.pdf > report.json
 ```
 
-Branch on the exit code first and read `summary` from the JSON second, because
-`pdfua-1-structural` findings are warnings and a rule that could not run is
-info, and both exit 0. The full script is under
+Branch on the exit code first and read `summary` from the JSON second. Under
+`pdfua-1-structural`, `mcid-in-parent-tree` and `struct-elem-type` report
+errors and exit 1; the other rules report warnings, and one of those that could
+not run reports info; both exit 0. An error rule that could not run also
+reports info but exits 1. The full script is under
 [Exit codes](cli/scripting.md#exit-codes).
 
 ## Commands