From ca9751625743ef381b63f7e919a072c6fb532029 Mon Sep 17 00:00:00 2001 From: jgaalen Date: Mon, 12 Oct 2026 16:27:54 +0200 Subject: [PATCH 1/2] Correct XPath, JSONPath and null rendering bugs and repair admin documentation --- README.md | 4 +- docs/COMPATIBILITY.md | 8 + docs/UPSTREAM_BUG_FIXES.md | 54 ++++++ internal/adminassets/PROVENANCE.json | 8 +- .../swagger/wiremock-admin-api.json | 62 +++++++ pkg/breakmock/admin_assets.go | 2 +- pkg/breakmock/matcher.go | 33 +++- pkg/breakmock/matcher_compat_test.go | 20 ++- pkg/breakmock/matcher_distance_path.go | 5 +- pkg/breakmock/template.go | 2 +- pkg/breakmock/upstream_bug_regression_test.go | 161 ++++++++++++++++++ testdata/matcher_cases.json | 10 +- .../upstream-bug-fixes/validation.json | 142 +++++++++++++++ 13 files changed, 494 insertions(+), 17 deletions(-) create mode 100644 docs/UPSTREAM_BUG_FIXES.md create mode 100644 pkg/breakmock/upstream_bug_regression_test.go create mode 100644 testdata/reports/upstream-bug-fixes/validation.json diff --git a/README.md b/README.md index 36d74df..9aa0a25 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,9 @@ Its source is preserved only on the historical `codex/rust-port` branch; see **Development status:** a compatibility port under active construction, not a complete drop-in replacement. The pinned reference is **WireMock 4.0.0-beta.40**. Upstream is a beta release; BreakMock exposes its implemented messaging features by default, without a beta flag. See [COMPATIBILITY.md](docs/COMPATIBILITY.md) -for tested coverage and known gaps. Performance and compatibility are separate +for tested coverage and known gaps. We intentionally correct [five upstream bugs](docs/UPSTREAM_BUG_FIXES.md), +including matcher and JSON-template errors, so those cases differ from Java. +Performance and compatibility are separate claims; neither is inferred from the implementation language. ## Why refactor WireMock to Go? diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 3223a90..99597c9 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -10,6 +10,14 @@ This repository does **not** claim compatibility with every possible mapping, extension or transport configuration. Checked-in differential reports document a bounded corpus; a passing fixture is not proof of an entire feature's equivalence. +## Intentional upstream bug corrections + +BreakMock now corrects five confirmed upstream bugs by default, covering XPath +predicates, JSONPath empty-array absence, null JSON serialization, the trailing-slash +docs route and the scenario-state OpenAPI operation. See [the corrections and +migration notes](UPSTREAM_BUG_FIXES.md). Strict Java comparisons remain unchanged +and report affected cases as failures; these differences are not byte parity. + ## Implemented and exercised - HTTP mappings, priorities, insertion ordering, scenarios, text/JSON/binary/file diff --git a/docs/UPSTREAM_BUG_FIXES.md b/docs/UPSTREAM_BUG_FIXES.md new file mode 100644 index 0000000..dad0a01 --- /dev/null +++ b/docs/UPSTREAM_BUG_FIXES.md @@ -0,0 +1,54 @@ +# Corrections to upstream WireMock bugs + +BreakMock corrects these confirmed bugs by default. These are intentional +behavior differences from the pinned Java WireMock 4.0.0-beta.40 reference, +not claims that every response remains byte-identical to Java. + +| Upstream issue | Correction | +| --- | --- | +| [#3510](https://github.com/wiremock/wiremock/issues/3510) | A scalar `matchesXPath` predicate uses XPath boolean conversion: false, zero, NaN and empty strings do not match. Node-set matching and explicitly nested value matchers retain their behavior. | +| [#3159](https://github.com/wiremock/wiremock/issues/3159) | `matchesJsonPath` with `absent` does not match a present empty array selected by a definite path such as `$.foo`. Empty wildcard/filter selections and missing properties still count as absent. Near-miss distance agrees with the corrected match. | +| [#3282](https://github.com/wiremock/wiremock/issues/3282) | `toJson` emits the literal `null` for null/missing input, producing valid JSON when used as a value. Existing non-null serialization is unchanged. | +| [#3332](https://github.com/wiremock/wiremock/issues/3332) | GET and HEAD on `/__admin/docs/` work like `/__admin/docs`, including absolute links to the specification and Swagger UI. | +| [#3319](https://github.com/wiremock/wiremock/issues/3319) | The bundled OpenAPI specification documents `PUT /__admin/scenarios/{name}/state`, its optional body/reset behavior, and its 200/404/422 responses. The endpoint implementation is unchanged. | + +## Compatibility and migration + +Mappings that relied on false XPath expressions matching, or an empty array being +absent, will select differently. Templates that relied on null `toJson` input +rendering no characters now receive `null`. Use an explicit conditional when an +empty output is required. There is no legacy-bug compatibility switch. + +Changing the served specification necessarily changes its response bytes. The +original upstream asset hash is retained in `internal/adminassets/PROVENANCE.json`, +with the modified specification hash recorded separately. + +The existing live Java differential scripts are unchanged: they still report +these differences as failures, without filtering cases, normalizing the changed +output, or rewriting the Java reference. The pre-existing JVM byte-array identity +diagnostic also remains a strict mismatch. Therefore strict parity CI is not +expected to be green solely because these correctness fixes pass. + +Captured Java matcher fixture results are retained. Two cases now also record +an explicit `breakmockMatches: false` expectation for #3159, which Go unit tests +assert. The live Java comparison continues to compare actual server results. + +Regression tests in `pkg/breakmock/upstream_bug_regression_test.go` cover positive +and negative predicates, empty-array versus empty-selection semantics, valid null +JSON, HTTP routing, documentation links and the OpenAPI operation. Run: + +```sh +go test -race ./... +go vet ./... +python3 scripts/verify.py --jar /tmp/wiremock.jar --output-dir /tmp/compatibility +``` + +## Validation results + +Go unit/race tests, vet, all 11 Toxiproxy executable integration checks and the +Docker build pass. The full unchanged Java suite passes 15 of 19 invocations: +matcher has two intentional empty-array differences, templates have two null +serialization differences, admin has the updated specification, and diagnostics +retains the existing JVM-identity mismatch. The final matcher/distance rerun +confirms the same two differences and no distance-suite failures. +See [the validation summary](../testdata/reports/upstream-bug-fixes/validation.json). diff --git a/internal/adminassets/PROVENANCE.json b/internal/adminassets/PROVENANCE.json index d1d6a55..81a3a5a 100644 --- a/internal/adminassets/PROVENANCE.json +++ b/internal/adminassets/PROVENANCE.json @@ -40,5 +40,11 @@ "assets/swagger-ui/swagger-ui-dist/swagger-ui.js.map": "e94e33402dac546894ee2e2a13c76d9e0e2b1dc2e78d5c9fadd90bd2b3397159", "doc-index.html": "2109bc58585f8b73aa37dc419340ba3282eb6b3a6ac31a32a85e94cc3aff70a7", "swagger/wiremock-admin-api.json": "4306cf1dada9455ad6b41e1344552f0dd6980edb3cf2471395a14dd8fee64d97" + }, + "localModifications": { + "swagger/wiremock-admin-api.json": { + "sha256": "e66006261d3dd75b25f33e9b2c92d7dced6020221492a6e151eecd35c7d74f0c", + "reason": "Document PUT /__admin/scenarios/{name}/state (WireMock issue #3319). Original upstream asset hash remains in files." + } } -} \ No newline at end of file +} diff --git a/internal/adminassets/swagger/wiremock-admin-api.json b/internal/adminassets/swagger/wiremock-admin-api.json index 3cc4ea2..ffb158a 100644 --- a/internal/adminassets/swagger/wiremock-admin-api.json +++ b/internal/adminassets/swagger/wiremock-admin-api.json @@ -1228,6 +1228,68 @@ } } }, + "/__admin/scenarios/{name}/state": { + "put": { + "operationId": "setScenarioState", + "summary": "Set or reset the state of a scenario", + "tags": [ + "Scenarios" + ], + "description": "Sets an existing scenario to a state referenced by its mappings. An empty request body resets it to Started.", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + }, + "description": "Scenario name" + } + ], + "requestBody": { + "required": false, + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "state" + ], + "properties": { + "state": { + "type": "string", + "description": "A state referenced by this scenario’s mappings" + } + } + }, + "example": { + "state": "Done" + } + } + } + }, + "responses": { + "200": { + "description": "State changed or reset", + "content": { + "application/json": { + "schema": { + "type": "object" + }, + "example": {} + } + } + }, + "404": { + "description": "Scenario does not exist" + }, + "422": { + "description": "Invalid JSON or unsupported/missing state" + } + } + } + }, "/__admin/scenarios/reset": { "post": { "operationId": "resetAllScenarios", diff --git a/pkg/breakmock/admin_assets.go b/pkg/breakmock/admin_assets.go index 1217a7f..6ae3e75 100644 --- a/pkg/breakmock/admin_assets.go +++ b/pkg/breakmock/admin_assets.go @@ -15,7 +15,7 @@ func serveAdminAsset(w http.ResponseWriter, r *http.Request) bool { p := strings.TrimPrefix(r.URL.Path, "/__admin/") name := "" switch p { - case "docs": + case "docs", "docs/": name = "doc-index.html" case "docs/swagger": name = "swagger/wiremock-admin-api.json" diff --git a/pkg/breakmock/matcher.go b/pkg/breakmock/matcher.go index 68c1a5a..6bf441b 100644 --- a/pkg/breakmock/matcher.go +++ b/pkg/breakmock/matcher.go @@ -7,6 +7,7 @@ import ( "encoding/json" "fmt" "io" + "math" "math/big" "mime" "mime/multipart" @@ -517,6 +518,11 @@ func matchJSONPath(expected any, actual string) bool { return matchValue(nested, string(raw), true) } if len(array) == 0 { + // A definite path can select a present empty array. An indefinite + // path with zero results instead means no nodes were selected. + if _, absent := nested["absent"].(bool); absent && matcherJSONPathDefinite(expression) { + return false + } return matchValue(nested, "", false) } for _, v := range array { @@ -547,6 +553,17 @@ func matcherJSONValueString(v any) string { return string(raw) } +// XPath object syntax may contain only expression/namespace metadata, without +// a nested value matcher. It then has the same predicate semantics as a string. +func xpathHasValueMatcher(pattern map[string]any) bool { + for key := range pattern { + if key != "expression" && key != "xPathNamespaces" { + return true + } + } + return false +} + func matchXPath(expected any, p map[string]any, actual string) bool { expression, ok := expected.(string) nested := map[string]any(nil) @@ -584,8 +601,18 @@ func matchXPath(expected any, p map[string]any, actual string) bool { } evaluation := compiled.Evaluate(xmlquery.CreateXPathNavigator(doc)) if _, ok := evaluation.(*xpath.NodeIterator); !ok { - if nested == nil { - return true + if !xpathHasValueMatcher(nested) { + // XPath 1.0 boolean conversion applies to scalar predicates too. + switch value := evaluation.(type) { + case bool: + return value + case float64: + return value != 0 && !math.IsNaN(value) + case string: + return value != "" + default: + return false + } } return matchValue(nested, fmt.Sprint(evaluation), true) } @@ -595,7 +622,7 @@ func matchXPath(expected any, p map[string]any, actual string) bool { if node.Type == xmlquery.ElementNode || node.Type == xmlquery.DocumentNode { rendered = strings.TrimPrefix(node.OutputXMLWithOptions(xmlquery.WithOutputSelf(), xmlquery.WithIndentation(" "), xmlquery.WithEmptyTagSupport()), "\n") + "\n" } - if nested == nil || matchValue(nested, rendered, true) { + if !xpathHasValueMatcher(nested) || matchValue(nested, rendered, true) { return true } } diff --git a/pkg/breakmock/matcher_compat_test.go b/pkg/breakmock/matcher_compat_test.go index eedd226..da5e204 100644 --- a/pkg/breakmock/matcher_compat_test.go +++ b/pkg/breakmock/matcher_compat_test.go @@ -17,18 +17,26 @@ func TestJavaMatcherFixtures(t *testing.T) { t.Fatal(err) } var cases []struct { - Name string - Pattern map[string]any - Body string - Matches bool + Name string + Pattern map[string]any + Body string + Matches bool + BreakMockMatches *bool `json:"breakmockMatches"` } if err = json.Unmarshal(raw, &cases); err != nil { t.Fatal(err) } for _, c := range cases { t.Run(c.Name, func(t *testing.T) { - if got := MatchRequest(c.Pattern, Request{Method: "POST", URL: "/test", Body: []byte(c.Body)}); got != c.Matches { - t.Fatalf("got %v, Java reference %v", got, c.Matches) + want := c.Matches + // Keep the captured Java result, but assert corrected semantics for + // explicitly documented upstream bugs. Live differential checks still + // compare the original Java and Go bytes without exemptions. + if c.BreakMockMatches != nil { + want = *c.BreakMockMatches + } + if got := MatchRequest(c.Pattern, Request{Method: "POST", URL: "/test", Body: []byte(c.Body)}); got != want { + t.Fatalf("got %v, want %v (Java reference %v)", got, want, c.Matches) } }) } diff --git a/pkg/breakmock/matcher_distance_path.go b/pkg/breakmock/matcher_distance_path.go index 214e383..e8e7e93 100644 --- a/pkg/breakmock/matcher_distance_path.go +++ b/pkg/breakmock/matcher_distance_path.go @@ -31,6 +31,9 @@ func jsonPathMatchDistance(expected any, actual string) float64 { switch v := value.(type) { case nil: case []any: + if _, absent := nested["absent"].(bool); absent && len(v) == 0 && matcherJSONPathDefinite(expression) { + return ValueMatchDistance(nested, "[]", true) + } if _, jsonEqual := nested["equalToJson"]; jsonEqual { values = append(values, matcherJSONValueString(v)) } else { @@ -57,7 +60,7 @@ func jsonPathMatchDistance(expected any, actual string) float64 { } func xpathMatchDistance(expected any, p map[string]any, actual string) float64 { nested, ok := expected.(map[string]any) - if !ok { + if !ok || !xpathHasValueMatcher(nested) { if matchXPath(expected, p, actual) { return 0 } diff --git a/pkg/breakmock/template.go b/pkg/breakmock/template.go index 41103c8..7ec2ee0 100644 --- a/pkg/breakmock/template.go +++ b/pkg/breakmock/template.go @@ -237,7 +237,7 @@ func renderTemplateContext(source string, request Request, parameters map[string }) add("toJson", func(o *raymond.Options) any { if o.Param(0) == nil { - return "" + return "null" } return templatePrettyJSON(templateOmitNulls(o.Param(0)), 0) }) diff --git a/pkg/breakmock/upstream_bug_regression_test.go b/pkg/breakmock/upstream_bug_regression_test.go new file mode 100644 index 0000000..ff89f36 --- /dev/null +++ b/pkg/breakmock/upstream_bug_regression_test.go @@ -0,0 +1,161 @@ +// Copyright 2026 Breaking-IT. Licensed under the Apache License, Version 2.0. +package breakmock + +import ( + "encoding/json" + "net/http/httptest" + "strings" + "testing" +) + +func TestXPathBooleanPredicates(t *testing.T) { + for _, tc := range []struct { + expression string + want bool + }{ + {"false()", false}, {"true()", true}, {"1 = 2", false}, {"count(//planet) = 99", false}, + {"count(//planet) = 2", true}, {"contains('abc','z')", false}, {"boolean(//missing)", false}, + {"//planet", true}, {"//missing", false}, {"0", false}, {"1", true}, {"number('invalid')", false}, {"''", false}, {"'value'", true}, + } { + t.Run(tc.expression, func(t *testing.T) { + got := matchXPath(tc.expression, nil, ``) + if got != tc.want { + t.Fatalf("match %q = %v, want %v", tc.expression, got, tc.want) + } + object := map[string]any{"expression": tc.expression} + if got := matchXPath(object, nil, ``); got != tc.want { + t.Fatalf("object predicate = %v, want %v", got, tc.want) + } + if distance := xpathMatchDistance(object, nil, ``); (distance == 0) != tc.want { + t.Fatalf("object distance %v disagrees with match %v", distance, tc.want) + } + }) + } + if !matchXPath(map[string]any{"expression": "false()", "equalTo": "false"}, nil, "") { + t.Fatal("explicit primitive value matcher must still work") + } +} + +func TestJSONPathAbsentDistinguishesEmptyArray(t *testing.T) { + for _, tc := range []struct { + path, body string + want bool + }{ + {"$.foo", `{}`, true}, {"$.foo", `{"foo":[]}`, false}, {"$.foo", `{"foo":[1]}`, false}, + {"$.foo", `{"foo":{}}`, false}, {"$.foo", `{"foo":""}`, false}, + {"$.foo[*]", `{"foo":[]}`, true}, {"$.foo[0]", `{"foo":[]}`, true}, + {"$.foo[?(@.active==true)]", `{"foo":[{"active":false}]}`, true}, + } { + t.Run(tc.path+tc.body, func(t *testing.T) { + got := matchJSONPath(map[string]any{"expression": tc.path, "absent": true}, tc.body) + if got != tc.want { + t.Fatalf("got %v want %v", got, tc.want) + } + distance := jsonPathMatchDistance(map[string]any{"expression": tc.path, "absent": true}, tc.body) + if (distance == 0) != tc.want { + t.Fatalf("distance %v disagrees with match %v", distance, tc.want) + } + }) + } +} + +func TestToJSONNullIsValidJSON(t *testing.T) { + for _, source := range []string{`{"field": {{toJson missing}} }`, `{"field": {{toJson parameters.value}} }`} { + got, err := RenderTemplate(source, Request{URL: "/"}, map[string]any{"value": nil}) + if err != nil { + t.Fatal(err) + } + if got != `{"field": null }` || !json.Valid([]byte(got)) { + t.Fatalf("invalid null serialization: %q", got) + } + } +} + +func TestAdminDocsTrailingSlash(t *testing.T) { + s, err := NewServer(Options{RootDir: t.TempDir()}) + if err != nil { + t.Fatal(err) + } + defer s.Close() + for _, method := range []string{"GET", "HEAD"} { + a, b := httptest.NewRecorder(), httptest.NewRecorder() + s.ServeHTTP(a, httptest.NewRequest(method, "/__admin/docs", nil)) + s.ServeHTTP(b, httptest.NewRequest(method, "/__admin/docs/", nil)) + if b.Code != 200 || a.Body.String() != b.Body.String() || a.Header().Get("Content-Type") != b.Header().Get("Content-Type") { + t.Fatalf("%s slash route differs: %d", method, b.Code) + } + if method == "GET" && !strings.Contains(b.Body.String(), `href="/__admin/docs/swagger"`) { + t.Fatal("spec link must remain absolute") + } + } +} + +func TestScenarioStateOpenAPI(t *testing.T) { + w := httptest.NewRecorder() + if !serveAdminAsset(w, httptest.NewRequest("GET", "/__admin/docs/swagger", nil)) { + t.Fatal("missing spec") + } + var spec struct { + Paths map[string]map[string]json.RawMessage `json:"paths"` + } + if err := json.Unmarshal(w.Body.Bytes(), &spec); err != nil { + t.Fatal(err) + } + raw := spec.Paths["/__admin/scenarios/{name}/state"]["put"] + if len(raw) == 0 { + t.Fatal("scenario state PUT is missing from OpenAPI") + } + var operation struct { + Responses map[string]any `json:"responses"` + Parameters []struct { + Name string + In string + Required bool + } + } + if err := json.Unmarshal(raw, &operation); err != nil { + t.Fatal(err) + } + for _, status := range []string{"200", "404", "422"} { + if operation.Responses[status] == nil { + t.Errorf("missing documented %s response", status) + } + } + if len(operation.Parameters) != 1 || operation.Parameters[0].Name != "name" || operation.Parameters[0].In != "path" || !operation.Parameters[0].Required { + t.Fatal("missing required scenario name") + } +} + +func TestCorrectedBugsThroughHTTP(t *testing.T) { + s, err := NewServer(Options{RootDir: t.TempDir()}) + if err != nil { + t.Fatal(err) + } + defer s.Close() + mappings := []Mapping{ + {"request": map[string]any{"method": "POST", "url": "/xpath", "bodyPatterns": []any{map[string]any{"matchesXPath": "1 = 2"}}}, "response": map[string]any{"body": "wrong match"}}, + {"request": map[string]any{"method": "POST", "url": "/absent", "bodyPatterns": []any{map[string]any{"matchesJsonPath": map[string]any{"expression": "$.foo", "absent": true}}}}, "response": map[string]any{"body": "missing"}}, + {"request": map[string]any{"method": "GET", "url": "/null"}, "response": map[string]any{"body": `{"field": {{toJson missing}} }`, "transformers": []any{"response-template"}}}, + } + for _, mapping := range mappings { + if _, err := s.Store.AddMapping(mapping); err != nil { + t.Fatal(err) + } + } + for _, tc := range []struct { + method, path, body string + status int + response string + }{ + {"POST", "/xpath", "", 404, ""}, + {"POST", "/absent", `{"foo":[]}`, 404, ""}, + {"POST", "/absent", `{}`, 200, "missing"}, + {"GET", "/null", "", 200, `{"field": null }`}, + } { + w := httptest.NewRecorder() + s.ServeHTTP(w, httptest.NewRequest(tc.method, tc.path, strings.NewReader(tc.body))) + if w.Code != tc.status || (tc.response != "" && w.Body.String() != tc.response) { + t.Errorf("%s %s: got %d %q", tc.method, tc.path, w.Code, w.Body.String()) + } + } +} diff --git a/testdata/matcher_cases.json b/testdata/matcher_cases.json index 3c81ac5..7a02913 100644 --- a/testdata/matcher_cases.json +++ b/testdata/matcher_cases.json @@ -1157,7 +1157,9 @@ ] }, "body": "{\"values\":[\"a\",\"b\"],\"empty\":[],\"nothing\":null}", - "matches": true + "matches": true, + "breakmockMatches": false, + "breakmockCorrection": "WireMock #3159: an existing empty array is not absent" }, { "name": "JSONPath array 99", @@ -4349,7 +4351,9 @@ ] }, "body": "{\"empty\":[],\"obj\":{},\"values\":[null]}", - "matches": true + "matches": true, + "breakmockMatches": false, + "breakmockCorrection": "WireMock #3159: an existing empty array is not absent" }, { "name": "v4 JSONPath empty/null $.objNone", @@ -4477,4 +4481,4 @@ "body": "{\"empty\":[],\"obj\":{},\"values\":[null]}", "matches": false } -] \ No newline at end of file +] diff --git a/testdata/reports/upstream-bug-fixes/validation.json b/testdata/reports/upstream-bug-fixes/validation.json new file mode 100644 index 0000000..3593574 --- /dev/null +++ b/testdata/reports/upstream-bug-fixes/validation.json @@ -0,0 +1,142 @@ +{ + "date": "2026-10-12", + "reference": "WireMock 4.0.0-beta.40", + "checks": { + "go_test_race": "passed", + "go_vet": "passed", + "toxiproxy_integration": "11/11 passed", + "docker_build": "passed" + }, + "full_differential": { + "reference": "4.0.0-beta.40", + "candidate_language": "go", + "candidate_args": [], + "suites": [ + { + "suite": "fixtures_differential.py", + "exit_code": 0 + }, + { + "suite": "differential.py", + "exit_code": 0 + }, + { + "suite": "matcher_differential.py", + "exit_code": 1 + }, + { + "suite": "template_differential.py", + "exit_code": 1 + }, + { + "suite": "admin_differential.py", + "exit_code": 1 + }, + { + "suite": "runtime_differential.py", + "exit_code": 0 + }, + { + "suite": "gzip_differential.py", + "exit_code": 0 + }, + { + "suite": "diagnostics_differential.py", + "exit_code": 1 + }, + { + "suite": "matcher_distance_differential.py", + "exit_code": 0 + }, + { + "suite": "webhook_differential.py", + "exit_code": 0 + }, + { + "suite": "fault_differential.py", + "exit_code": 0 + }, + { + "suite": "entity_differential.py", + "exit_code": 0 + }, + { + "suite": "messaging_differential.py", + "exit_code": 0 + }, + { + "suite": "streaming_differential.py", + "exit_code": 0 + }, + { + "suite": "status_differential.py", + "exit_code": 0 + }, + { + "suite": "streaming_journal_differential.py", + "exit_code": 0 + }, + { + "suite": "message_wait_differential.py", + "exit_code": 0 + }, + { + "suite": "journal_concurrency_differential.py", + "exit_code": 0 + }, + { + "suite": "message_wait_differential.py:disabled-journal", + "exit_code": 0 + } + ] + }, + "final_matcher_rerun": { + "reference": "4.0.0-beta.40", + "candidate_language": "go", + "candidate_args": [], + "suites": [ + { + "suite": "matcher_differential.py", + "exit_code": 1 + }, + { + "suite": "matcher_distance_differential.py", + "exit_code": 0 + } + ] + }, + "mismatches": { + "matcher": { + "passed": 432, + "total": 434, + "strict_mismatches": [ + "JSONPath array 98", + "v4 JSONPath empty/null $.empty{'absent': True}" + ] + }, + "template": { + "passed": 427, + "total": 429, + "strict_mismatches": [ + "toJson missing", + "toJson value null" + ] + }, + "admin": { + "passed": 129, + "total": 130, + "strict_mismatches": [ + "asset docs/swagger" + ] + }, + "diagnostics": { + "passed": 40, + "total": 41, + "strict_mismatches": [ + "binary body mismatch" + ] + } + }, + "comparison_policy": "Live Java comparison scripts unchanged; intentional corrections remain failures, not exemptions.", + "binary_sha256": "d099f879964c2f0f270c557e9abd8d69778c8ac52e3c07d51a6e91fc323b0f0a" +} From 8b3a1f7370d75204a1fcb96a6e23273a162c83f5 Mon Sep 17 00:00:00 2001 From: jgaalen Date: Tue, 13 Oct 2026 09:08:00 +0200 Subject: [PATCH 2/2] Verify intended upstream bug corrections in the Java differential suites The five corrections in this branch made the Java differential suites fail by design, so CI could not pass. List each affected case in testdata/intended_differences.json. For a listed case the suite now requires Java to return its documented upstream result and BreakMock its documented corrected result, both exactly (the OpenAPI document by the SHA-256 values in PROVENANCE.json). All other cases are still compared for equality. A listed case fails if either side changes, so an upstream fix or a lost correction is reported. A build without the corrections fails exactly these five cases; with them, verify.py passes all 20 suite runs. Co-Authored-By: Claude Opus 5.5 --- docs/COMPATIBILITY.md | 8 ++++-- docs/UPSTREAM_BUG_FIXES.md | 26 +++++++++++-------- scripts/admin_differential.py | 7 ++++-- scripts/intended_differences.py | 28 +++++++++++++++++++++ scripts/matcher_differential.py | 7 ++++-- scripts/template_differential.py | 5 +++- testdata/intended_differences.json | 40 ++++++++++++++++++++++++++++++ 7 files changed, 103 insertions(+), 18 deletions(-) create mode 100644 scripts/intended_differences.py create mode 100644 testdata/intended_differences.json diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index deb200b..33fd16c 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -15,8 +15,12 @@ bounded corpus; a passing fixture is not proof of an entire feature's equivalenc BreakMock now corrects five confirmed upstream bugs by default, covering XPath predicates, JSONPath empty-array absence, null JSON serialization, the trailing-slash docs route and the scenario-state OpenAPI operation. See [the corrections and -migration notes](UPSTREAM_BUG_FIXES.md). Strict Java comparisons remain unchanged -and report affected cases as failures; these differences are not byte parity. +migration notes](UPSTREAM_BUG_FIXES.md). These differences are not byte parity. +The affected differential cases are listed in `testdata/intended_differences.json`: +for each, the suite requires Java to still return its documented upstream result +**and** BreakMock to return its documented corrected result, exactly. Every other +case is still compared for equality, and a listed case fails if either side +changes, including if the two sides start to match. ## Implemented and exercised diff --git a/docs/UPSTREAM_BUG_FIXES.md b/docs/UPSTREAM_BUG_FIXES.md index dad0a01..799502b 100644 --- a/docs/UPSTREAM_BUG_FIXES.md +++ b/docs/UPSTREAM_BUG_FIXES.md @@ -23,11 +23,16 @@ Changing the served specification necessarily changes its response bytes. The original upstream asset hash is retained in `internal/adminassets/PROVENANCE.json`, with the modified specification hash recorded separately. -The existing live Java differential scripts are unchanged: they still report -these differences as failures, without filtering cases, normalizing the changed -output, or rewriting the Java reference. The pre-existing JVM byte-array identity -diagnostic also remains a strict mismatch. Therefore strict parity CI is not -expected to be green solely because these correctness fixes pass. +The live Java differential suites pin each affected case in +`testdata/intended_differences.json` (via `scripts/intended_differences.py`). A +listed case passes only when Java returns its documented upstream result and +BreakMock returns the documented corrected result, both exactly: `matched` versus +`unmatched` for #3159, an empty body versus `null` for #3282, and the original +versus modified OpenAPI document (SHA-256 recorded in +`internal/adminassets/PROVENANCE.json`) for #3319. Nothing else is filtered or +normalized, and the Java reference is not rewritten. If upstream fixes a bug, or +BreakMock loses a correction, the case fails. A build without these corrections +fails exactly these five cases. Captured Java matcher fixture results are retained. Two cases now also record an explicit `breakmockMatches: false` expectation for #3159, which Go unit tests @@ -46,9 +51,8 @@ python3 scripts/verify.py --jar /tmp/wiremock.jar --output-dir /tmp/compatibilit ## Validation results Go unit/race tests, vet, all 11 Toxiproxy executable integration checks and the -Docker build pass. The full unchanged Java suite passes 15 of 19 invocations: -matcher has two intentional empty-array differences, templates have two null -serialization differences, admin has the updated specification, and diagnostics -retains the existing JVM-identity mismatch. The final matcher/distance rerun -confirms the same two differences and no distance-suite failures. -See [the validation summary](../testdata/reports/upstream-bug-fixes/validation.json). +Docker build pass. After merging current `main` (2026-10-13), `scripts/verify.py` +passes all 20 suite runs against the pinned JAR, with the five intended +differences above verified on both sides. The earlier run before the registry, +which reported them as failures, is kept in +[the validation summary](../testdata/reports/upstream-bug-fixes/validation.json). diff --git a/scripts/admin_differential.py b/scripts/admin_differential.py index 847da50..164a801 100644 --- a/scripts/admin_differential.py +++ b/scripts/admin_differential.py @@ -5,6 +5,7 @@ Use fresh empty roots for persistence checks. WireMock reference must be 4.0.0-beta.40. """ import argparse,json,re,urllib.request,urllib.error,sys,threading +import intended_differences from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer p=argparse.ArgumentParser();p.add_argument('--reference',default='http://127.0.0.1:18089');p.add_argument('--candidate',default='http://127.0.0.1:18090');p.add_argument('--output');args=p.parse_args() def call(base,method,path,body=None,headers=None): @@ -80,8 +81,10 @@ def step(name,method,path,body=None,headers=None): normalized_wire=[normalize_wire(v[2],v[1],semantic[i]) if path.startswith('/__admin/') and isinstance(v[1],(dict,list)) else v[2] for i,v in enumerate(raw)] semantic_ok=raw[0][0]==raw[1][0] and semantic[0]==semantic[1] ok=raw[0][0]==raw[1][0] and normalized_wire[0]==normalized_wire[1] - results.append({'name':name,'pass':ok,'semantic_pass':semantic_ok,**({} if ok else {'reference':raw[0],'candidate':raw[1],'normalized_reference_body':normalized_wire[0],'normalized_candidate_body':normalized_wire[1]})}) - print(('PASS ' if ok else 'FAIL ')+name);return raw + intended=intended_differences.check('admin_differential.py',name,*[(v[0],v[2] if isinstance(v[2],bytes) else str(v[2]).encode()) for v in raw]) + if intended:ok=semantic_ok=intended[0] + results.append({'name':name,'pass':ok,'semantic_pass':semantic_ok,**({'intended_difference':intended[1]} if intended else {}),**({} if ok else {'reference':raw[0],'candidate':raw[1],'normalized_reference_body':normalized_wire[0],'normalized_candidate_body':normalized_wire[1]})}) + print(('PASS ' if ok else 'FAIL ')+name+(f' [intended difference {intended[1]}]' if intended else ''));return raw step('version JSON','GET','/__admin/version') step('version text','GET','/__admin/version',headers={'Accept':'text/plain'}) step('health','GET','/__admin/health') diff --git a/scripts/intended_differences.py b/scripts/intended_differences.py new file mode 100644 index 0000000..fbcd936 --- /dev/null +++ b/scripts/intended_differences.py @@ -0,0 +1,28 @@ +# Copyright 2026 Breaking-IT. Apache-2.0. +"""Intended differences from the pinned WireMock reference (upstream bug fixes). + +A registered case passes only when the reference returns its documented upstream +result and BreakMock returns its documented corrected result, both exactly. This +replaces equality for those cases alone; it never filters or normalizes others. +""" +import hashlib, json, pathlib + +_CASES = {(c['suite'], c['name']): c for c in json.loads( + (pathlib.Path(__file__).resolve().parent.parent / 'testdata/intended_differences.json').read_text())['cases']} + + +def _matches(expected, status, body): + if status != expected['status']: + return False + if 'sha256' in expected: + return hashlib.sha256(body).hexdigest() == expected['sha256'] + return body == expected['body'].encode() + + +def check(suite, name, reference, candidate): + """Return None when unregistered, else (passed, issue). Arguments are (status, body bytes).""" + case = _CASES.get((suite, name)) + if case is None: + return None + passed = _matches(case['reference'], *reference) and _matches(case['candidate'], *candidate) + return passed, case['issue'] diff --git a/scripts/matcher_differential.py b/scripts/matcher_differential.py index 5433fe3..a98eae9 100644 --- a/scripts/matcher_differential.py +++ b/scripts/matcher_differential.py @@ -4,6 +4,7 @@ compare generated unmatched-request diagnostic bodies or prove full parity. """ import argparse, json, urllib.request, urllib.error +import intended_differences p=argparse.ArgumentParser();p.add_argument('--reference',default='http://127.0.0.1:18085');p.add_argument('--candidate',default='http://127.0.0.1:18083');p.add_argument('--output');args=p.parse_args() def call(base,path,body=None,headers=None,method=None): if isinstance(body,dict):body=json.dumps(body).encode();headers={'Content-Type':'application/json',**(headers or {})} @@ -85,8 +86,10 @@ def add(name,pattern,body=None,path='/test',headers=None,method=None):cases.appe if status!=201:responses.append(('rejected',status,data));continue status,data=call(base,path,body,headers,method);responses.append((status,data)) ok=responses[0]==responses[1] - results.append({'name':name,'passed':ok,'reference':repr(responses[0]),'candidate':repr(responses[1]),'request':pattern,'body':body,'path':path}) - print(('PASS ' if ok else 'FAIL ')+name) + intended=intended_differences.check('matcher_differential.py',name,*responses) if all(len(r)==2 for r in responses) else None + if intended:ok=intended[0] + results.append({'name':name,'passed':ok,'reference':repr(responses[0]),'candidate':repr(responses[1]),'request':pattern,'body':body,'path':path,**({'intended_difference':intended[1]} if intended else {})}) + print(('PASS ' if ok else 'FAIL ')+name+(f' [intended difference {intended[1]}]' if intended else '')) if not ok:failed.append((name,responses));print(responses) print(f'{len(cases)-len(failed)}/{len(cases)} matched reference') if args.output: diff --git a/scripts/template_differential.py b/scripts/template_differential.py index 7cbc499..0603c89 100644 --- a/scripts/template_differential.py +++ b/scripts/template_differential.py @@ -17,6 +17,7 @@ def call(base,path,data=None,method=None,headers=None): except urllib.error.HTTPError as e:r=e return r.status,r.read() from oracle import identify +import intended_differences identify(a.reference,'java'); identify(a.candidate,'go') cases=[] parameters_by_name={} @@ -95,7 +96,9 @@ def add(name,template,body='',path='/template-test?q=one&q=two',headers=None,par diagnostic = name in ('jsonBody template quotes','jsonBody template newline') and results[0][0]==500 comparison='status_and_exact_body_bytes' ok=results[0]==results[1] - print(('PASS ' if ok else 'FAIL ')+name+(' [JVM diagnostic: strict bytes required]' if diagnostic else '')) + intended=intended_differences.check('template_differential.py',name,*results) if all(len(r)==2 for r in results) else None + if intended:ok=intended[0];comparison='intended_difference '+intended[1] + print(('PASS ' if ok else 'FAIL ')+name+(' [JVM diagnostic: strict bytes required]' if diagnostic else '')+(f' [intended difference {intended[1]}]' if intended else '')) record={'name':name,'passed':ok,'comparison':comparison,'reference_status':results[0][0],'candidate_status':results[1][0],'reference_body':results[0][-1][:1200].decode(errors='replace'),'candidate_body':results[1][-1][:1200].decode(errors='replace'),'reference_body_bytes':len(results[0][-1]),'candidate_body_bytes':len(results[1][-1])} records.append(record) if not ok:failed.append(name);print(' reference:',repr(results[0][-1][:300]),'\n candidate:',repr(results[1][-1][:300])) diff --git a/testdata/intended_differences.json b/testdata/intended_differences.json new file mode 100644 index 0000000..41cc487 --- /dev/null +++ b/testdata/intended_differences.json @@ -0,0 +1,40 @@ +{ + "description": "Intended differences from WireMock 4.0.0-beta.40, documented in docs/UPSTREAM_BUG_FIXES.md. For each listed case the differential suites require the reference to still return its documented upstream result AND BreakMock to return its documented corrected result, exactly. Any other result fails, including the two sides matching.", + "cases": [ + { + "suite": "matcher_differential.py", + "name": "JSONPath array 98", + "issue": "https://github.com/wiremock/wiremock/issues/3159", + "reference": {"status": 200, "body": "matched"}, + "candidate": {"status": 200, "body": "unmatched"} + }, + { + "suite": "matcher_differential.py", + "name": "v4 JSONPath empty/null $.empty{'absent': True}", + "issue": "https://github.com/wiremock/wiremock/issues/3159", + "reference": {"status": 200, "body": "matched"}, + "candidate": {"status": 200, "body": "unmatched"} + }, + { + "suite": "template_differential.py", + "name": "toJson missing", + "issue": "https://github.com/wiremock/wiremock/issues/3282", + "reference": {"status": 200, "body": ""}, + "candidate": {"status": 200, "body": "null"} + }, + { + "suite": "template_differential.py", + "name": "toJson value null", + "issue": "https://github.com/wiremock/wiremock/issues/3282", + "reference": {"status": 200, "body": ""}, + "candidate": {"status": 200, "body": "null"} + }, + { + "suite": "admin_differential.py", + "name": "asset docs/swagger", + "issue": "https://github.com/wiremock/wiremock/issues/3319", + "reference": {"status": 200, "sha256": "4306cf1dada9455ad6b41e1344552f0dd6980edb3cf2471395a14dd8fee64d97"}, + "candidate": {"status": 200, "sha256": "e66006261d3dd75b25f33e9b2c92d7dced6020221492a6e151eecd35c7d74f0c"} + } + ] +}