Skip to content

Speed up large JSON responses and refresh Java/Go benchmarks - #2

Merged
jgaalen merged 5 commits into
mainfrom
claude/wiremock-go-perf-benchmark-0b071b
Oct 13, 2026
Merged

jgaalen merged 5 commits into
mainfrom
claude/wiremock-go-perf-benchmark-0b071b

Conversation

@jgaalen

@jgaalen jgaalen commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Benchmarking ~100 KiB JSON responses exposed per-request costs that the 17-byte benchmark workloads hid. This PR removes them, fixes a header-decoding incompatibility found while verifying, and refreshes the Java-vs-Go benchmark results.

Against WireMock 4.0.0-beta.40 (same host and session; all results measured from a clean checkout of 68cd311, recorded in each report):

Throughput Server CPU Memory
Small responses (17-byte body, small template) +22–50% −23–34% at 10k RPS −90–92% (28–43 vs 259–490 MiB)
~100 KiB JSON (inline, file, templates, gzip) 1.3–10.6× −54–96% per request −91–99% peak RSS
~1 MiB JSON 3.1–86× −85–99% per request 60–145 MiB vs 3.0–5.0 GiB (journal off)
~10 MiB JSON (journal disabled) 3.0–208× −92–99.5% per request 0.4–0.9 GiB vs 11.1–12.4 GiB

The lead grows with body size: Java's per-request CPU rises with the body, while BreakMock serves cached bytes and renders templates in one pass. At 10k RPS, small-response p99 latency was not lower for Go (2.5–3.8 ms vs Java's 2.0–6.1 ms).

Before this PR, BreakMock was 30–51× slower than Java for large templated bodies.

Changes

  • Templates ≥ 64 KiB were never cached and were re-lexed per request (~14 ms per 100 KiB). Caches are now bounded by total size (64 MiB) instead of per-template size.
  • Helper detection scanned the body for ~70 helper names on every render; each cached template now records which names it mentions.
  • jsonBody is served from its stored wire form instead of re-validated and re-encoded per request.
  • Gzip output for non-templated bodies is cached by content (a changed body gives a new key).
  • Body files are cached and revalidated per request by inode, size, mtime and ctime (ctime catches same-size in-place rewrites with a restored mtime); misses still read through os.Root. Not cached on platforms without ctime.
  • Journal keeps one copy of each response body and encodes body/bodyAsBase64 when read, instead of copying 2.3× the body per request.
  • Copies removed from entity text conversion and the Handlebars evaluator (full-evaluator garbage per request 784 KB → 123 KB).
  • Gzip cache covers bodies up to 16 MiB (64 MiB budget), so large static bodies are compressed once.
  • Binary-mismatch diagnostic now renders like WireMock: Java's default byte[] string, [B@ plus a random identity hash (previously base64). diagnostics_differential.py masks only that token, width-for-width, requires one on each side and compares every other byte; builds printing anything else still fail. This was the last failing compatibility case; CI's verify.py steps now pass.
  • Header bytes decoded as ISO-8859-1, as Jetty does. Previously a header containing byte E9 and one containing UTF-8 é matched different stubs on Java and Go; matching, templates, cookies and journal now agree with WireMock.

Small-response performance is unchanged (in-process A/B within ±2%, BenchmarkSmallResponse).

Verification

  • scripts/verify.py against the pinned JAR: all 20 suite runs pass (diagnostics 41/41 in three consecutive runs).
  • New scripts/dynamic_template_differential.py (registered in verify.py), 238/238 vs Java at normal and >64 KiB sizes: per-request values, random UUID/string/int/now/pickRandom freshness, mapping/file/jsonBody edits with and without gzip, raw non-ASCII headers, and journal bodies under 120 concurrent requests. Original and optimized builds failed exactly the same checks before the header fix, so no regression was found.
  • template_differential.py --pad 80000 --repeat 3: all 429 template cases inside 80 KB templates, byte-identical to Java.
  • Large-JSON benchmark: every response from both Go builds byte-identical to Java's, gzip included.
  • go test -race ./..., go vet ./..., and the cgo-free build's tests pass. New unit tests cover large-template responses (including concurrency), journal body output, body-file cache freshness/symlink escape (fails if the ctime check is removed), and header decoding.

Known gaps (pre-existing, documented in docs/COMPATIBILITY.md)

  • Journal responseDefinition holds the template source where Java logs the rendered body; gzip responses are journaled compressed where Java logs them uncompressed. Being fixed in separate branches.
  • Template syntax errors: Java renders some malformed templates as literal text and formats errors differently. Queued separately.
  • Intentional divergence: Java keeps serving a cached template after a templated body file is edited; BreakMock serves the edited file.

Docs and data

  • README benchmark section rewritten with the new results; docs/LARGE_JSON_OPTIMIZATION.md has the full report.
  • Raw data: testdata/reports/java-go-2026-10-13/, testdata/reports/benchmark-small-ab.txt and testdata/reports/benchmark-large-json{,-1mib,-10mib}.json (with binary and JAR hashes).

🤖 Generated with Claude Code

jgaalen and others added 5 commits October 12, 2026 22:11
Large response bodies exposed per-request costs that small benchmarks hid:
- Cache templates by total size instead of skipping sources over 64 KiB,
  which were re-lexed on every request (~14 ms per 100 KiB).
- Record which helper names a cached template mentions instead of
  scanning the body for ~70 names per render.
- Serve stored jsonBody bytes directly; cache gzip output of static
  bodies by content; cache body files, validated per request by inode,
  size, mtime and ctime; encode journal response bodies on read.
- Remove whole-body copies in entity text and the Handlebars evaluator.

Decode request header bytes as ISO-8859-1, as Jetty does, so matching,
templates and the journal agree with WireMock for non-ASCII headers.

Add json_benchmark.py (any builds or the WireMock JAR, byte-checked),
dynamic_template_differential.py (per-request values, random helpers,
mapping/file edits, journal integrity; registered in verify.py) and a
--pad/--repeat mode for template_differential.py.

Against WireMock 4.0.0-beta.40: small responses 14-33% more throughput,
18-36% less CPU at 10k RPS, ~90% less memory; ~100 KiB JSON 1.6-11.8x
throughput and 55-96% less CPU, with byte-identical responses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extend the Java comparison to ~1 MiB and ~10 MiB JSON bodies. BreakMock's
lead grows with size: 3.2-79x throughput at 1 MiB and 3.2-180x at 10 MiB,
with 86-99.6% less CPU per request. Raise the gzip cache limit to 16 MiB
bodies (64 MiB budget) so large static bodies are not recompressed.

Render a binary-body mismatch the way WireMock does, as Java's default
byte[] string "[B@" plus a random identity hash. The diagnostics suite
masks only that token, width-for-width, requires it on both sides and
compares every other byte. Earlier builds that printed base64 still fail.
verify.py now passes all suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README's small-response results came from a build made before the last
commits, while changes were uncommitted, so their reports recorded the old
base revision. Rerun every published benchmark (small responses, ~100 KiB,
~1 MiB and ~10 MiB JSON) from a clean checkout of 68cd311. Each report now
records that commit, a clean tree and the binary hash. Regenerate the
README and large JSON report tables from the new data, and add the
in-process small-response A/B output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A message action writes its reply frame before the handler journals the
"sent" and "received" events, so a client reading /__admin/messages right
after the reply could see neither. This made the streaming journal
differential flaky on CI ("websocket matched received and sent journal").

Incoming message handling now holds a read lock, and journal reads and
clears briefly take the write lock, so they wait for in-flight handling.
Event order and error handling are unchanged. A deterministic test reads
the journal from inside the reply send; without the barrier it sees an
empty journal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jgaalen
jgaalen merged commit 92fac52 into main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant