Skip to content

Agent MCP bridge for coding harnesses, and faster AI Repair - #289

Merged
jgaalen merged 22 commits into
mainfrom
claude/breaktest-mcp-harness-7e5c84
Nov 10, 2026
Merged

jgaalen merged 22 commits into
mainfrom
claude/breaktest-mcp-harness-7e5c84

Conversation

@jgaalen

@jgaalen jgaalen commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Lets coding harnesses (Claude Code, Codex, OpenCode, Pi and others) read and change the plan open in BreakTest through MCP, see changes appear live, run tests and read real traffic. AI Repair is rebuilt on the same tools and is much faster.

Bridge and MCP server

  • Standard MCP framing. The stdio server accepts line-delimited JSON, as MCP clients send it, next to the old Content-Length framing. Before this, a standard client could not connect.
  • Fewer, smaller calls.
    • batch_open_plan runs several tools in one call; a later step can use an earlier step's node id ("@name").
    • A batch step with forEach runs its tool on every node matching a filter (type, class, name, below a node, static assets) as one undo step, with per-node templates such as {{node.name}} or {{node.path}}. Nodes inside a disabled parent, such as a kept repair copy, are skipped.
  • Reading.
    • get_plan_outline: compact tree.
    • get_element_open_plan: a node's settings, table rows and children; settings at their default are listed by name and type.
    • list_variables and list_functions.
    • String "...Mode" settings show their allowed values.
  • Editing.
    • add_http_request builds a request from a URL.
    • set_enabled_open_plan, export_subtree / import_subtree, undo, redo, select_node.
    • rawProperties paths reach nested elements and list rows; rawLists fills lists without typed tables.
    • tableMode replace, append, upsert or remove for table rows. Request headers are a table on HTTP requests.
  • Plan and scenarios. save_open_plan, open_plan, new_plan, add_scenario, add_profile, set_scenario_workload (threads, ramp-up, duration, loops, model, schedules, pacing, profile).
  • Running. start_test (with a confirmation dialog unless breaktest.agent.confirm_run=false), stop_test, test_status, test_results (per-label counts, errors, percentiles, error samples) and get_log.
  • Real traffic. list_samples / get_sample return the full request, headers and response of a validated sample or of a run, in windows.
  • Validation summary. validate_open_plan summary=true returns a few kilobytes: one row per request, the first failure with short heads, ranked repair actions, and on a green run response previews per transaction. A repeat of an unchanged green run within the same AI run only says so.
  • Instances and headless. list_instances / select_instance choose between open windows. open_plan_headless runs the plan tools on a file without a window.
  • Control.
    • breaktest.agent.permissions=read|edit|run limits what an agent may do.
    • Results carry planChangedSinceLastCall when the user edited the plan in between.
    • BREAKTEST_MCP_TOOLSETS=core advertises a smaller set; describe_tools lists the rest.
  • One declaration per new tool in AgentToolSpecs, used by both the server and the GUI service.

AI Repair

  • File mode removed: the edit-surface radio buttons, file-backed prompt, repair target and summary sidecar, the reload after a run, refresh_open_plan_from_file and breaktest.ai.edit_surface.
  • Thread group copy and Revert. "Keep a copy of the thread group" (ticked by default, breaktest.ai.keep_thread_group_copy) places a disabled copy before the run; the repair works on the original, so scenarios keep pointing at it. One copy per thread group. A Revert button in the AI log puts the whole plan back as it was before the last run.
  • Variables go to a User Defined Variables element under the Shared Profile.
  • Analysis packet.
    • The initial correlation analysis now reaches the agent; it failed on the dialog's thread group path.
    • It works on unsaved plans, using the recording held in memory.
    • It marks each proposal apply, optional, review or skip. Cache validators such as If-None-Match are optional, so their sources stay in their Parallel Controllers.
    • It is about 37% smaller.
  • Prompt.
    • Tool signatures are inlined.
    • WebSocket and generation guidance is only sent when relevant.
    • The launcher announces the start.
    • Bridge arguments can come from standard input.
    • Focused edits follow a short path: outline, one read, one batch, report.
    • Codex is no longer pointed at the installed breaktest-jmeter-repair skill, which described the old workflow.
  • Naming. Planner variable names are told apart by the consuming request (csrf_token_payments), and the thread group path is no longer doubled.

Fixes

  • Restoring the pre-run backup named the backup as the plan's file, so the next Save would have overwritten the backup.
  • list_http_arguments_open_plan and set_http_argument_value_open_plan saw no rows.
  • Scoped search_open_plan_values returned object ids instead of node paths.
  • Node paths doubled "Test Plan" for plans loaded from a file.
  • excludeUserDefinedVariables only skipped the Test Plan table.
  • Several GUI tools were logged twice, at the cost of extra socket calls.

Test stability (not caused by this branch)

  • HTTPHC5H2ThreadLifecycleTest: accepts the ClosedSelectorException from the fallback close only when the stop reached the five-second reactor wait.
  • WebSocketPayloadEditorTest: the large-payload case has a ten-minute limit (135 s on a macOS stress-JIT runner).
  • ParameterCompletionTableTest: waits for settled editor focus under Xvfb.

Motivation and Context

Harnesses should be able to work on a running BreakTest directly: inspect the plan, edit it live, validate, read real requests and responses, and run tests. AI Repair runs on the same bridge.

On the demo checkout plan with Pi and DeepSeek v4.1 flash, a full AI Repair went from 46 model turns, 2.4M input tokens and 2m51s to 8 turns, 118k tokens and 34s, with the same green result.

How Has This Been Tested?

  • ./gradlew autostyleCheck, checkstyle, and compileTestJava compileTestKotlin for all modules.
  • :src:core:test for org.apache.jmeter.ai.*, *AiAutoScripting* and *AiPrompt*; :src:dist-check:test for *AgentElement*, including AgentElementCoverageTest, which creates all 139 installed elements and pins the lists only rawLists can fill. The full suite runs in CI; all 11 checks pass.
  • Live through the MCP server against a separate BreakTest window and headless: batch build and validate, save, scenario workload, a real start/status/stop, test_results, sample reading, text round-trip, undo, open plan, instance selection, and a for-each on a 78-sampler plan copy (header upsert on 56 requests, think times on 8 transactions, 14 static assets disabled, in one call).
  • Several AI Repair runs by the maintainer with Pi, OpenCode and Codex, including the thread group copy and Revert.

Not exercised:

  • The for-each step and table modes through an AI edit request.
  • The start-test confirmation dialog and the permission setting in a live window (unit-tested only).
  • The overhead of the live-results hook in JMeterThread under heavy load. It is only active while the GUI agent service is on.

Things a reviewer should know

  • Breaking: file mode and refresh_open_plan_from_file are removed.
  • Engine hook: JMeterThread calls AgentRunStats for every sample while the agent service is on.
  • Node ids: undo and redo rebuild the tree, so node ids reset; the result says so and paths still resolve.
  • Docs: xdocs and the changelog are not updated. bin/jmeter.properties, the settings catalog and the AI help text are.
  • Not covered by tools: HAR import, recorder control and HTML report generation.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

  • My code follows the existing style of this project.
  • I have updated the documentation accordingly.
  • I have run the relevant tests, or explained why they were not run.
  • I have read and accept the Contributor License Agreement for this contribution.

🤖 Generated with Claude Code

audit and others added 8 commits November 9, 2026 23:31
A coding agent needed about 25 round trips for a small plan edit. Add
general tools so one call can do the work:

- batch_open_plan runs several tools in order; later steps can reference
  earlier results with @name
- get_plan_outline and get_element_open_plan give compact reads
- set_enabled_open_plan, save_open_plan, set_scenario_workload
- start_test, stop_test, test_status; start asks for confirmation in the
  GUI unless breaktest.agent.confirm_run=false, and reports scenario
  errors as text instead of blocking on a dialog
- rawProperties on add/configure for settings the schema does not expose
- BREAKTEST_MCP_TOOLSETS=core advertises only the general tools;
  describe_tools lists the rest

The new tools are declared once in AgentToolSpecs and used by both the
MCP server and the GUI service.

Fixes:
- accept line-delimited JSON on stdio, as MCP clients send it, next to
  Content-Length framing
- HTTP argument listing and editing saw no rows
- scoped value search returned object ids instead of node paths
- node paths doubled "Test Plan" for plans loaded from a file
- type filters matched on package name
- several GUI tools were logged twice at the cost of extra socket calls

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… coverage

Run and observe
- test_results: per-request and per-transaction counts, errors, average,
  approximate p90/p95, max and rate, plus error kinds and first failing
  samples; fed by AgentRunStats from JMeterThread sample delivery, only
  while the GUI agent service is on
- get_log: filtered tail of the BreakTest log
- start_test can run a named scenario

Edit faster
- add_http_request builds a request from a URL
- export_subtree / import_subtree move a subtree through JMX text
- batch_open_plan returns brief results for edit steps
- undo, redo, select_node

Reach more
- rawProperties paths reach nested elements and list rows; rawLists
  replaces the rows of lists without typed table support
- request headers are a table on HTTP requests
- add_scenario, add_profile, more workload settings (model, schedules,
  delay, pacing, profile)
- open_plan, new_plan
- list_functions, list_variables
- list_instances / select_instance choose between running windows
- open_plan_headless runs the plan tools on a file without a window

Control and efficiency
- breaktest.agent.permissions=read|edit|run limits what an agent may do
- results carry planChangedSinceLastCall when the user edited the tree
- the core toolset shortens long descriptions

AgentElementCoverageTest creates every installed element and pins the
lists that only rawLists can fill.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- open_plan_headless no longer needs a window to set the plan file, and
  save writes the file directly in that mode
- the MCP server and CLI exit when done even after Swing was started
- undo and redo report that node ids were reset
- start_test, open_plan and new_plan explain that they need a window
- document breaktest.agent.permissions and breaktest.agent.confirm_run

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- list_samples lists the samples of the last validation run or of the
  current/last real run
- get_sample returns the request line and body, request headers,
  response headers and response body of one sample, with failed
  assertions and sub-samples; long bodies come in windows by offset and
  limit, or starting at a searched text
- real runs keep the full first passing sample and the first two
  failures of each request (up to 200), so reading traffic does not
  copy every sample under load

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The live repair and specific-request prompts named only the older
tools. Tell the agent, at the step where each applies, about:
- get_sample / list_samples for full request and response evidence
- batch_open_plan for edits without a dedicated batch tool
- get_plan_outline / get_element_open_plan for cheap re-inspection
- list_variables and list_functions
- add_http_request, rawProperties and rawLists when generating flows

The file-backed repair prompt is unchanged: it validates a file, and
get_sample reads the samples of a live validation.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
set_user_defined_variable_open_plan wrote into the Test Plan's own
variables table. It now uses a User Defined Variables element under the
Shared Profile, created on first use, so the variables reach every
Thread Group and scenario and are visible as their own node. A value an
earlier run left on the Test Plan moves along; a plan without a Shared
Profile keeps the old location. Merging variables from a repaired clone
follows the same rule.

excludeUserDefinedVariables now skips every standalone User Defined
Variables element in searches and replacements, not only the Test Plan
table, so a credential variable keeps its original value there too.

The file-backed repair prompt tells the agent to use the same place,
and tool descriptions no longer say "top-level".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
File mode
- AI Repair always edits the open plan through the bridge. The edit
  surface radio buttons, the file-backed prompt, the repair target and
  summary sidecar, the reload after a run and the
  refresh_open_plan_from_file tool are removed, with the
  breaktest.ai.edit_surface property.

Thread group copy
- The start dialog has "Keep a copy of the thread group", ticked by
  default (breaktest.ai.keep_thread_group_copy). Before the run a
  disabled copy named "<name> (before AI repair)" is placed after the
  thread group. The agent works on the original, which keeps its name
  and id, so scenarios keep pointing at the repaired version. The copy
  gets its own id, replaces a copy kept by an earlier run, and is not
  listed as an agent change. The prompts tell the agent to leave it
  alone.

Revert
- The AI log has a Revert button that, after confirmation, puts the
  whole plan back as it was before the last run: the thread group, its
  copy and anything else, such as shared variables.
- Restoring the pre-run backup now inserts the tree directly. Opening
  the backup as a file named the backup as the plan's file, so the next
  Save would have overwritten the backup instead of the plan.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Both failed on main and on other pull requests before this branch.

HTTP/2 reactor lifecycle tests
- A stopping user waits five seconds for its HTTP/2 reactor to end and
  then closes the client anyway; closing it while the reactor still
  runs makes HttpCore log a ClosedSelectorException, which
  stoppingActiveRequestsDoesNotLogWarningsOrReactorErrors rejects. The
  failing CI runs took just over five seconds, and closing without the
  wait reproduces the exact message locally.
- The wait is now httpsampler.http2.reactor_shutdown_wait_ms (default
  unchanged, 5000). The test raises it, since it checks the orderly
  path, not the fallback.

WebSocket payload editor test
- Laying out 400 KB of wrapped text took 135 s on a macOS stress-JIT
  runner against 2-4 s locally, over the default two-minute limit. The
  test gets its own ten-minute limit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@jgaalen
jgaalen force-pushed the claude/breaktest-mcp-harness-7e5c84 branch from 1ecc4e1 to 27a0273 Compare November 10, 2026 08:11
audit and others added 14 commits November 10, 2026 09:27
… raising the wait

The previous commit made the reactor shutdown wait configurable and
raised it in stoppingActiveRequestsDoesNotLogWarningsOrReactorErrors.
On the macOS stress-JIT runner the reactor then still had not ended
after ten seconds, so the test failed on "stopped user must finish
promptly" instead. A longer wait does not help there.

- Remove the httpsampler.http2.reactor_shutdown_wait_ms property again;
  product code is back to what it was.
- The test measures how long the stop took. When it reached the
  five-second reactor wait, the ClosedSelectorException that the
  fallback close makes HttpCore log is accepted. A stop that stayed
  within the wait must still be silent.

Why the reactor does not end in time on that runner is not addressed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The start dialog names the selected thread group by its full tree path,
hidden root included ("Test Plan / Test Plan / ..."). The service
compared that with the visible path only, so the launcher's analysis
threw, the log said "Initial correlation analysis unavailable", and the
agent spent its first turns inspecting the plan itself. This already
happened for plans that were never saved; the node path fix earlier on
this branch extended it to plans loaded from a file.

- selectThreadGroup also accepts that full path
- the log line now carries the reason, and the failure is logged
- an unknown name in a tool schema request is named in the error

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…on unsaved plans

A repair run took 46 model turns and 2.4M input tokens, mostly because
large tool results were re-sent on every turn and the agent sliced them
with scripts.

- validate_open_plan summary=true returns a few kilobytes: one row per
  reached request, the first failure with short request/response heads,
  ranked repair actions and request candidates, and on a green run a
  response preview per transaction (or per request outside one) for
  assertions. Details stay available through get_sample. On a small
  real plan a failing run went from 8.3 KB to 1.2 KB.
- Recording tools and the repair planner no longer refuse a plan that
  was never saved: an imported recording is held in memory. The planner
  used to skip recording evidence entirely for such plans, which left
  the agent to find correlations by trial and error.
- The bridge's tool list without names returns names and one-line
  summaries (11 KB instead of 53 KB).
- A correlation batch item without a literal now says which item, why,
  and what to call to only add an extractor.
- The repair and specific-request prompts ask for the summary, tell the
  agent to read it directly, and to take assertion markers from the
  previews instead of reading every sample.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…turns

Follow-up to a run that took 16 model turns, 475k input tokens and 82 s,
of which one 20 s step was the agent classifying the analysis packet.

Analysis packet
- Each proposal carries a recommendation with why: apply, optional,
  review or skip. Cache validators (If-None-Match and similar) are
  optional: a stale value only turns a 304 into a 200, while applying
  one usually moves its source out of a Parallel Controller. Response
  echoes are skip, overlapping tokens review.

Prompt
- WebSocket guidance is sent only for a thread group with WebSocket
  samplers, generation guidance only when the user gave instructions.
  The repair template goes from 21.9 KB to 14.4 KB.
- The signatures of the tools the workflow names are inlined (1.6 KB),
  so no turn is spent on reading schemas.
- The start is announced by the launcher; the agent no longer spends
  its first call on it.

Bridge
- "-" reads the arguments from standard input, so multi-item JSON is
  one shell call with a heredoc instead of write-file-then-call.

Validation summary
- A green run that repeats the previous green run only says so.
- Assertion previews are 400 characters and prefer JSON answers over
  page shells.
- The summary carries the agent change counts, replacing the separate
  change-list call before the report.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The "unchanged since previous run" shortcut never triggered: it
  compared the sample rows in order, and requests in a Parallel
  Controller finish in a different order on every run. Outcomes are now
  counted per label, by pass or fail.
- The analysis packet lists optional and skipped proposals without their
  evidence, drops the repeated ordering reason and alternative-source
  evidence, and gives the node list as text rows. On the demo checkout
  plan it goes from 21.1 KB to 13.3 KB.
- A second variable for the same field is told apart by the request that
  uses it (csrf_token_payments) before falling back to a number, and the
  "x_" prefix of header names is dropped.
- The start dialog no longer names the hidden tree root, so the thread
  group path is not doubled in prompts and reports.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s optional

A repair run lost a turn on get_sample rejecting a part name: the result
calls the request line and body "request", but the argument had to be
"requestBody". Aliases (request, response, body, headers) and a single
comma-separated string are accepted, and an unknown part is named in the
error.

The dynamic audit reported the stale If-None-Match value as an
opaque-token candidate, which the agent then investigated although the
repair packet had already marked it optional. The compact audit now
labels such values cache-validator, optional, no action needed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The "unchanged since previous run" shortcut compared against the last
validation of the whole session. A new AI Repair run therefore got
"unchanged" on its very first validation, with no request rows and no
response previews, and the agent fetched every sample one by one to
choose assertions.

The shortcut now compares only against the last summary handed out, and
that baseline is cleared when an AI run starts and when the plan is
replaced (open, new, headless open, restore from backup). get_sample and
the cached responses are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI Repair told Codex to use an installed breaktest-jmeter-repair skill.
The installed skill describes the old workflow (compact validation,
inspect_open_plan, file-backed mode) and is about 60 KB, so it
contradicted the current prompt and enlarged every Codex request. The
prompt is self-contained now; Codex gets the same bootstrap as the other
tools, which tells it not to look for or invoke that skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… discovery

A request to set the think time of eight transactions took eight model
turns. Two went to list_available_elements and describe_element, only to
learn the delay setting names that get_element_open_plan left out because
they were at their default; three went to selecting, re-reading and
listing changes after a batch that already confirmed every edit.

- get_element_open_plan lists the other settable settings with their
  types (otherSettings), so one read is enough to change any of them.
- String "...Mode" settings expose their allowed values from the class's
  constants, e.g. delayMode Disabled|Fixed|Random|Gaussian Random.
- The specific-request prompt describes the short path for a focused
  edit (outline, one read, one batch, report), keeps element discovery
  for adding new element types, and drops the closing inspection and
  change list for edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Edit requests of the "change X on every Y" kind needed one batch step
per node, hit the 100-step limit on larger plans, and could only rewrite
whole tables, so adding one header meant reading and rewriting every
request's headers.

- A batch step with forEach runs its tool once for every node matching a
  filter (type, class, name, below a node, static assets include,
  exclude or only), as one undo step. The node is the target unless the
  arguments name one; "@node" and {{node.name}}, {{node.treePath}},
  {{node.parentName}} or {{node.<setting>}} fill in per-node values.
  Nodes inside a disabled ancestor, such as a kept repair copy, are
  skipped unless includeDisabled is set. The result is counts, failures
  and, for read tools, the per-node results.
- configure_element_open_plan and add_element_open_plan take tableMode:
  replace (default), append, upsert by name, or remove by name.
- The edit prompt tells the agent to use forEach for such requests, to
  prefer one shared element (Header Manager, HTTP Request Defaults) when
  it gives the same result, and the table modes.

On a copy of a 78-sampler checkout plan, one call upserted a header on
56 requests, set think times on 8 transactions and disabled 14 static
assets, returning 884 characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding the tableMode parameter with a Kotlin default removed the
three-argument signature that WebSocketSamplerTest (Java) calls, so the
protocol module's tests no longer compiled and every CI job failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Ubuntu Xvfb job failed in focusingExistingExpressionPreservesText-
AndNormalKeys: the editor's first focusGained had fired, but by the next
EDT turn it no longer owned focus. The sibling test failed the same way
on an earlier run. awaitEditorFocus now waits until the editor actually
owns focus, and on failure says which component took it.

Not related to the agent changes in this branch; the test was not
touched and passes locally on macOS three times in a row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jgaalen jgaalen changed the title Agent MCP bridge: batch edits, live results, sample reading, and AI Repair copy/revert Agent MCP bridge for coding harnesses, and faster AI Repair Nov 10, 2026
@jgaalen
jgaalen merged commit 6564597 into main Nov 10, 2026
18 of 19 checks passed
@jgaalen
jgaalen deleted the claude/breaktest-mcp-harness-7e5c84 branch November 10, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant