Repository navigation
Agent MCP bridge for coding harnesses, and faster AI Repair - #289
Merged
Merged
Conversation
A coding agent needed about 25 round trips for a small plan edit. Add general tools so one call can do the work: - batch_open_plan runs several tools in order; later steps can reference earlier results with @name - get_plan_outline and get_element_open_plan give compact reads - set_enabled_open_plan, save_open_plan, set_scenario_workload - start_test, stop_test, test_status; start asks for confirmation in the GUI unless breaktest.agent.confirm_run=false, and reports scenario errors as text instead of blocking on a dialog - rawProperties on add/configure for settings the schema does not expose - BREAKTEST_MCP_TOOLSETS=core advertises only the general tools; describe_tools lists the rest The new tools are declared once in AgentToolSpecs and used by both the MCP server and the GUI service. Fixes: - accept line-delimited JSON on stdio, as MCP clients send it, next to Content-Length framing - HTTP argument listing and editing saw no rows - scoped value search returned object ids instead of node paths - node paths doubled "Test Plan" for plans loaded from a file - type filters matched on package name - several GUI tools were logged twice at the cost of extra socket calls Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… coverage Run and observe - test_results: per-request and per-transaction counts, errors, average, approximate p90/p95, max and rate, plus error kinds and first failing samples; fed by AgentRunStats from JMeterThread sample delivery, only while the GUI agent service is on - get_log: filtered tail of the BreakTest log - start_test can run a named scenario Edit faster - add_http_request builds a request from a URL - export_subtree / import_subtree move a subtree through JMX text - batch_open_plan returns brief results for edit steps - undo, redo, select_node Reach more - rawProperties paths reach nested elements and list rows; rawLists replaces the rows of lists without typed table support - request headers are a table on HTTP requests - add_scenario, add_profile, more workload settings (model, schedules, delay, pacing, profile) - open_plan, new_plan - list_functions, list_variables - list_instances / select_instance choose between running windows - open_plan_headless runs the plan tools on a file without a window Control and efficiency - breaktest.agent.permissions=read|edit|run limits what an agent may do - results carry planChangedSinceLastCall when the user edited the tree - the core toolset shortens long descriptions AgentElementCoverageTest creates every installed element and pins the lists that only rawLists can fill. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- open_plan_headless no longer needs a window to set the plan file, and save writes the file directly in that mode - the MCP server and CLI exit when done even after Swing was started - undo and redo report that node ids were reset - start_test, open_plan and new_plan explain that they need a window - document breaktest.agent.permissions and breaktest.agent.confirm_run Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- list_samples lists the samples of the last validation run or of the current/last real run - get_sample returns the request line and body, request headers, response headers and response body of one sample, with failed assertions and sub-samples; long bodies come in windows by offset and limit, or starting at a searched text - real runs keep the full first passing sample and the first two failures of each request (up to 200), so reading traffic does not copy every sample under load Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The live repair and specific-request prompts named only the older tools. Tell the agent, at the step where each applies, about: - get_sample / list_samples for full request and response evidence - batch_open_plan for edits without a dedicated batch tool - get_plan_outline / get_element_open_plan for cheap re-inspection - list_variables and list_functions - add_http_request, rawProperties and rawLists when generating flows The file-backed repair prompt is unchanged: it validates a file, and get_sample reads the samples of a live validation. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
set_user_defined_variable_open_plan wrote into the Test Plan's own variables table. It now uses a User Defined Variables element under the Shared Profile, created on first use, so the variables reach every Thread Group and scenario and are visible as their own node. A value an earlier run left on the Test Plan moves along; a plan without a Shared Profile keeps the old location. Merging variables from a repaired clone follows the same rule. excludeUserDefinedVariables now skips every standalone User Defined Variables element in searches and replacements, not only the Test Plan table, so a credential variable keeps its original value there too. The file-backed repair prompt tells the agent to use the same place, and tool descriptions no longer say "top-level". Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
File mode - AI Repair always edits the open plan through the bridge. The edit surface radio buttons, the file-backed prompt, the repair target and summary sidecar, the reload after a run and the refresh_open_plan_from_file tool are removed, with the breaktest.ai.edit_surface property. Thread group copy - The start dialog has "Keep a copy of the thread group", ticked by default (breaktest.ai.keep_thread_group_copy). Before the run a disabled copy named "<name> (before AI repair)" is placed after the thread group. The agent works on the original, which keeps its name and id, so scenarios keep pointing at the repaired version. The copy gets its own id, replaces a copy kept by an earlier run, and is not listed as an agent change. The prompts tell the agent to leave it alone. Revert - The AI log has a Revert button that, after confirmation, puts the whole plan back as it was before the last run: the thread group, its copy and anything else, such as shared variables. - Restoring the pre-run backup now inserts the tree directly. Opening the backup as a file named the backup as the plan's file, so the next Save would have overwritten the backup instead of the plan. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Both failed on main and on other pull requests before this branch. HTTP/2 reactor lifecycle tests - A stopping user waits five seconds for its HTTP/2 reactor to end and then closes the client anyway; closing it while the reactor still runs makes HttpCore log a ClosedSelectorException, which stoppingActiveRequestsDoesNotLogWarningsOrReactorErrors rejects. The failing CI runs took just over five seconds, and closing without the wait reproduces the exact message locally. - The wait is now httpsampler.http2.reactor_shutdown_wait_ms (default unchanged, 5000). The test raises it, since it checks the orderly path, not the fallback. WebSocket payload editor test - Laying out 400 KB of wrapped text took 135 s on a macOS stress-JIT runner against 2-4 s locally, over the default two-minute limit. The test gets its own ten-minute limit. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
jgaalen
force-pushed
the
claude/breaktest-mcp-harness-7e5c84
branch
from
November 10, 2026 08:11
1ecc4e1 to
27a0273
Compare
… raising the wait The previous commit made the reactor shutdown wait configurable and raised it in stoppingActiveRequestsDoesNotLogWarningsOrReactorErrors. On the macOS stress-JIT runner the reactor then still had not ended after ten seconds, so the test failed on "stopped user must finish promptly" instead. A longer wait does not help there. - Remove the httpsampler.http2.reactor_shutdown_wait_ms property again; product code is back to what it was. - The test measures how long the stop took. When it reached the five-second reactor wait, the ClosedSelectorException that the fallback close makes HttpCore log is accepted. A stop that stayed within the wait must still be silent. Why the reactor does not end in time on that runner is not addressed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The start dialog names the selected thread group by its full tree path,
hidden root included ("Test Plan / Test Plan / ..."). The service
compared that with the visible path only, so the launcher's analysis
threw, the log said "Initial correlation analysis unavailable", and the
agent spent its first turns inspecting the plan itself. This already
happened for plans that were never saved; the node path fix earlier on
this branch extended it to plans loaded from a file.
- selectThreadGroup also accepts that full path
- the log line now carries the reason, and the failure is logged
- an unknown name in a tool schema request is named in the error
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…on unsaved plans A repair run took 46 model turns and 2.4M input tokens, mostly because large tool results were re-sent on every turn and the agent sliced them with scripts. - validate_open_plan summary=true returns a few kilobytes: one row per reached request, the first failure with short request/response heads, ranked repair actions and request candidates, and on a green run a response preview per transaction (or per request outside one) for assertions. Details stay available through get_sample. On a small real plan a failing run went from 8.3 KB to 1.2 KB. - Recording tools and the repair planner no longer refuse a plan that was never saved: an imported recording is held in memory. The planner used to skip recording evidence entirely for such plans, which left the agent to find correlations by trial and error. - The bridge's tool list without names returns names and one-line summaries (11 KB instead of 53 KB). - A correlation batch item without a literal now says which item, why, and what to call to only add an extractor. - The repair and specific-request prompts ask for the summary, tell the agent to read it directly, and to take assertion markers from the previews instead of reading every sample. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…turns Follow-up to a run that took 16 model turns, 475k input tokens and 82 s, of which one 20 s step was the agent classifying the analysis packet. Analysis packet - Each proposal carries a recommendation with why: apply, optional, review or skip. Cache validators (If-None-Match and similar) are optional: a stale value only turns a 304 into a 200, while applying one usually moves its source out of a Parallel Controller. Response echoes are skip, overlapping tokens review. Prompt - WebSocket guidance is sent only for a thread group with WebSocket samplers, generation guidance only when the user gave instructions. The repair template goes from 21.9 KB to 14.4 KB. - The signatures of the tools the workflow names are inlined (1.6 KB), so no turn is spent on reading schemas. - The start is announced by the launcher; the agent no longer spends its first call on it. Bridge - "-" reads the arguments from standard input, so multi-item JSON is one shell call with a heredoc instead of write-file-then-call. Validation summary - A green run that repeats the previous green run only says so. - Assertion previews are 400 characters and prefer JSON answers over page shells. - The summary carries the agent change counts, replacing the separate change-list call before the report. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The "unchanged since previous run" shortcut never triggered: it compared the sample rows in order, and requests in a Parallel Controller finish in a different order on every run. Outcomes are now counted per label, by pass or fail. - The analysis packet lists optional and skipped proposals without their evidence, drops the repeated ordering reason and alternative-source evidence, and gives the node list as text rows. On the demo checkout plan it goes from 21.1 KB to 13.3 KB. - A second variable for the same field is told apart by the request that uses it (csrf_token_payments) before falling back to a number, and the "x_" prefix of header names is dropped. - The start dialog no longer names the hidden tree root, so the thread group path is not doubled in prompts and reports. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s optional A repair run lost a turn on get_sample rejecting a part name: the result calls the request line and body "request", but the argument had to be "requestBody". Aliases (request, response, body, headers) and a single comma-separated string are accepted, and an unknown part is named in the error. The dynamic audit reported the stale If-None-Match value as an opaque-token candidate, which the agent then investigated although the repair packet had already marked it optional. The compact audit now labels such values cache-validator, optional, no action needed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The "unchanged since previous run" shortcut compared against the last validation of the whole session. A new AI Repair run therefore got "unchanged" on its very first validation, with no request rows and no response previews, and the agent fetched every sample one by one to choose assertions. The shortcut now compares only against the last summary handed out, and that baseline is cleared when an AI run starts and when the plan is replaced (open, new, headless open, restore from backup). get_sample and the cached responses are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI Repair told Codex to use an installed breaktest-jmeter-repair skill. The installed skill describes the old workflow (compact validation, inspect_open_plan, file-backed mode) and is about 60 KB, so it contradicted the current prompt and enlarged every Codex request. The prompt is self-contained now; Codex gets the same bootstrap as the other tools, which tells it not to look for or invoke that skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… discovery A request to set the think time of eight transactions took eight model turns. Two went to list_available_elements and describe_element, only to learn the delay setting names that get_element_open_plan left out because they were at their default; three went to selecting, re-reading and listing changes after a batch that already confirmed every edit. - get_element_open_plan lists the other settable settings with their types (otherSettings), so one read is enough to change any of them. - String "...Mode" settings expose their allowed values from the class's constants, e.g. delayMode Disabled|Fixed|Random|Gaussian Random. - The specific-request prompt describes the short path for a focused edit (outline, one read, one batch, report), keeps element discovery for adding new element types, and drops the closing inspection and change list for edits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Edit requests of the "change X on every Y" kind needed one batch step per node, hit the 100-step limit on larger plans, and could only rewrite whole tables, so adding one header meant reading and rewriting every request's headers. - A batch step with forEach runs its tool once for every node matching a filter (type, class, name, below a node, static assets include, exclude or only), as one undo step. The node is the target unless the arguments name one; "@node" and {{node.name}}, {{node.treePath}}, {{node.parentName}} or {{node.<setting>}} fill in per-node values. Nodes inside a disabled ancestor, such as a kept repair copy, are skipped unless includeDisabled is set. The result is counts, failures and, for read tools, the per-node results. - configure_element_open_plan and add_element_open_plan take tableMode: replace (default), append, upsert by name, or remove by name. - The edit prompt tells the agent to use forEach for such requests, to prefer one shared element (Header Manager, HTTP Request Defaults) when it gives the same result, and the table modes. On a copy of a 78-sampler checkout plan, one call upserted a header on 56 requests, set think times on 8 transactions and disabled 14 static assets, returning 884 characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding the tableMode parameter with a Kotlin default removed the three-argument signature that WebSocketSamplerTest (Java) calls, so the protocol module's tests no longer compiled and every CI job failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Ubuntu Xvfb job failed in focusingExistingExpressionPreservesText- AndNormalKeys: the editor's first focusGained had fired, but by the next EDT turn it no longer owned focus. The sibling test failed the same way on an earlier run. awaitEditorFocus now waits until the editor actually owns focus, and on failure says which component took it. Not related to the agent changes in this branch; the test was not touched and passes locally on macOS three times in a row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lets coding harnesses (Claude Code, Codex, OpenCode, Pi and others) read and change the plan open in BreakTest through MCP, see changes appear live, run tests and read real traffic. AI Repair is rebuilt on the same tools and is much faster.
Bridge and MCP server
Content-Lengthframing. Before this, a standard client could not connect.batch_open_planruns several tools in one call; a later step can use an earlier step's node id ("@name").forEachruns its tool on every node matching a filter (type, class, name, below a node, static assets) as one undo step, with per-node templates such as{{node.name}}or{{node.path}}. Nodes inside a disabled parent, such as a kept repair copy, are skipped.get_plan_outline: compact tree.get_element_open_plan: a node's settings, table rows and children; settings at their default are listed by name and type.list_variablesandlist_functions.add_http_requestbuilds a request from a URL.set_enabled_open_plan,export_subtree/import_subtree,undo,redo,select_node.rawPropertiespaths reach nested elements and list rows;rawListsfills lists without typed tables.tableModereplace,append,upsertorremovefor table rows. Request headers are a table on HTTP requests.save_open_plan,open_plan,new_plan,add_scenario,add_profile,set_scenario_workload(threads, ramp-up, duration, loops, model, schedules, pacing, profile).start_test(with a confirmation dialog unlessbreaktest.agent.confirm_run=false),stop_test,test_status,test_results(per-label counts, errors, percentiles, error samples) andget_log.list_samples/get_samplereturn the full request, headers and response of a validated sample or of a run, in windows.validate_open_plan summary=truereturns a few kilobytes: one row per request, the first failure with short heads, ranked repair actions, and on a green run response previews per transaction. A repeat of an unchanged green run within the same AI run only says so.list_instances/select_instancechoose between open windows.open_plan_headlessruns the plan tools on a file without a window.breaktest.agent.permissions=read|edit|runlimits what an agent may do.planChangedSinceLastCallwhen the user edited the plan in between.BREAKTEST_MCP_TOOLSETS=coreadvertises a smaller set;describe_toolslists the rest.AgentToolSpecs, used by both the server and the GUI service.AI Repair
refresh_open_plan_from_fileandbreaktest.ai.edit_surface.breaktest.ai.keep_thread_group_copy) places a disabled copy before the run; the repair works on the original, so scenarios keep pointing at it. One copy per thread group. A Revert button in the AI log puts the whole plan back as it was before the last run.breaktest-jmeter-repairskill, which described the old workflow.csrf_token_payments), and the thread group path is no longer doubled.Fixes
list_http_arguments_open_planandset_http_argument_value_open_plansaw no rows.search_open_plan_valuesreturned object ids instead of node paths.excludeUserDefinedVariablesonly skipped the Test Plan table.Test stability (not caused by this branch)
HTTPHC5H2ThreadLifecycleTest: accepts theClosedSelectorExceptionfrom the fallback close only when the stop reached the five-second reactor wait.WebSocketPayloadEditorTest: the large-payload case has a ten-minute limit (135 s on a macOS stress-JIT runner).ParameterCompletionTableTest: waits for settled editor focus under Xvfb.Motivation and Context
Harnesses should be able to work on a running BreakTest directly: inspect the plan, edit it live, validate, read real requests and responses, and run tests. AI Repair runs on the same bridge.
On the demo checkout plan with Pi and DeepSeek v4.1 flash, a full AI Repair went from 46 model turns, 2.4M input tokens and 2m51s to 8 turns, 118k tokens and 34s, with the same green result.
How Has This Been Tested?
./gradlew autostyleCheck, checkstyle, andcompileTestJava compileTestKotlinfor all modules.:src:core:testfororg.apache.jmeter.ai.*,*AiAutoScripting*and*AiPrompt*;:src:dist-check:testfor*AgentElement*, includingAgentElementCoverageTest, which creates all 139 installed elements and pins the lists onlyrawListscan fill. The full suite runs in CI; all 11 checks pass.test_results, sample reading, text round-trip, undo, open plan, instance selection, and a for-each on a 78-sampler plan copy (header upsert on 56 requests, think times on 8 transactions, 14 static assets disabled, in one call).Not exercised:
JMeterThreadunder heavy load. It is only active while the GUI agent service is on.Things a reviewer should know
refresh_open_plan_from_fileare removed.JMeterThreadcallsAgentRunStatsfor every sample while the agent service is on.xdocsand the changelog are not updated.bin/jmeter.properties, the settings catalog and the AI help text are.Types of changes
Checklist:
🤖 Generated with Claude Code