Repository navigation
Conversation
- Give compile, release_match, disclaimer, and live_verify one api field (jcsmp default, python); release_match replaces maven_release_match and disclaimer replaces java_disclaimer - Python compile installs each requirements.txt into a fresh venv, then runs py_compile and an import smoke; Python release_match checks the PyPI pin - Python live_verify writes config.json after the subject model finishes; add OUTPUT_EVAL_BROKER_TRUST_STORE for tcps:// hosts (no default trust store) - Add file_exists count, and skip .venv and __pycache__ in file globs - Add six Python cases mirroring the JCSMP set, plus persisted-request routing - Exclude heredoc checklist text from the verify.sh run check in both Suggested cases (false positive on Sonnet) - Document the Python graders and the trust store variable in evals/README.md
- Limit the verify.sh heredoc exclusion to text inside a heredoc body - Delete config.json after the Python live round trip and fail as INFRA when the write fails - Correct README claims about compile cases, the import smoke, and timeout scope
aelsammak
marked this pull request as ready for review
October 20, 2026 16:23
|
Which test failed on sonnet? Does it fail all the time or just flaky? |
Author
@mtite-sol |
mtite-sol
requested changes
Oct 20, 2026
mtite-sol
left a comment
mtite-sol
left a comment
There was a problem hiding this comment.
Requesting changes for the heredoc exclude_pattern on the verify.sh graders; details inline.
mtite-sol
requested changes
Oct 20, 2026
mtite-sol
left a comment
mtite-sol
left a comment
There was a problem hiding this comment.
Just one change requested but otherwise LGTM!
- Replace the four heredoc exclude_pattern regexes with an opt-in ignore_heredoc_bodies flag on the tool_use grader, so each grader keeps one copy of the verify.sh verb list - Add a strip_heredoc_bodies jq function that drops heredoc bodies from Bash commands before tojson, closing the hole where a lowercase or digit delimiter (EOF2, Eof) hid a real verify.sh run - Escape the quote class as \\x27 so the jq string literal compiles; a compile error would silently empty every tool_use grader - Add ignore_heredoc_bodies to the tool_use schema and the runner grader notes
mtite-sol
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change?
Add output eval coverage for the Solace Python API. The JCSMP-only graders (
compile,maven_release_match,java_disclaimer,live_verify) could not check a Python project, so the graders now take anapifield, and new Python cases exercise the skill content from EBP-3105, EBP-3106, and EBP-3107.How is this accomplished?
compile,release_match,disclaimer, andlive_verifytakeapi: jcsmp|python(defaultjcsmp).release_matchreplacesmaven_release_match, anddisclaimerreplacesjava_disclaimer. The ticket named separatepy_compileandpypi_release_matchgraders; one field on each grader keeps the set consistent.compile: for eachrequirements.txt, a fresh venv outside the project,pip install -r,py_compileof every script, and an import smoke of every module. Pythonrelease_match: asolace-pubsubplus==<PyPI info.version>line.live_verifywritesconfig.jsoninto the generated project after the subject model finishes, then runs./verify.sh roundtrip. Atcps://host also needs the newOUTPUT_EVAL_BROKER_TRUST_STOREvariable, because the Python API has no default trust store. The runner exits before any case when it is missing.file_existsgetscount, and file globs skip.venv/and__pycache__/.must_pass, compile, release match, live verify), design first on a bare build prompt (must_pass), debug redirect (must_pass), Solace Suggested overtcps://with two projects, design summary with a docs WebFetch, and persisted request routed to Guaranteed Pub/Sub.verify.shrun check. Sonnet wrote the checklist with a heredoc that quotes the handed-back command.evals/README.mddocuments the Python graders and the trust store variable.Eval results (local, live verify ran against a Solace Cloud
tcps://broker)--model opus): 17/17. Live verify passed for the JCSMP and Python compile cases.claude-sonnet-5-5): 16/17 on the full leg, with live verify passed for both compile cases.Anything reviewers should focus on/be aware of?
N/A