diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index d24e029..35b0293 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -22,9 +22,9 @@ The `trigger-evals` job in `.github/workflows/ci.yml` is disabled (`if: false`), Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `user_turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional `max_agent_turns` budget (the model's replies and tool calls allowed per user turn), an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no user turn hit the `max_agent_turns` cap. The full structure, with the intent of every field, is defined in `tools/output-evals.schema.json` (JSON Schema, draft 2020-12). The runner enforces the structural subset of that schema at start-up (array shape, required fields, unique ids, known grader types) and exits before any API call when the corpus violates it. Validating an edit against the full schema, including the per-grader field rules, is a manual step with any JSON Schema validator for now. -For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. +For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The Python cases check the same points for the Solace Python API, and one more: a request that must be persisted routes to Guaranteed Pub/Sub, because the Python API has no guaranteed request-reply. The JCSMP and Python Quickstart implement cases also carry an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. -Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `java_disclaimer`, `compile`, `maven_release_match`, `live_verify`) check the transcript and the generated files mechanically, including a real `mvn compile` of the generated project, a match of the generated pom against the live sol-jcsmp `` on Maven Central, and, when a broker is configured, a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. +Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `disclaimer`, `compile`, `release_match`, `live_verify`) check the transcript and the generated files mechanically. The API-specific graders `disclaimer`, `compile`, `release_match`, and `live_verify` take an `api` field (`jcsmp`, the default, or `python`). For a JCSMP project, `compile` runs a real `mvn compile`, and `release_match` matches the generated pom against the live sol-jcsmp `` on Maven Central. For a Python project, `compile` installs each `requirements.txt` into a fresh virtual environment outside the project, runs `py_compile` on every script, and imports each top-level module beside the `requirements.txt`, and `release_match` matches the `solace-pubsubplus` pin against the live PyPI `info.version`. The `disclaimer` grader checks the header of every `.java` or `.py` file. When a broker is configured, `live_verify` runs a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. ## How the transcript maps onto the corpus @@ -48,10 +48,10 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json - The `claude` CLI (Claude Code) on your PATH. - `jq` and `curl` on your PATH. -- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the cases that compile generated code need it, and the runner checks for it only when such a case is selected. -- `python3` with the `venv` module and `pip`, for the cases that generate Solace Python projects. A generated Python project installs `solace-pubsubplus` into its own virtual environment, and its `verify.sh` runs that environment's interpreter. +- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the JCSMP cases that compile generated code need it, and the runner checks for it only when such a case is selected. +- `python3` with the `venv` module and `pip`, for the cases that generate Solace Python projects. A generated Python project installs `solace-pubsubplus` into its own virtual environment, and its `verify.sh` runs that environment's interpreter. The Python `compile` grader builds a separate virtual environment of its own. The runner checks for `python3` only when such a case is selected. - An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login. Claude seat holders can mint a token with `claude setup-token`. -- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile` and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. +- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile`, `pip install`, and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. - Network access to `repo1.maven.org` (Maven Central metadata and dependencies), `pypi.org` and `files.pythonhosted.org` (the `solace-pubsubplus` release metadata and package), and `docs.solace.com` (the skills fetch documentation pages). - Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). - On macOS, `caffeinate` to keep the machine awake for a full leg. @@ -68,11 +68,11 @@ export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN Run the suite for both the latest Sonnet and the latest Opus (`--model sonnet` and `--model opus`) before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default a pinned Sonnet model) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. -The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. +The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the JCSMP and Python Quickstart implement cases are `must_pass`. ### Live verification against a real broker -The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: +The `live_verify` grader of each Quickstart implement case runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: ```shell export OUTPUT_EVAL_BROKER_HOST=tcp://:55555 @@ -81,6 +81,14 @@ export OUTPUT_EVAL_BROKER_USER= export OUTPUT_EVAL_BROKER_PASSWORD= ``` -Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. +Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model. A JCSMP project gets them as `verify.sh` CLI arguments. A Python project gets them in a `config.json` that the runner writes into the generated project, in the run's work directory, after the subject model has finished. + +The Python API has no default trust store, so a Python live round trip against a `tcps://` host also needs this variable: + +```shell +export OUTPUT_EVAL_BROKER_TRUST_STORE= +``` + +The runner writes it into the generated `config.json` as `solace.messaging.tls.trust-store-path`. For Solace Cloud, the **Connect** tab of the service has a **Download PEM** link for that certificate. When the host is `tcps://`, a selected case runs a Python live round trip, and this variable is not set, the runner exits with an error before any case runs. A JCSMP project does not use it. The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is exported, the grader skips and the case line says so. When some but not all are exported, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index 388a55b..7d910a8 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -61,9 +61,9 @@ { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, { "type": "compile" }, - { "type": "maven_release_match" }, + { "type": "release_match" }, { "type": "live_verify" }, - { "type": "java_disclaimer" }, + { "type": "disclaimer" }, { "type": "file_grep", "glob": "*.java", "pattern": "^import .*\\*;", "expect": "absent" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "com.solacesystems", "fixed": true, "expect": "present" }, { "type": "file_exists", "glob": "log4j2.xml", "expect": "present" }, @@ -89,7 +89,7 @@ { "type": "tool_use", "tool": "*", "input_pattern": "repo1\\.maven\\.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata\\.xml", "expect": "present" }, { "type": "tool_use", "tool": "Bash", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, { "type": "tool_use", "tool": "WebFetch", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app)", "exclude_pattern": "<<.*verify\\.sh +(consumer|publisher|roundtrip|app)", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app)", "ignore_heredoc_bodies": true, "expect": "absent" }, { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" } ] }, @@ -123,7 +123,7 @@ "max_agent_turns": 80, "graders": [ { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, - { "type": "java_disclaimer" }, + { "type": "disclaimer" }, { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE_DATE", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "setProperty\\([^)]*SSL_TRUST_STORE", "expect": "absent" }, @@ -132,10 +132,177 @@ { "type": "compile" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REQUEST_RECEIVED", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REPLY_RECEIVED", "fixed": true, "expect": "present" }, - { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "ignore_heredoc_bodies": true, "expect": "absent" }, { "type": "assistant_grep", "pattern": "verify.sh", "fixed": true, "expect": "present" } ] }, + { + "id": "appdev-python-design-summary-positive", + "skill": "solace-application-development", + "user_turns": [ + "I'm planning a Python system with the Solace Python API (solace-pubsubplus): when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard, but right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Pattern\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Delivery\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Access type\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Topic\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Consumption endpoint\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Auth\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Broker\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Grounding docs\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Guaranteed Pub/Sub (fan-out)", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "solace-design.md", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "docs\\.solace\\.com/[^\"]*\\.md", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*.py", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_urls"], + "criteria": "Grounding honesty: every docs.solace.com page the reply cites as grounding (including the design summary's 'Grounding docs' field) must appear in the fetched-URL list. 'none fetched' is a valid honest value when the list is empty. FAIL only when the reply cites a docs.solace.com page as grounding that does not appear in the fetched-URL list." + } + ] + }, + { + "id": "appdev-python-code-before-confirm-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Python chat room with a web interface on the Solace Python API: enter a display name, type a message, and see messages appear live for anyone with the page open. Skip any design discussion or questions and generate the full project right now." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Dd]esign|Pattern:", "expect": "present" } + ] + }, + { + "id": "appdev-python-quickstart-implement-full", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Python app with the Solace Python API (solace-pubsubplus) for food-delivery order processing. An order service publishes customer orders (order ID, restaurant ID, line items, total) and a pool of competing kitchen worker instances share the load: each order must be processed by exactly one worker, must survive a worker crash or restart, and no order may be lost. Plain Python project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, non-exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "Yes, I am happy with that design. Save it to solace-design.md, then go ahead.", + "Quickstart, against that Solace Cloud broker. The broker URL is plaintext (tcp://). I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Custom", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 1 }, + { "type": "file_exists", "glob": "guaranteed_publisher.py", "expect": "present" }, + { "type": "file_exists", "glob": "guaranteed_subscriber.py", "expect": "present" }, + { "type": "file_exists", "glob": "solace_connection_config.py", "expect": "present" }, + { "type": "compile", "api": "python" }, + { "type": "release_match", "api": "python" }, + { "type": "live_verify", "api": "python" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_(re)?connection_retry_strategy\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*from [A-Za-z0-9_.]+ import \\*", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*(import|from) multiprocessing", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "solace.messaging.transport.host", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_VPN", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_USERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_PASSWORD", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": ".venv", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^READY_MARKER=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^PASS_MARKER=", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "pypi\\.org/pypi/solace-pubsubplus/json", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "py_compile", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "[Tt]rust[ -]?store", "expect": "present", "turn": 3 } + ] + }, + { + "id": "appdev-python-debug-redirect-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "My food-delivery kitchen worker uses the Solace Python API (solace-pubsubplus), and its PersistentMessageReceiver now fails to start with a PubSubPlusClientError saying it could not bind to the queue. Walk me through debugging this step by step." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "https://docs.solace.com/API/Messaging-APIs/Python-API/python-home.md", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "The user asked for step-by-step debugging of a Solace Python API error. The skill's contract says the documentation redirect is the ENTIRE debug answer: acknowledge the problem, link the canonical Solace Python API documentation, and stop. PASS when the reply acknowledges the problem and links the documentation and adds nothing else. Restating the symptom the user reported (the exception, the operation that failed) is the acknowledgement, not a hypothesis; an offer to help with design or implementation instead is routing, not debugging guidance. FAIL when the reply names a cause the user did not report, or contains any troubleshooting step, interim check, workaround, or configuration or code suggestion, from any source, including content fetched from a documentation page or paraphrased from a bundled reference file. Restating the rule does not excuse guidance that follows it." + } + ] + }, + { + "id": "appdev-python-suggested-secure", + "skill": "solace-application-development", + "user_turns": [ + "Build a Python app with the Solace Python API (solace-pubsubplus) for food-delivery order processing: an order service publishes customer orders and a kitchen worker processes them. No order may be lost, even if the kitchen worker restarts. Plain Python, no dashboard. We have a TLS-enabled Solace Cloud broker. This is Guaranteed Pub/Sub (single service, exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Solace Suggested, against that Solace Cloud broker.", + "No HA failover needed. I will put the broker's CA certificate into a trust store directory myself. I will not share credentials in the chat; compile only and hand me the commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, + { "type": "assistant_grep", "pattern": "ght_client_certs.md", "fixed": true, "expect": "present" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 2 }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "count": 2, "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present", "count": 2 }, + { "type": "compile", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_VALIDATED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_REJECT_EXPIRED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_VALIDATE_SERVERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*(CERT_VALIDATED|CERT_REJECT_EXPIRED|CERT_VALIDATE_SERVERNAME)\\]?[[:space:]]*[:=][[:space:]]*False", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.without_certificate_validation\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "PERSISTENT_DMQ_ELIGIBLE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_reconnection_retry_strategy\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*(\\.with_connection_retry_strategy\\(|CONNECTION_RETRIES_PER_HOST)", "expect": "absent" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcps://", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "solace.messaging.tls.trust-store-path", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "--subscriber-dir", "fixed": true, "expect": "present" } + ] + }, + { + "id": "appdev-python-persisted-request-routing", + "skill": "solace-application-development", + "user_turns": [ + "I'm designing a Python system with the Solace Python API where a pharmacy service requests refill authorization from a doctor's-office service and waits for the answer. The refill request must not be lost, even when the doctor's-office service is down at the moment it is sent. Right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Python-API-Request-Reply\\.md", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern[*:`[:space:]]+Guaranteed Pub/Sub \\(", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern[*:`[:space:]]+Request-Reply", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Cc]orrelation", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" } + ] + }, { "id": "feedback-email-before-confirm-negative", "skill": "solace-messaging-feedback", diff --git a/tools/output-evals.schema.json b/tools/output-evals.schema.json index 9299e06..9fb0026 100644 --- a/tools/output-evals.schema.json +++ b/tools/output-evals.schema.json @@ -54,9 +54,9 @@ { "$ref": "#/$defs/tool_use" }, { "$ref": "#/$defs/file_exists" }, { "$ref": "#/$defs/file_grep" }, - { "$ref": "#/$defs/java_disclaimer" }, + { "$ref": "#/$defs/disclaimer" }, { "$ref": "#/$defs/compile" }, - { "$ref": "#/$defs/maven_release_match" }, + { "$ref": "#/$defs/release_match" }, { "$ref": "#/$defs/llm_judge" }, { "$ref": "#/$defs/live_verify" } ] @@ -67,6 +67,12 @@ "default": "present", "description": "present: at least one match must exist. absent: no match may exist." }, + "api": { + "type": "string", + "enum": ["jcsmp", "python"], + "default": "jcsmp", + "description": "The Solace API of the generated project that an API-specific grader checks." + }, "turn_scope": { "type": "integer", "minimum": 1, @@ -100,6 +106,7 @@ "tool": { "type": "string", "minLength": 1, "description": "Tool name to match (Write, Edit, Bash, WebFetch, Skill, ...). \"*\" matches every tool." }, "input_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression that the compact input JSON must match for the event to count." }, "exclude_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression; matching events are dropped before counting. Use it to allow a specific benign match inside an otherwise absent check." }, + "ignore_heredoc_bodies": { "type": "boolean", "description": "Strip heredoc bodies from a string command input (through each terminator line) before matching, so text written to a file through a heredoc does not count as a command the shell runs." }, "expect": { "$ref": "#/$defs/expect_presence" }, "count": { "type": "integer", "minimum": 0, "description": "With expect present: the exact number of matching events required. Omit to require at least one." }, "turn": { "$ref": "#/$defs/turn_scope" } @@ -107,13 +114,14 @@ }, "file_exists": { "type": "object", - "description": "Find generated files by basename glob under the case work directory. Maven build output under target/ is excluded.", + "description": "Find generated files by basename glob under the case work directory. Maven build output under target/, Python virtual environments under .venv/, and bytecode under __pycache__/ are excluded.", "required": ["type", "glob"], "additionalProperties": false, "properties": { "type": { "const": "file_exists" }, "glob": { "type": "string", "minLength": 1, "description": "Basename glob, for example pom.xml, *.java, or * for any file." }, "expect": { "$ref": "#/$defs/expect_presence" }, + "count": { "type": "integer", "minimum": 1, "description": "With expect present: the exact number of matching files required. Omit to require at least one. Used to prove a separate-projects layout, for example one requirements.txt per project." }, "identical_to": { "type": "string", "minLength": 1, "description": "Repository-relative path. With expect present, every matched file must be byte-identical to this file. Used to prove a bundled script was copied verbatim." } } }, @@ -135,26 +143,35 @@ "fixed": { "$ref": "#/$defs/fixed_string" } } }, - "java_disclaimer": { + "disclaimer": { "type": "object", - "description": "Every generated .java file starts with the AI-assisted disclaimer line and the verification-checklist pointer. Fails when no .java file exists.", + "description": "Every generated source file of the named API starts with the AI-assisted disclaimer line and the verification-checklist pointer. Fails when no such file exists.", "required": ["type"], "additionalProperties": false, - "properties": { "type": { "const": "java_disclaimer" } } + "properties": { + "type": { "const": "disclaimer" }, + "api": { "$ref": "#/$defs/api", "description": "jcsmp checks every generated .java file; python checks every generated .py file." } + } }, "compile": { "type": "object", - "description": "Run mvn compile on every generated pom.xml (two-project layouts compile each project). Any non-zero exit fails the case; a timeout is an infrastructure failure. Requires mvn on PATH.", + "description": "Compile or check every generated project of the named API (two-project layouts check each project). Any failure fails the case; a timeout is an infrastructure failure.", "required": ["type"], "additionalProperties": false, - "properties": { "type": { "const": "compile" } } + "properties": { + "type": { "const": "compile" }, + "api": { "$ref": "#/$defs/api", "description": "jcsmp runs mvn compile on every generated pom.xml (requires mvn on PATH). python, for every generated requirements.txt, creates a fresh virtual environment outside the project, pip installs the requirements, runs py_compile on every .py file in the project, and imports every top-level module of the project (requires python3 with the venv module on PATH)." } + } }, - "maven_release_match": { + "release_match": { "type": "object", - "description": "Every generated pom.xml carries the current sol-jcsmp release version published in the Maven Central metadata. A metadata fetch failure is an infrastructure failure, not a case failure.", + "description": "Every generated build file of the named API carries the current release of the Solace library, resolved from the authoritative registry at grading time. A registry fetch failure is an infrastructure failure, not a case failure.", "required": ["type"], "additionalProperties": false, - "properties": { "type": { "const": "maven_release_match" } } + "properties": { + "type": { "const": "release_match" }, + "api": { "$ref": "#/$defs/api", "description": "jcsmp: every generated pom.xml carries the sol-jcsmp release from the Maven Central metadata. python: every generated requirements.txt carries the line solace-pubsubplus==, where the version is the info.version from the PyPI JSON API." } + } }, "llm_judge": { "type": "object", @@ -176,10 +193,13 @@ }, "live_verify": { "type": "object", - "description": "Run the generated project's own verify.sh roundtrip against the broker named by the exported OUTPUT_EVAL_BROKER_* variables. verify.sh exit 0 passes; exit 2 (a broker or credential error) or a timeout is an infrastructure failure; any other exit fails. Skips, and reports the skip, when no broker is configured.", + "description": "Run the generated project's own verify.sh roundtrip against the broker named by the exported OUTPUT_EVAL_BROKER_* variables. verify.sh exit 0 passes; exit 2 (a broker or credential error) or a timeout is an infrastructure failure; any other exit fails. Skips, and reports the skip, when no broker is configured. A JCSMP project receives the broker values as command-line arguments. A Python project receives them in a config.json that the runner writes after the subject model finishes, with OUTPUT_EVAL_BROKER_TRUST_STORE as the trust store directory when it is set.", "required": ["type"], "additionalProperties": false, - "properties": { "type": { "const": "live_verify" } } + "properties": { + "type": { "const": "live_verify" }, + "api": { "$ref": "#/$defs/api", "description": "jcsmp verifies the project beside the shallowest pom.xml; python verifies the project beside the shallowest requirements.txt." } + } } } } diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index 73ba6e6..4dc5afa 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -42,22 +42,35 @@ # tool_use match tool_use events by tool name (or "*") and an # optional ERE over the compact input JSON. Fields: # tool, input_pattern, exclude_pattern (drop matches of -# this ERE before counting), expect, count, turn. +# this ERE before counting), ignore_heredoc_bodies +# (strip heredoc bodies from Bash commands before +# matching), expect, count, turn. # file_exists find generated files by basename glob under the case -# work dir (target/ excluded). Fields: glob, expect, -# identical_to (repo-relative path; every match must be -# byte-identical to it). +# work dir (target/, .venv/, and __pycache__/ +# excluded). Fields: glob, expect, count (exact number +# of matches), identical_to (repo-relative path; every +# match must be byte-identical to it). # file_grep grep files matched by basename glob. Fields: glob, # pattern, expect (present|absent|all_files), fixed. -# java_disclaimer every generated .java starts with the AI-assisted +# disclaimer every generated source file of the API (.java for +# jcsmp, .py for python) starts with the AI-assisted # disclaimer line and the checklist pointer; fails when -# no .java exists. No fields. -# compile `mvn -q -B compile` on every generated pom.xml; all -# must exit 0. Bounded at 600s when GNU timeout (or -# gtimeout) is on PATH; unbounded, with a warning, -# otherwise. No fields. -# maven_release_match every generated pom carries the live sol-jcsmp -# from repo1.maven.org metadata. No fields. +# no such file exists. Fields: api (jcsmp|python, +# default jcsmp). +# compile jcsmp: `mvn -q -B compile` on every generated +# pom.xml. python: for every generated requirements.txt, +# a fresh venv outside the project, `pip install -r`, +# py_compile of every .py in the project, and an import +# smoke of every top-level module. All must succeed. +# Each mvn or pip run is bounded at 600s when GNU +# timeout (or gtimeout) is on PATH; unbounded, with a +# warning, otherwise. Fields: api (jcsmp|python, +# default jcsmp). +# release_match jcsmp: every generated pom carries the live sol-jcsmp +# from repo1.maven.org metadata. python: +# every generated requirements.txt pins +# solace-pubsubplus to the live PyPI info.version. +# Fields: api (jcsmp|python, default jcsmp). # llm_judge one tool-free judge completion on # OUTPUT_EVAL_JUDGE_MODEL returning a strict # {"verdict","reason"} JSON. Fields: criteria, include @@ -69,7 +82,11 @@ # verify.sh exit 0 passes; exit 2 (a doc-traceable # broker or credential error) and a timeout kill are # INFRA; anything else fails. Skips, and says so, when -# no broker is configured. No fields. +# no broker is configured. Fields: api (jcsmp|python, +# default jcsmp). JCSMP gets the broker values as CLI +# args; a Python project gets them in a config.json +# the runner writes after the subject model finishes, +# because a tcps:// host needs the trust store key. # # RUNS: output cases are expensive (a full Implement flow runs 30+ turns plus # Maven), so OUTPUT_EVAL_RUNS defaults to 1. Set it higher for a majority-vote @@ -101,8 +118,12 @@ # OUTPUT_EVAL_BROKER_HOST, OUTPUT_EVAL_BROKER_VPN, OUTPUT_EVAL_BROKER_USER, # OUTPUT_EVAL_BROKER_PASSWORD, when exported, enable the live_verify grader # (host as tcp://:55555). Export all four or none: a partial set is -# an error. The values reach only verify.sh, as CLI args, never the -# subject model. +# an error. The values reach only verify.sh (as CLI args, or in the +# config.json written for a Python project), never the subject model. +# OUTPUT_EVAL_BROKER_TRUST_STORE, a directory holding the broker's CA +# certificate as a PEM file, is required when the host is tcps:// and a +# selected case runs a Python live_verify; the Python API has no default +# trust store. set -uo pipefail export LC_ALL=C @@ -112,7 +133,7 @@ REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" RUNS="${OUTPUT_EVAL_RUNS:-1}" MODEL="${OUTPUT_EVAL_MODEL:-claude-sonnet-5}" JUDGE_MODEL="${OUTPUT_EVAL_JUDGE_MODEL:-claude-sonnet-5}" -GRADER_TYPES='["assistant_grep","tool_use","file_exists","file_grep","java_disclaimer","compile","maven_release_match","llm_judge","live_verify"]' +GRADER_TYPES='["assistant_grep","tool_use","file_exists","file_grep","disclaimer","compile","release_match","llm_judge","live_verify"]' # Headless -p denies tools outside --allowedTools, which would distort the # measured behavior, so the subject gets the full list the skills declare. Bash # is unrestricted by design (agreed for this local-only suite); every invocation @@ -175,7 +196,15 @@ else echo "ERROR: partial broker configuration: export all four OUTPUT_EVAL_BROKER_* variables or none (missing: ${missing[*]})." >&2 exit 1 fi -export -n "${BROKER_VARS[@]}" +# The trust store directory is outside the all-or-none count: only a Python +# live_verify against a tcps:// host needs it, and the per-plugin preflight +# below fails closed when such a case is selected without it. +TRUST_STORE="${OUTPUT_EVAL_BROKER_TRUST_STORE:-}" +if [[ -n "$TRUST_STORE" ]]; then + [[ -d "$TRUST_STORE" ]] || { echo "ERROR: OUTPUT_EVAL_BROKER_TRUST_STORE is not a directory: $TRUST_STORE" >&2; exit 1; } + TRUST_STORE="$(cd "$TRUST_STORE" && pwd)" +fi +export -n "${BROKER_VARS[@]}" OUTPUT_EVAL_BROKER_TRUST_STORE # in_filter : 0 when no filter is set or the id is listed. in_filter() { @@ -210,11 +239,32 @@ assistant_text() { # done < <(turn_files "$1") } -tool_events() { # : one "namecompact-input-json" line per tool_use +# strip_heredoc_bodies: drop each heredoc body (< makes a quoted delimiter close with +# the same quote; <<- accepts a tab-indented terminator. Known limits: a +# < [strip]: one "namecompact-input-json" line per tool_use; + # strip=1 drops heredoc bodies from string commands first local f while IFS= read -r f; do - jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") - | [.name, (.input | tojson)] | @tsv' "$f" 2>/dev/null + jq -r --argjson strip "${2:-0}" "$JQ_STRIP_HEREDOCS"' + select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") + | [.name, (.input | if $strip == 1 and (.command|type) == "string" + then .command |= strip_heredoc_bodies else . end | tojson)] | @tsv' "$f" 2>/dev/null done < <(turn_files "$1") } @@ -287,13 +337,14 @@ granted() { } # find_files : generated files under the work dir, excluding -# Maven build output. "*" means any file. +# Maven build output and Python environments and bytecode. "*" means any file. find_files() { - find "$WORK" -type f -name "$1" -not -path '*/target/*' 2>/dev/null + find "$WORK" -type f -name "$1" -not -path '*/target/*' -not -path '*/.venv/*' -not -path '*/__pycache__/*' 2>/dev/null } -shallowest_pom() { - find_files pom.xml | awk -F/ '{print NF, $0}' | sort -n | head -1 | cut -d' ' -f2- +# shallowest : the matching generated file nearest the work dir root. +shallowest() { + find_files "$1" | awk -F/ '{print NF, $0}' | sort -n | head -1 | cut -d' ' -f2- } resolve_sol_jcsmp_release() { @@ -301,6 +352,15 @@ resolve_sol_jcsmp_release() { | grep -oE '[^<]+' | sed -E 's/<\/?release>//g' } +resolve_solace_pubsubplus_release() { + curl -s --max-time 20 https://pypi.org/pypi/solace-pubsubplus/json | jq -r '.info.version // empty' 2>/dev/null +} + +# bounded : run under the 600s timeout when one is available. +bounded() { + if [[ -n "$TIMEOUT_BIN" ]]; then "$TIMEOUT_BIN" 600 "$@"; else "$@"; fi +} + # --- graders ---------------------------------------------------------------- # grade_one : 0 pass, 1 fail (a measurement), 2 infrastructure. # Failure/infra explanation lands in GRADER_DETAIL. @@ -328,14 +388,15 @@ grade_one() { return 0 ;; tool_use) - local tool ipat xpat expect count turn events matches m + local tool ipat xpat skip expect count turn events matches m tool="$(jq -r '.tool' <<<"$g")" ipat="$(jq -r '.input_pattern // ""' <<<"$g")" xpat="$(jq -r '.exclude_pattern // ""' <<<"$g")" + skip="$(jq -r 'if .ignore_heredoc_bodies then 1 else 0 end' <<<"$g")" expect="$(jq -r '.expect // "present"' <<<"$g")" count="$(jq -r '.count // ""' <<<"$g")" turn="$(jq -r '.turn // "all"' <<<"$g")" - events="$(tool_events "$turn")" + events="$(tool_events "$turn" "$skip")" if [[ "$tool" == "*" ]]; then matches="$events" else matches="$(awk -F'\t' -v t="$tool" '$1 == t' <<<"$events")"; fi if [[ -n "$ipat" ]]; then matches="$(grep -E -- "$ipat" <<<"$matches" || true)"; fi @@ -354,9 +415,10 @@ grade_one() { fi ;; file_exists) - local glob expect ident files f ok=1 + local glob expect count ident files f m ok=1 glob="$(jq -r '.glob' <<<"$g")" expect="$(jq -r '.expect // "present"' <<<"$g")" + count="$(jq -r '.count // ""' <<<"$g")" ident="$(jq -r '.identical_to // ""' <<<"$g")" files="$(find_files "$glob")" if [[ "$expect" == "absent" ]]; then @@ -364,6 +426,10 @@ grade_one() { GRADER_DETAIL="forbidden file(s) on disk matching $glob: $(head -3 <<<"$files" | tr '\n' ' ')"; return 1 fi [[ -z "$files" ]] && { GRADER_DETAIL="no file matching $glob was generated"; return 1; } + if [[ -n "$count" ]]; then + m="$(grep -c . <<<"$files")" + [[ "$m" -eq "$count" ]] || { GRADER_DETAIL="$m file(s) matching $glob, expected $count"; return 1; } + fi if [[ -n "$ident" ]]; then while IFS= read -r f; do cmp -s "$f" "$REPO_ROOT/$ident" || { ok=0; GRADER_DETAIL="$f is not byte-identical to $ident"; } @@ -400,10 +466,18 @@ grade_one() { GRADER_DETAIL="$miss lacks: $pattern"; return 1 ;; esac ;; - java_disclaimer) - local files f bad="" - files="$(find_files '*.java')" - [[ -z "$files" ]] && { GRADER_DETAIL="no .java files were generated"; return 1; } + disclaimer) + # The header text is the same in both APIs; only the comment prefix + # (// or #) differs, so a fixed-string match covers both. + local api ext files f bad="" + api="$(jq -r '.api // "jcsmp"' <<<"$g")" + case "$api" in + jcsmp) ext=java ;; + python) ext=py ;; + *) GRADER_DETAIL="unknown disclaimer api '$api'"; return 2 ;; + esac + files="$(find_files "*.$ext")" + [[ -z "$files" ]] && { GRADER_DETAIL="no .$ext files were generated"; return 1; } while IFS= read -r f; do head -3 "$f" | grep -qF 'AI-assisted code. Review before production use.' || bad="$f (missing disclaimer line)" head -4 "$f" | grep -qF 'See the verification checklist: solace-verification-checklist.md' || bad="$f (missing checklist pointer)" @@ -412,9 +486,36 @@ grade_one() { GRADER_DETAIL="$bad"; return 1 ;; compile) + local api poms pom reqs req proj venv mods rc n=0 log + api="$(jq -r '.api // "jcsmp"' <<<"$g")" + if [[ "$api" == "python" ]]; then + # One fresh venv per project, built outside the work dir from the + # project's own requirements.txt, so the verdict never depends on the + # .venv the subject model created. Every requirements.txt is a project + # root, so a two-project layout (Solace Suggested) is proven whole. + reqs="$(find_files requirements.txt)" + [[ -z "$reqs" ]] && { GRADER_DETAIL="no generated requirements.txt to install"; return 1; } + while IFS= read -r req; do + n=$((n + 1)); proj="$(dirname "$req")"; venv="$RUN_DIR/pyvenv-$n"; log="$RUN_DIR/py-compile-$n.log" + python3 -m venv "$venv" > "$log" 2>&1 \ + || { GRADER_DETAIL="python3 -m venv failed, log: $log"; return 2; } + bounded "$venv/bin/pip" install -q -r "$req" >> "$log" 2>&1; rc=$? + [[ "$rc" -eq 124 ]] && { GRADER_DETAIL="pip install for $req timed out after 600s, log: $log"; return 2; } + [[ "$rc" -eq 0 ]] || { GRADER_DETAIL="pip install -r $req failed (exit $rc), log: $log"; return 1; } + mods="$(find "$proj" -maxdepth 1 -type f -name '*.py' -exec basename {} .py \; | sort | paste -sd, -)" + [[ -z "$mods" ]] && { GRADER_DETAIL="no .py file beside $req"; return 1; } + ( cd "$proj" && find . -name '*.py' -not -path './.venv/*' -print0 \ + | xargs -0 "$venv/bin/python" -m py_compile ) >> "$log" 2>&1 \ + || { GRADER_DETAIL="py_compile failed in $proj, log: $log"; return 1; } + # Each generated script keeps its __main__ guard, so importing it + # resolves every solace.messaging name without connecting. + ( cd "$proj" && "$venv/bin/python" -c "import ${mods//,/, }" ) >> "$log" 2>&1 \ + || { GRADER_DETAIL="import smoke of $mods failed in $proj, log: $log"; return 1; } + done <<<"$reqs" + return 0 + fi # Every generated pom compiles, so a two-project layout (Solace Suggested) # is proven whole, not by whichever pom sorts first. - local poms pom rc n=0 log poms="$(find_files pom.xml)" [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } while IFS= read -r pom; do @@ -430,8 +531,19 @@ grade_one() { done <<<"$poms" return 0 ;; - maven_release_match) - local rel poms pom + release_match) + local api rel poms pom reqs req + api="$(jq -r '.api // "jcsmp"' <<<"$g")" + if [[ "$api" == "python" ]]; then + rel="$(resolve_solace_pubsubplus_release)" + [[ -z "$rel" ]] && { GRADER_DETAIL="could not resolve solace-pubsubplus info.version from pypi.org"; return 2; } + reqs="$(find_files requirements.txt)" + [[ -z "$reqs" ]] && { GRADER_DETAIL="no generated requirements.txt to check against release $rel"; return 1; } + while IFS= read -r req; do + grep -qxF "solace-pubsubplus==$rel" "$req" || { GRADER_DETAIL="$req does not pin the live solace-pubsubplus release $rel"; return 1; } + done <<<"$reqs" + return 0 + fi rel="$(resolve_sol_jcsmp_release)" [[ -z "$rel" ]] && { GRADER_DETAIL="could not resolve sol-jcsmp from repo1.maven.org"; return 2; } poms="$(find_files pom.xml)" @@ -483,18 +595,31 @@ grade_one() { GRADER_DETAIL="judge verdict unparseable after 2 attempts ($jfile)"; return 2 ;; live_verify) - local pom proj log rc + local api anchor build proj log rc if [[ "$LIVE_VERIFY" != "run" ]]; then LIVE_SKIPPED=1; return 0; fi - pom="$(shallowest_pom)" - [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to verify against the broker"; return 1; } - proj="$(dirname "$pom")" - [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $pom"; return 1; } + api="$(jq -r '.api // "jcsmp"' <<<"$g")" + if [[ "$api" == "python" ]]; then build=requirements.txt; else build=pom.xml; fi + anchor="$(shallowest "$build")" + [[ -z "$anchor" ]] && { GRADER_DETAIL="no generated $build to verify against the broker"; return 1; } + proj="$(dirname "$anchor")" + [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $anchor"; return 1; } log="$RUN_DIR/live-verify.log" LIVE_RAN=1 - if [[ -n "$TIMEOUT_BIN" ]]; then - ( cd "$proj" && "$TIMEOUT_BIN" 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + if [[ "$api" == "python" ]]; then + # The subject model has finished, so the values written here never + # reach it. The Python verify.sh refuses CLI args beside a config.json, + # and this file replaces any placeholder copy the model left behind. + ( umask 077 && jq -n --arg h "$OUTPUT_EVAL_BROKER_HOST" --arg v "$OUTPUT_EVAL_BROKER_VPN" \ + --arg u "$OUTPUT_EVAL_BROKER_USER" --arg p "$OUTPUT_EVAL_BROKER_PASSWORD" --arg t "$TRUST_STORE" \ + '{"solace.messaging.transport.host": $h, "solace.messaging.service.vpn-name": $v, + "solace.messaging.authentication.basic.username": $u, "solace.messaging.authentication.basic.password": $p} + + (if $t == "" then {} else {"solace.messaging.tls.trust-store-path": $t} end)' > "$proj/config.json" ) \ + || { rm -f "$proj/config.json"; GRADER_DETAIL="could not write $proj/config.json"; return 2; } + ( cd "$proj" && bounded bash ./verify.sh roundtrip ) > "$log" 2>&1; rc=$? + # A kept WORKDIR must not hold the broker password. + rm -f "$proj/config.json" else - ( cd "$proj" && bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + ( cd "$proj" && bounded bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? fi case "$rc" in 0) return 0 ;; @@ -559,20 +684,33 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do # Maven is only required when a selected case compiles, checks the pom, or # runs verify.sh. The live-verify banner prints before any case starts so the # operator sees the broker state in the first seconds, not at the end of the run. - needs_mvn=0; needs_live=0 + needs_mvn=0; needs_py=0; needs_live=0; needs_live_py=0 while IFS= read -r row; do cid="$(jq -r '.id' <<<"$row")" in_filter "$cid" || continue - jq -e '.graders | any(.type=="compile" or .type=="maven_release_match" or .type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_mvn=1 + jq -e '.graders | any((.type=="compile" or .type=="release_match" or .type=="live_verify") and (.api // "jcsmp")=="jcsmp")' <<<"$row" >/dev/null 2>&1 && needs_mvn=1 + jq -e '.graders | any((.type=="compile" or .type=="live_verify") and .api=="python")' <<<"$row" >/dev/null 2>&1 && needs_py=1 jq -e '.graders | any(.type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_live=1 + jq -e '.graders | any(.type=="live_verify" and .api=="python")' <<<"$row" >/dev/null 2>&1 && needs_live_py=1 done < <(jq -c '.[]' "$evals_file") if [[ "$needs_mvn" -eq 1 ]] && ! command -v mvn >/dev/null 2>&1; then - echo "ERROR: 'mvn' is not on PATH and a selected case carries a compile, maven_release_match, or live_verify grader." >&2 + echo "ERROR: 'mvn' is not on PATH and a selected case carries a JCSMP compile, release_match, or live_verify grader." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + if [[ "$needs_py" -eq 1 ]] && ! python3 -m venv --help >/dev/null 2>&1; then + echo "ERROR: 'python3' with the venv module is not on PATH and a selected case carries a Python compile or Python live_verify grader." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + if [[ "$needs_live_py" -eq 1 && "$LIVE_VERIFY" == "run" && -z "$TRUST_STORE" ]] \ + && [[ "$(tr '[:upper:]' '[:lower:]' <<<"$OUTPUT_EVAL_BROKER_HOST")" == tcps://* ]]; then + echo "ERROR: OUTPUT_EVAL_BROKER_HOST is tcps:// and a selected case runs a Python live_verify: export OUTPUT_EVAL_BROKER_TRUST_STORE (a directory holding the broker's CA certificate as a PEM file)." >&2 rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" exit 1 fi - if [[ "$needs_mvn" -eq 1 && -z "$TIMEOUT_BIN" ]]; then - echo "WARNING: neither 'timeout' nor 'gtimeout' is on PATH; mvn compile and verify.sh run unbounded (install GNU coreutils to cap them at 600s)." >&2 + if [[ "$needs_mvn" -eq 1 || "$needs_py" -eq 1 ]] && [[ -z "$TIMEOUT_BIN" ]]; then + echo "WARNING: neither 'timeout' nor 'gtimeout' is on PATH; mvn compile, pip install, and verify.sh run unbounded (install GNU coreutils to cap them at 600s)." >&2 fi if [[ "$needs_live" -eq 1 ]]; then LIVE_NEEDED=1