Repository navigation
MindEval 2 on master: multi-session simulator, and --judge_version for mindeval1/mindeval2 - #1
Conversation
zeppombal
left a comment
zeppombal
left a comment
There was a problem hiding this comment.
Thorough pass over the port. Each inline finding was reproduced at 978ae09 with a scripted litellm, and with litellm 1.98.1's real Anthropic transformation where noted. Most of mindeval/ is MindEval2 @ a18de8b apart from renames, so inline comments on that code are marked (inherited). It's new to this repo, but a follow-up PR or an upstream fix is fine for those.
Introduced by this PR
- 978ae09 re-seeds every member's arc, so the MindEval2 parity claims in the description no longer hold (
tests/test_judge.py:268). --judge_paramsisn't part of the judge identity (judge.py:345), and it isn't screened forapi_key/api_base, which then end up injudge.json(generate_judgments.py:24).- mindeval1 gives the judge information the counselor didn't have, and leaves out the memory it did have (
judge.py:263). Its rows also keep no answer text (judge.py:312). run_benchmark.shignores exit codes.
Inherited, worth a follow-up
- The preflight on
batch[0]can stall a judge directory for good (judge.py:404). - A stale
strayinMember.speakdrops a clean reply, or loops without bound (member.py:283). - A raw
reasoning_effortgoes to Anthropic and gets a 400 (llm.py:106). A prosecommit_movebreaks a Claude patient's session (member.py:238). - U+2028 in
verdicts.jsonlcauses re-judging (judge.py:335). Traces are decoded as strict UTF-8 (judge.py:88). - Compaction and evolution retry with no backoff (
arc.py:201). Moves an archetype avoids are still named in its prompt (member.py:400).
Smaller things (no inline comment)
release_locks()pops every lock in the process. Intest_both_versions_run_concurrently_on_one_run_dir, mindeval1'sfinallyreleases mindeval2's lock mid-pass, so the test doesn't really exercise per-version locking: with the lock regressed to one sharedjudge/.lock, it still passed 5 of 15 runs. Awith hold_lock(dir):that owns its handle fixes both.- mindeval2 now renders every pending prompt up front and keeps them all until the pass ends, where MindEval2 rendered in
_judge_one. In a synthetic run (400-char turns, 10 KB template) that measured about +250 MB at default scale and about +700 MB at--max_turns 60, and--limitdoesn't bound it. Hash each prompt, drop it, and render again inone(). - Stale names:
llm.py:6says--counselor-params, and.env.example:18points to a README section "Judging (MQM)" that is now "Run judgments". - Optional cleanups:
print_summaryreadssummary["replies"]/["sessions"]directly instead of viaunits_label, and the coverage/error code is duplicated across both judge classes.requires_counselor_system_promptandreference_sha256exist only sojudge_runcan run mindeval2-only checks, which also makesrender_prompt's no-system-prompt branch unreachable.- The CLI's
Literal[...]andjudge_version == "mindeval2"duplicateJUDGE_VERSION_DICT. - Making
JudgeErrora subclass ofInputErrorwould drop the translation inmain. - Test helpers are duplicated between
test_smoke.pyandtest_judge.pyand could move to aconftest.py.
- Minor inherited items:
speak()offers the todo tools withtool_choice: "auto"on every agenda turn, and all 50 profiles have a goal. The preflight and README only cover a namedtool_choice, and vLLM needs--enable-auto-tool-choicefor auto.compact_log's[:max_entries]keeps the oldest entries, so an overshoot drops the new session's entries,[open]ones included.--patient_paramsis merged shallowly, so anyextra_bodykey dropschat_template_kwargs.enable_thinking.- Patient calls don't strip an inline
…</think>the way the counselor path does (split_inline_trace). hold_lockreports anyOSError, e.g.ENOLCKon NFS, as "another mindeval process is running".
| def test_fingerprint_is_pinned(): | ||
| # The config of MindEval2 a18de8be (7962e7caed157571) with the YAML comments removed: the same data, | ||
| # other bytes. A change here means config/*.yaml or prompts/*.j2 changed. | ||
| assert load_config().fingerprint == "37868bb5e406de0d" |
There was a problem hiding this comment.
The config data is unchanged, but the benchmark isn't. The fingerprint hashes raw bytes (config.py:229) and is also seed material: arc_id_of (arc.py:44) and _run_id (session.py:181) include it, and every schedule draw and per-call seed derives from those. So 978ae09 re-rolls every member's arc under the same --seed 0. Compared with MindEval2 @ a18de8b on the same profiles:
- inter-session gaps differ for 50/50 members, the anchor hour for 50/50, the weekday for 43/50, the beats for 49/50 and the concealment schedule for 16/50; every episode seed differs;
- 48/50 session-1 member system prompts differ in the time line, e.g. m000119 goes from
It is Friday evening.toIt is Sunday evening..
So "MindEval2's behaviour is unchanged", "the member system prompts of all 50 profiles render identically" and the parity check in the description don't hold at the head commit. They look like they were checked before this commit, or rendered without time_context. If MindEval2 numbers need to carry over, I'd drop this commit; stale names in YAML comments are harmless. Otherwise, please say in the description that MindEval2 runs aren't reproducible from this repo. Separately, hashing the parsed config, or keeping the fingerprint out of the seed material, would stop a comment edit from re-rolling the benchmark.
There was a problem hiding this comment.
We kept the comment-free YAML but took the fingerprint out of the seed material: since e694a58, arc_id_of and _run_id draw from a pinned namespace equal to MindEval2's own fingerprint (7962e7caed157571), so schedules, time lines and per-call seeds match MindEval2 @ a18de8b again over all 50 profiles x 3 sessions, and a comment edit can't re-draw anything. 701ae29 adds that namespace to run.json's identity, so a run directory started at 978ae09 is refused on resume instead of mixing two drawings. The one prompt difference left is the deliberate avoided-move fix from the member.py:400 thread.
|
|
||
| # One directory holds one judge. The identity is pinned by the first verdict that succeeds, so a | ||
| # mistyped model or a bad template that never produced a verdict can simply be corrected. | ||
| identity = {"model": spec.model, "prompt_sha256": judge.prompt_sha256, "temperature": spec.temperature, |
There was a problem hiding this comment.
--judge_params is new in this port (MindEval2's judge had no params), but it isn't part of the identity. A later pass with different params is accepted into the same verdict set, and judge.json (L407-409) is rewritten with only the latest params. For example, pass 1 with --judge_params '{"reasoning_effort": "low"}' --limit 100, then pass 2 with '{"reasoning_effort": "high"}': exit 0, summary.json averages both configs, and judge.json reports high for all of them. That contradicts the docstring's "one directory never mixes two judges". run.json's IDENTITY already pins counselor.params and patient.params, so adding "params": spec.params here would be consistent, once secrets are kept out of params (see generate_judgments.py:24).
There was a problem hiding this comment.
|
|
||
|
|
||
| def main(output_dir: Path, judge_version: Literal["mindeval1", "mindeval2"] = "mindeval2", | ||
| judge_temperature: float | None = 1.0, judge_max_tokens: int = 16384, judge_params: dict = {}, |
There was a problem hiding this comment.
judge_params doesn't get the reserved-key check that generate_interactions.py:195-198 applies to --counselor_params/--patient_params; MindEval2 did that check in _params at parse time. Because judge_run writes "params": spec.params to judge.json:
--judge_params '{"api_key": "sk-…"}'leaves the key in plain text injudge/<version>/judge.json, against the README's "keys are never written to the outputs". The smoke test checks this only for the env-var path.- An
api_basein params is used wheneverMINDEVAL_JUDGE_API_BASEis unset, whilejudge.jsonrecords"api_base": null.
v1 users will do exactly this, since v1's --judge_model_api_params carried model and api_base. I'd suggest one shared check_params(flag, params) in utils.py for all three flags, and judge_params: dict | None = None like the other two; today --judge_params null is rejected while --counselor_params null is accepted.
There was a problem hiding this comment.
Done in cf39649 as suggested: one check_params(flag, params) in utils.py rejects model, messages, api_base and api_key for all three --*_params flags, and judge_params is dict | None = None, so --judge_params null is accepted like the other two. A test checks that an api_key is refused and never written to judge.json.
| # rebuilding from `turns` instead keeps v1's mapping (member "user", counselor "assistant"). | ||
| messages = [{"role": "user" if t["speaker"] == "member" else "assistant", "content": t["text"]} | ||
| for t in turns] | ||
| prompt = self.template.substitute(formulation=session["formulation"], |
There was a problem hiding this comment.
A question about what mindeval1 shows the judge. In v1, <member_details> held the same intake fields the clinician got as <member_context> (v1 prompts.py), so judge and clinician had the same information, and the few-shots reward using it ("your profile notes you've been averaging 5 or fewer hours…"). Here that changes:
<member_details>is the member's private formulation (backstory, and what they came in withholding), which the counselor never sees.- In sessions 2 and later the counselor worked from its
# Memorylog, which this judge never sees. So a counselor that correctly recalls session 1 can look like criterion 5's "hallucinate details not present in the conversation". That's about two thirds of sessions.
Would it be closer to v1 to put what the counselor had, i.e. the rendered memory from counselor_system_prompt, in <member_details>, with or without the formulation? Whichever way it goes, the choice deserves a line in the README; otherwise mindeval1 scores on v2 runs are hard to interpret.
There was a problem hiding this comment.
We went with what the counselor had, without the formulation: since 107f1ba, <member_details> is exactly the text the counselor received in {{ memory }} for that session (the "no earlier sessions" line in session 1, the compacted log after that), and mindeval1 no longer reads the formulation at all. The rubric and few-shots are unchanged, 04cefd7 pins the memory per member and session in the tests, and the README says so (178581c).
| if meta.get("finish_reason") == "length": | ||
| # A cut-off answer is not a verdict: whatever JSON it holds may be a fragment of the reasoning. | ||
| raise JudgeError(f"the judge hit its token limit ({spec.max_tokens}) before finishing its answer") | ||
| return {**judge.parse(text), "usage": meta.get("usage")} |
There was a problem hiding this comment.
v1 kept unparsed_judgment and thinking_trace for every judgment. A mindeval1 row here keeps only the seven numbers and usage; mindeval2 at least has justification. A failed parse leaves only the exception message in _error. So a mis-parse can't be audited or re-parsed without paying for a new judgment; for example, the regex will take the 2 from commentary like Clinical Accuracy & Competence: 2 issues stand out… written before the ratings. Could this keep the answer text (truncated) and meta.get("reasoning_content") on the row, and put a snippet of the unparseable text into _error?
There was a problem hiding this comment.
65cd3dc keeps the answer text and reasoning_content on every mindeval1 row and puts a snippet of the unparseable answer in _error, and becc156 splits an inline </think> trace off first, so the kept text is the verdict rather than the start of the reasoning. parse_judge_scores now takes each criterion's last match, so the Clinical Accuracy & Competence: 2 issues stand out… case gets the real rating (with a test).
| else: | ||
| # A server that does not honour a forced function answers in prose; the move is parsed | ||
| # out of it, and `via` records which mechanism produced it. | ||
| via, arguments, call_id = "content_json", content, None |
There was a problem hiding this comment.
(inherited) On the content_json path, the model's prose becomes the arguments of a synthetic commit_move call in the member's history. For a Claude patient, patient.yaml's claude entry sets force_move: false, so the call uses tool_choice: auto, which is exactly when prose answers happen. litellm can't convert that history into Anthropic's tool_use.input, so every later call in the session raises Failed to parse tool call arguments for tool 'commit_move' before any HTTP request, and the session ends as error. This happens whether or not the prose held a parseable move, and it also stops member_turn's "Call commit_move again" retry from ever reaching the model. I checked this with litellm 1.98.1's real Anthropic transformation. Storing valid JSON as the arguments (the parsed move, or "{}" when it didn't parse) would keep the history convertible.
There was a problem hiding this comment.
Fixed in 255a6a4: the synthetic commit_move call in the member's history now always carries valid JSON (the parsed move, or "{}"), while the trace's own arguments keeps the prose. Checked against litellm 1.98.1's Anthropic transform_request, which no longer raises on that history.
| dest = jdir / "verdicts.jsonl" | ||
| done: dict[tuple[str, str], dict] = {} | ||
| if dest.is_file(): | ||
| for line in dest.read_text(encoding="utf-8").splitlines(): |
There was a problem hiding this comment.
(inherited, one-line fix) Rows are written with json.dumps(..., ensure_ascii=False) (L393), which leaves U+2028, U+2029 and U+0085 raw, and read back with splitlines(), which splits on them. A verdict whose justification quotes a reply containing U+2028, like the smoke test's REPLY, becomes two undecodable halves. It's then skipped as torn, re-judged and re-appended on every rerun, and at temperature 1.0 it can change summary.json on a pass that should be a no-op. Using .split("\n"), as read_session and session.read_trace do, fixes it.
There was a problem hiding this comment.
Fixed in 135b9f5 with .split("\n") as suggested, plus a test that a verdict quoting U+2028 is read back whole and not re-judged.
| finished.""" | ||
| turns, system_prompt, formulation, finished = [], None, "", False | ||
| try: | ||
| lines = [x for x in trace.read_text(encoding="utf-8").split("\n") if x.strip()] |
There was a problem hiding this comment.
(inherited) Only FileNotFoundError is caught, and the decode is strict. The README says the judge can run while generate_interactions.py is still writing. A trace read mid-append, or left by a killed writer, can end inside a multi-byte character; ’ and — are common here with ensure_ascii=False. That raises UnicodeDecodeError out of main: a traceback and exit 1, which the CLI documents as "pending, rerun". A tight reader-vs-TraceWriter loop hit it 31 times in ~640k reads. A torn tail already means "not finished", so errors="replace", or catching UnicodeDecodeError as well, is enough. verdicts.jsonl has the same strict decode at L335.
There was a problem hiding this comment.
Fixed in 135b9f5: the trace read and the verdicts.jsonl re-read both decode with errors="replace", so a torn multi-byte tail reads as an unfinished line rather than a traceback, with a test for the trace case.
| transcript=_conversation(session.member_messages, "Member"), max_entries=max_entries) | ||
| seed = provider_seed(turn_seed(arc_seed, arc_id, episode_id, "compact_log")) | ||
| err = None | ||
| for attempt in range(3): |
There was a problem hiding this comment.
(inherited) compact_log, and evolve_situation at L234, make 3 back-to-back attempts with no sleep, and _save_state (L368) only runs after compaction. So a few seconds of 503s or refused connections from the shared patient server, right after a session ends, fail the member. On the next invocation set_aside_partial moves that finished session to partial/ and regenerates it: up to ~90 calls, and a different conversation. The member calls back off 5/15/45/60 s for exactly this reason ("losing a session costs far more than waiting"), so reusing member._pause here seems right.
There was a problem hiding this comment.
| memory_text: str | None = None, time_context: str | None = None) -> Member: | ||
| """The member for one session. `knobs` are this session's (concealment and affect move across | ||
| sessions); `memory_text` is the running log as the member carries it, None in session 1.""" | ||
| knob_lines = cfg.knob_lines(knobs, exclude=("verbosity",)) |
There was a problem hiding this comment.
(inherited) An archetype removes its avoided moves from the enum and the vocab check, but knob_lines (and secondary_note, L447) still tell the member to use some of them by name, contrary to the comment at L402-403. In 10 of the 50 session-1 prompts, backticked moves are missing from the member's enum. For example, people_pleaser m000144 avoids do_disagree, do_resist and do_express_frustration, yet the medium frustration_reactivity line (knobs.yaml:26-30) tells it "do_express_frustration when the situation itself is what is wearing you down, and occasionally do_disagree or do_resist". With guided decoding the label gets forced to another move while the words follow the instruction; with Claude it costs "unknown move" retries. Filtering these sentences, or at least the move names, by avoided would keep the prompt consistent with the tool.
There was a problem hiding this comment.
Fixed in c1e3a9d, which replaces a first attempt in 255a6a4 that dropped whole sentences: now only the avoided move names leave knob_lines and secondary_note (a list item leaves its list, a "(do_x)" disappears), so the drawn affect and the moves still allowed stay. A test renders every archetype against every knob line and the secondary note. At seed 0 this changes the prompts of 14 of the 50 members over 3 sessions, and it is the one deliberate prompt difference from MindEval2 @ a18de8b.
|
Thanks for such a careful review; the reproductions made every finding quick to confirm, and everything is fixed in this PR, the inherited bugs included, rather than left for a follow-up. On the seeds, we kept the comment-free YAML but took the fingerprint out of the seed material: arcs are now drawn from a pinned namespace equal to MindEval2's fingerprint, so every member's schedule, time context and per-call seeds match MindEval2 @ a18de8b again and a comment edit can never re-draw the benchmark (e694a58, 701ae29). For mindeval1, |
…r mindeval1/mindeval2 master becomes MindEval 2: the multi-session patient simulator (50 synthetic members, 3 sessions each, with a memory log carried between sessions) is ported here as the `mindeval` package. MindEval 1 stays as it was on the `legacy` branch, a copy of 1c17f9e. - Scripts run by path with jsonargparse and snake_case flags: mindeval/scripts/generate_interactions.py and generate_judgments.py; judge_prompts.py, utils.py and run_benchmark.sh keep their names; the README keeps v1's sections and adds What's New. - --judge_version picks the judge. mindeval2 (the default) is an MQM judge, one verdict per counselor reply, its prompt shipped in judge_prompts.py as a cacheable system message plus a user message; mindeval1 is v1's five-criterion rubric, kept byte for byte, one verdict per session. Each writes to judge/<version>/. The default judge is GPT-6.1 Sol. - Seeds and arcs are drawn from a pinned namespace equal to MindEval2's fingerprint (7962e7caed157571), so a config edit cannot re-draw the schedules; the namespace is part of run.json's identity. - Situation evolution between sessions sees the running log and the next session's archetype, concealment and affect; moves an archetype avoids are taken out of its prompt by name alone. - Judge passes resume, refuse to mix judges, and repair a torn last verdict; summary.json reports MQM incidents beside the mean; reads tolerate U+2028 and torn UTF-8. - Code under Apache 2.0; models and data (profiles, judge prompts) under CC BY-NC-SA 4.0. CI runs ruff, pytest and the codename check. - The paper (assets/paper.pdf) and the member realism annotations (data/member_realism_annotations.json). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
39dbe16 to
147545c
Compare
2 participants
Summary
masterbecomes MindEval 2: MindEval2's multi-session patient simulator (50 synthetic members, 3 sessions each, with a memory log carried between sessions) is ported here as themindevalpackage.legacybranch, a copy of 1c17f9e.python mindeval/scripts/generate_interactions.pyandpython mindeval/scripts/generate_judgments.py;judge_prompts.py,utils.pyandrun_benchmark.shkeep their names;--judge_version:mindeval2(the default) is an MQM judge, one verdict per counselor reply. It needs the counselor system prompt from the trace. Its prompt ships inmindeval/judge_prompts.pyas a system message and a user message, andMINDEVAL_JUDGE_PROMPTcan replace both with one file.mindeval1is v1's five-criterion rubric. The rubric and few-shots are kept byte-for-byte.<member_details>holds the counselor's memory for that session, which is what the counselor itself saw. It gives one verdict per session and does not read the counselor system prompt.judge/<version>/, so both can run on one run at the same time.7962e7caed157571), not from the config's bytes (fingerprint now5987fd25e55199ab). A config or prompt edit can't re-draw the schedules, and the namespace is part ofrun.json's identity. Later-session situations do differ from MindEval2's, by design: see the Member changes below.Judge prompt
.env.examplesetsMINDEVAL_JUDGE_MODEL=openai/gpt-6.1-sol(azure/<deployment>on Azure), andgenerate_judgments.pydefaults to its settings:--judge_params '{"reasoning_effort": "high"}', no temperature,--judge_max_tokens 32768. Any other model can be set inMINDEVAL_JUDGE_*.mindeval/judge_prompts.py, so nothing needs downloading.MINDEVAL2_JUDGE_SYSTEM_PROMPT). It's identical on every call, so Azure/OpenAI, vLLM and Anthropic can cache it, about 90% of each call's input. The user message holds only the example: the conversation so far and the reply (MINDEVAL2_JUDGE_USER_PROMPT). A test checks that putting the example back in the middle reproduces the validated single-message prompt byte for byte, apart from "Below is part of a conversation", which now reads "The next message holds part of a conversation".Behaviour differences from MindEval2 @ a18de8b
All of these came out of the review; everything else behaves the same, which a parity harness checks over 50 members × 3 sessions.
Member
speak().commit_move.</think>is split off every patient call.Counselor
reasoning_effortare no longer also forced through raw.Judge
judge.jsonis never missing beside a verdict.--judge_paramsis screened and part of the judge's identity.judge.jsonwas removed are refused (exit 2), not pinned to whichever judge resumes them.summary.jsonreportsmqm.incidentsbeside the mean: the share of sessions with a Critical reply, and with a Major or worse. The mean dilutes one severe reply in a long session; this doesn't.CLI
--patient_paramsmerges deeply.run_benchmark.shstops on a refused stage and quotes its path.Test plan
uv sync --locked,ruff check .,pytest -q(53 passed)run.json.seed_namespace. Since c67711d, the evolution prompt also carries the log and the next session's conduct, so later-session situations differ by design.Notes for reviewers
LICENSE(Apache 2.0, SWORD Health) is new to this repo and covers the code, including the v1 rubric text inmindeval/judge_prompts.py, so the v1 maintainer should confirm this. Models and data are CC BY-NC-SA 4.0, includingdata/profiles.jsonlandMINDEVAL2_JUDGE_SYSTEM_PROMPT/MINDEVAL2_JUDGE_USER_PROMPT.MINDEVAL2_*→MINDEVAL_*;mindeval2 run|judge→ the two scripts;judge/summary.json→judge/mindeval2/summary.json;MINDEVAL_JUDGE_PROMPTis optional;🤖 Generated with Claude Code
https://claude.ai/code/session_012mjnStrFaThvioxBC2vB2w