[Tanishq (tanishqbaweja)]

Follow-up to #117. Item 2 of the reporter's follow-up comment there (LiveMathematicianBench _load_json fails on Windows, and other open(...) reads and writes without an explicit encoding=, such as the conversation.json writers, are likely affected) was not part of #135, and it still reproduces on main.

Problem

The train/eval pipeline opens skills, benchmark data and run files without an encoding, so they use the locale codec: the ANSI code page on Windows (cp1252, GBK, ...). The DocVQA and OfficeQA rollouts, skillopt/datasets/base.py, the config loader (#124), the prompt loader and the Codex and Claude backends already pin UTF-8; the modules changed here do not. Reproduced on Windows 11, Python 3.12, cp1252, with a local fake OpenAI-compatible server standing in for the models:

1. The released skills cannot be loaded. ckpt/README.md documents python scripts/eval_only.py --skill ckpt/<bench>/gpt5.5_skill.md. Five of the six skills (all but DocVQA) contain U+201D, whose UTF-8 bytes end in 0x9D, a byte cp1252 does not define:

   File "scripts/eval_only.py", line 612, in main
   UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 911: character maps to <undefined>
   

   This happens before any model call. scripts/train.py --skill_init ckpt/searchqa/gpt5.5_skill.md stops the same way while reading the initial skill.
2. SearchQA scores a correct answer 0. process_one writes the prompts and the trajectory before scoring, inside a broad except. With Ω in a retrieved passage, <answer>Paris</answer> gets hard=0 and a fail_reason of error: 'charmap' codec can't encode character .... agent_ok is already true, so the systemic-failure check never fires. When the reply itself holds such a character (The passage names the city → <answer>Paris</answer>), the baseline logs hard=0 and the run then crashes writing results.jsonl.
3. An optimizer edit containing → crashes the training step with UnicodeEncodeError when reflect.py saves the analyst's patch. The trainer writes the same text to the merged patch, the ranked edits, the edit-apply report and the candidate skill afterwards.
4. Reflection crashes on UTF-8 trajectories. DocVQA and OfficeQA write conversation.json as UTF-8. A closing curly quote (U+201D) in one makes fmt_minibatch_trajectories raise UnicodeDecodeError, and that read sits before the analyst's try, so the reflect stage fails.
5. The SpreadsheetBench seed skill reaches every prompt garbled. skills/initial.md is read as cp1252, so its — and → arrive as — and →.
6. Raw LiveMathematicianBench files fail to load (_load_json, 0.2.0 wheel: missing prompt .md data files + Windows temp-dir cleanup crash (claude backend) #117): a question containing ∏ raises UnicodeDecodeError (byte 0x8f).

Change

• 95 text-mode open() calls (63 writes, 32 reads) in scripts/eval_only.py, skillopt/engine/trainer.py, skillopt/gradient/reflect.py, skillopt/optimizer/slow_update.py and the SearchQA, LiveMathematicianBench, SpreadsheetBench and ALFWorld adapter, rollout and loader modules now pass encoding="utf-8", as the DocVQA and OfficeQA rollouts already do.
• Files that an earlier release may have saved in the locale codec are read through a new skillopt/utils/text_io.py: skills (including skill_init and --skill, which may point at a previous run's best_skill.md), history.json, runtime_state.json, minibatch patches, results.jsonl, trajectories, target prompts, and the slow-update and meta-skill markers. It tries UTF-8 and falls back to the locale codec, the same migration skillopt_sleep/state.py does since fix(sleep): read and write state.json as UTF-8 #302, so a run started before upgrading still resumes. Reading such a file as UTF-8 only fails, loudly for most of these reads and silently for some: _load_runtime_state swallows the error, so the resume falls back to history.json, or starts over when no step has finished yet.
• results.jsonl is decoded line by line, because a resumed rollout appends UTF-8 records to a file the old release started; a whole-file fallback fails on such a file. A line valid in neither codec, such as a record cut off mid-write, is decoded with replacement characters instead of failing the whole read; a cut-off record then fails to parse like any other malformed line, and its item runs again.

The benchmarks change together on purpose: fixing only the trainer's skill read would make SpreadsheetBench worse on Windows. Given the seed skill with a real →, the codegen rollout's own prompt write raises inside its broad except (unexpected: UnicodeEncodeError ...) and the item scores 0 before the model is called.

Not changed: the ALFWorld rollout's conversation.json write, which #348 changes on the same line (reflection reads either encoding), and the config read in the vendored alfworld_envs.py.

Tests

New tests/test_run_file_encoding.py, following tests/test_sleep_state.py. A UTF-8 runner cannot show the defect by behaviour alone, so the pipeline tests run in a subprocess under -X warn_default_encoding with EncodingWarning raised as an error for SkillOpt's own modules (a dependency's warning cannot fail them):

• SearchQA run_batch with Ω in the context and → in the skill and the reply: hard == 1, the prompts, trajectory and results.jsonl are UTF-8, and a second call resumes without a model call.
• eval_only.main() with ckpt/searchqa/gpt5.5_skill.md (target stubbed): hard == 1.0, and the skill reaches the target prompt intact.
• One training epoch (target and optimizer stubbed) with that skill as skill_init and an optimizer edit containing →: skill_v0000.md equals the released skill, and the saved patch and the candidate skill hold the edit.
• fmt_minibatch_trajectories over UTF-8 files with curly quotes, → and Ω.
• LiveMathematicianBench load_items, SearchQA load_raw_items and SpreadsheetBench load_items on a question containing ∏.
• SpreadsheetBench process_one_codegen with the seed skill reaches the LLM phase (empty-code-block), and its system prompt keeps — and →.

Two in-process tests force the locale codec to cp1252, so the fallback runs on any OS: cp1252 runtime_state.json, history.json and a skill load intact, and run_batch returns both records of a results.jsonl holding one cp1252 line and one UTF-8 line without a model call; and a record cut off mid-character is rolled out again instead of failing the resume.

Results:

• Against main (source reverted, tests kept): 8 failed. The six subprocess tests stop with EncodingWarning: 'encoding' argument not specified at the open() under test (for SpreadsheetBench as its fail_reason); the two resume tests cannot import the helper.
• With the helper reduced to plain UTF-8 reads, the cp1252 resume test fails because _load_runtime_state returns None. With a whole-file fallback instead of per-line decoding, it fails on the mixed results.jsonl. With strict per-line decoding, the cut-off record test fails with UnicodeDecodeError.
• This branch: 8 passed.
• python -m pytest -q on Windows: 80 failed and 1835 passed on main; 80 failed and 1843 passed on this branch. The outcome of every existing test is the same, and the 8 new tests pass. The 80 failures come from this machine's setup (Windows, a CRLF checkout on an exFAT drive) and fail identically on main.
• ruff check on the changed files reports nothing new apart from one more E402 in eval_only.py, where every import follows force_utf8_stdout_stderr() by design.

Manual check (Windows 11, Python 3.12, cp1252, local fake OpenAI-compatible server, SearchQA with 2/2/2 synthetic items):

Run	main	this branch
eval_only.py --skill ckpt/searchqa/gpt5.5_skill.md	UnicodeDecodeError, no model call	hard=1.0
train.py --skill_init ckpt/searchqa/gpt5.5_skill.md	UnicodeDecodeError, no model call	completes
train.py, target replies The passage names the city → <answer>Paris</answer>	baseline hard=0, then UnicodeEncodeError writing results.jsonl	completes, test hard=1.0
train.py, optimizer edit contains →	UnicodeEncodeError saving the patch	completes

A one-epoch run started on main with café in its output path leaves a cp1252 runtime_state.json; this branch resumes it with --num_epochs 2 from step 2.

[Tanishq (tanishqbaweja)]

@microsoft-github-policy-service agree