skill-eval

Deterministic regression testing for AI agent skills.

Most agent skills are just Markdown prompts — you can edit them, but you can't prove they still work. skill-eval turns "does this skill still behave correctly?" into a command with an exit code.

It scores scenario outputs against required and forbidden patterns, and runs the same scenarios twice — once with the skill loaded (with_skill) and once without (without_skill) — so you can measure whether the skill actually changes behaviour or just adds words.

Why this exists

Without skill-eval	With skill-eval
Skill quality is a matter of opinion	Every claim maps to a scenario with a pass/fail
Refactors silently break behaviour	Regression is caught by re-running scenarios
No idea if the skill adds value	with/without comparison shows the lift
Release readiness is vibes	Non-zero exit code blocks the release in CI

Install

python -m venv .venv && source .venv/bin/activate
pip install -e .          # runtime only (PyYAML)
pip install -e ".[test]"  # add pytest, for running the test suite

Requires Python 3.9+.

Quick start

# 1. Generate task cards + grading cards for each scenario
prepare-pressure-run tests/scenarios/my-skill/scenarios.yaml runs/iter-1

# 2. Have your agent produce runs/iter-1/<scenario>/<mode>/output.md
#    (give it task.md — never grading-card.md, or you contaminate the run)

# 3. Score the with_skill outputs
score-scenarios tests/scenarios/my-skill/scenarios.yaml runs/iter-1 --mode with_skill \
  --json with-grading.json --markdown with-report.md

# 4. Score the without_skill baseline
score-scenarios tests/scenarios/my-skill/scenarios.yaml runs/iter-1 --mode without_skill \
  --json without-grading.json --markdown without-report.md

with_skill should pass (exit 0) and without_skill should fail (non-zero). That gap is the evidence your skill does something.

Commands

Command	Purpose
prepare-pressure-run <scenarios.yaml> <run-dir>	Generate with/without task cards, grading cards, and run metadata.
check-pressure-run <run-dir>	List task cards still missing an output.md.
score-scenarios <scenarios.yaml> <outputs-dir>	Deterministic required/forbidden pattern scoring; JSON + Markdown reports.
validate-pressure-pack <skill-package>	Confirm a package ships QUALITY_TESTS.md, PRESSURE_TEST_REPORT.md, acceptance criteria, scenarios.
check-template-consistency <files...> --field <f>	Keep tests, auditors, templates, and examples aligned. --heading-count N for fixed structures.
run-codex-pressure <run-dir> --skill-package <p>	Collect real outputs via codex exec. Optional.

Scenario format

A plain YAML file, top-level scenarios list:

scenarios:
  - id: no_premature_release
    name: Skill must not release without evidence
    required_patterns:
      - "Release Gate"
      - "missing evidence"
    forbidden_patterns:
      - "可以直接发布"
      - "三年收入预测"

Outputs are discovered at <run-dir>/<id>/<mode>/output.md.

Forbidden matching is context-aware. A phrase like 可以直接发布 inside "不要直接发布" is not counted as a violation — the scorer inspects the preceding ~80 characters for negation markers. Use --strict-forbidden to disable this if you want literal matching.

CI usage

score-scenarios exits 0 only when every scenario passes, so it drops straight into a pipeline:

- run: score-scenarios tests/scenarios/my-skill/scenarios.yaml runs/iter-1 --mode with_skill

Testing this repo

pip install -e ".[test]"
python -m pytest -q      # 2 passed

Optional external dependency

run-codex-pressure shells out to the codex CLI. Without it, that one command degrades gracefully (per-task FAIL, exit 1) — the other five commands and the scoring engine are unaffected.

Related

Part of a three-repo family split from one skill suite:

• skill-eval (this repo) — the tool.
• skill-engineering-kit — the methodology: lifecycle, contracts, release gates.
• reality-project-skills — a business skill suite built with that methodology.

License

MIT — see LICENSE.