[binary-knight]

Groundwork for 1.0: a written compatibility policy, with a test behind each promise in it.

docs/COMPATIBILITY.md covers:

• the commands and their flags, and exit codes 0 (ran, gates passed), 2 (a gate failed) and 1 (could not run)
• the report format, schema_version 1, where fields can be added but not removed, renamed or retyped
• the scoring: weights of 25/15/8/3/0, the cap at 100, finding IDs that are never reused, and the two findings whose severity follows their evidence (FS-001, PRIV-003)
• the GitHub Action's inputs and outputs
• how a score can still change between minor versions: a new check or a corrected one, each recorded by version and in the changelog

verify, submit and benchmark are named experimental, and their help text says so. Human-readable output, the JSON of matrix, combine and verify, and the Python modules are outside the promise.

Tests. tests/test_compatibility.py checks every report in examples/ against the new schema, and checks that the schema declares every field they use. It also covers freshly written reports and assert results, the weights, which findings have variable severity, exit codes, the Action's inputs and outputs, and each stable command's flags against tests/fixtures/cli_surface.json. A missing command or flag fails. A new flag has to be added to the fixture.

Report changes. Reports now record agentsec_version beside probe_version, because scoring lives in the package and a scoring change was not visible in a report. Reports written by earlier versions do not have the field, and --baseline treats its absence as no change. --baseline and agentsec history note when either version differs.

A correction. The README and the gate's own note said a probe version difference fails --baseline. Since the gate was added in #6, the code has failed only when a finding is new or more severe, and passed with a note otherwise. The README and the note now describe that, and the policy promises it.

Also: JSON Schemas for the blast-radius report and the assert result in agentsec/data/, agentsec --version, PyPI classifiers (Development Status 4 - Beta), and jsonschema as a test dependency. Locally 512 tests pass and 13 skip without network.

🤖 Generated with Claude Code