Tool-Semantics

Know when an MCP change breaks the agent — not just the schema.

Behavioral compatibility testing for MCP tools and AI-agent interfaces.

---

Why Tool-Semantics?

AI agents do not call tools the way typed clients do. They choose tools from descriptions, invent arguments from schemas, and infer side effects from naming and prose. A change that remains JSON-Schema-valid can still:

• steer the model toward the wrong tool
• drop a required argument the model used to omit
• rename enums the model still emits
• quietly escalate from read-only to write/destructive behavior

Tool-Semantics captures normalized tool-interface snapshots and diffs them for structural and semantic risk — so teams can gate MCP and tool-API changes before agents ship broken workflows.

Compatibility layers

Layer	Question
1. Protocol	Can the client still speak to the server?
2. Schema	Are parameters and types still valid?
3. Tool selection	Will models still pick the right tool?
4. Execution	Do calls still succeed with prior argument patterns?
5. Intent / side effects	Did risk, confirmation needs, or outcomes change?

The MVP implements deterministic interface snapshots and structural comparison (layers 1–2, with warnings that point at 3–5), plus live MCP capture over stdio, Streamable HTTP, and legacy SSE. Model-based behavioral testing is available as an opt-in library; see the roadmap.

How it works

flowchart LR
  A[MCP / JSON manifest] --> B[Scanner]
  B --> C[Normalized snapshot]
  C --> D[Diff engine]
  D --> E[Compatibility report]
  E --> F[CLI / CI exit codes]

Loading

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Optional: scaffold the standard project layout
tool-semantics init

# Capture two interface versions
tool-semantics capture examples/github_server_v1.json -o .tool-semantics/v1.json
tool-semantics capture examples/github_server_v2.json -o .tool-semantics/v2.json

# Compare — exits 1 on breaking/critical changes
tool-semantics compare .tool-semantics/v1.json .tool-semantics/v2.json \
  --markdown-output .tool-semantics/report.md

See docs/project-layout.md for the .tool-semantics/ convention.

Demo

Example output

Comparing the included GitHub demo manifests surfaces removals, renames via addition, description drift, and newly required parameters:

Tool-Semantics: 1.0.0 → 2.0.0
┌──────────┬────────────────────────────┬─────────────────────────┬──────────────────────────────────────────┐
│ Severity │ Code                       │ Subject                 │ Change                                   │
├──────────┼────────────────────────────┼─────────────────────────┼──────────────────────────────────────────┤
│ breaking │ tool.removed               │ search_issues           │ Tool 'search_issues' was removed.        │
│ info     │ tool.added                 │ find_work_items         │ Tool 'find_work_items' was added; …      │
│ warning  │ tool.description_changed   │ create_issue            │ Tool description changed; …              │
│ breaking │ parameter.added_required   │ create_issue.repository │ Required parameter 'repository' was …    │
└──────────┴────────────────────────────┴─────────────────────────┴──────────────────────────────────────────┘
Result: breaking

Install (library)

pip install tool-semantics
# or from source: pip install -e .

from pathlib import Path
from tool_semantics.scanner import capture_manifest
from tool_semantics.diff import compare_snapshots
from tool_semantics.report import render_markdown

baseline = capture_manifest(Path("examples/github_server_v1.json"))
candidate = capture_manifest(Path("examples/github_server_v2.json"))
report = compare_snapshots(baseline, candidate)
print(render_markdown(report))
print("compatible:", report.is_compatible)

Exit codes

Code	Meaning
0	Compatible (no breaking/critical changes)
1	Breaking or critical changes detected
2	Input / capture / parse error

CLI reference

tool-semantics --version
tool-semantics capture <manifest.json> [-o .tool-semantics/snapshot.json] \
  [--provenance-output snapshot.provenance.json] [-v]
tool-semantics capture-mcp -o snap.json https://example.com/mcp
tool-semantics capture-mcp -o snap.json --http https://example.com/mcp
tool-semantics capture-mcp -o snap.json --sse https://example.com/sse
tool-semantics capture-mcp -o snap.json \
  [--provenance-output snap.provenance.json] -- python my_mcp_server.py
tool-semantics probe <snapshot.json> --probes <probes.json|yaml> \
  [--model] [--trials N] [--seed N] \
  [--json-output report.json] [--markdown-output report.md]
tool-semantics compare <baseline.json> <candidate.json> \
  [--json-output report.json] \
  [--markdown-output report.md] \
  [--config .tool-semantics.toml] \
  [--probes probes.json] [--probe-mode offline|model] \
  [--probe-target candidate|baseline|both] [--probe-trials N] \
  [-v]

• --verbose / -v logs paths, tool counts, and change totals to stderr (default Rich UX unchanged).
• --config loads ignore / policy / optional probe-gate rules; if omitted, .tool-semantics.toml in the cwd is used when present.
• capture-mcp speaks MCP JSON-RPC over stdio, Streamable HTTP, or legacy SSE; secrets-like keys are redacted by default. Bare URLs auto-detect HTTP then SSE — see docs/mcp-versions.md.
• probe runs offline behavioral probes by default; --model / --trials opt into model-backed evaluation (approved probes + API key) — see docs/probes.md.
• compare --probes (or [probes] in config) optionally gates CI on probe pass rates / metrics — see docs/config.md and docs/github-action.md.

Approved baselines and provenance

Capture the approved interface into a Git-tracked baseline. The snapshot is the contract that compare uses; review and commit it when an interface change is intentional.

tool-semantics capture examples/github_server_v1.json \
  -o .tool-semantics/baselines/github.json \
  --provenance-output .tool-semantics/baselines/github.provenance.json

The optional provenance sidecar records capture context and a digest of the snapshot. It is separate from the snapshot and never affects compatibility comparisons.

JSON reports include changes, is_compatible, and counts by severity.
Change-code catalog: docs/change-codes.md.
Ignore-config schema: docs/config.md.
GitHub Action: docs/github-action.md.
Publishing: docs/publishing.md.
Migration adapters: docs/adapters.md.

Optional risk field

MCP does not standardize risk. Tool-Semantics accepts an optional tool-level risk value (read_only, external_write, destructive, unknown). Missing values default to unknown (no false escalation). See the GitHub demo manifests for examples.

Offline probes

from pathlib import Path
from tool_semantics.scanner import capture_manifest
from tool_semantics.probes import Probe, ProbeKind, evaluate_probes

snapshot = capture_manifest(Path("examples/github_server_v1.json"))
report = evaluate_probes(
    snapshot,
    [
        Probe(
            id="search",
            intent="find issues",
            expected_tool="search_issues",
            required_params=["query"],
            kind=ProbeKind.POSITIVE,
        )
    ],
)
assert report.passed

Project layout

src/tool_semantics/   # scanner, models, diff engine, report, CLI
examples/             # demo MCP-style manifests
tests/                # pytest suite
docs/assets/          # README visuals

Roadmap

See ROADMAP.md for milestones. Live MCP supports stdio, Streamable HTTP, and legacy SSE (docs/mcp-versions.md). Model-backed probes / metrics / stability are available via the library API (docs/probes.md). Downstream consumers (myelinmesh, dogfood capture): docs/downstream.md.

Contributing

We welcome issues and PRs — especially documentation fixes, tests, and compatibility-rule ideas.

• Read CONTRIBUTING.md
• Follow the Code of Conduct
• Browse good first issues

Security

Do not auto-execute discovered MCP tools. See SECURITY.md for reporting guidance.

License

Apache License 2.0 — see LICENSE.