Mutiny

Behavioral fuzz-testing for AI agents.
Define what your agent must never do. Then prove it can't.


What	A fuzz engine you install into your agent project (`mutiny init` → `run` → `test`)
Why	Agents call tools that move money; prompts aren’t tests; chat evals miss tool-arg bugs
Different	Deterministic policy on tool-call traces (code proves) + minimize + permanent regressions — not LLM-as-judge, not “host our demo agent”

Adapter #1 ships today: OpenAI Agents SDK. Same Core for every future adapter — contributions welcome.

pip install mutiny-ai
mutiny init
mutiny run
mutiny test

Install package mutiny-ai; CLI command is mutiny. Do not use bare pip install mutiny or mutiny-sdk — those are unrelated projects.

Why Mutiny

You ship an agent that refunds money, deletes accounts, sends email. Your system prompt says “be careful.” Attackers — and ordinary users — don’t care about your system prompt.

What actually fails looks like this:

issue_refund({ "amount": 850, "approved": false })
delete_account({ "confirmed": false })

That’s not a bad answer. That’s a policy violation in action — and most eval stacks never see it.

Mutiny is built for AI agent testing that treats tool-call invariants like fuzz targets:

Pain	What Mutiny does
Prompt-only “safety”	Executable policy rules on real tool args
LLM-as-judge flakiness	Code proves violations on the trace
One-off red-team chats	Evolutionary behavioral fuzzing + minimize
“We fixed it… maybe”	Frozen regression testing under `.mutiny/tests/`

Keyword-shaped, product-shaped: agent safety, tool-call verification, OpenAI Agents SDK projects, regression suites you can replay in CI.

Features

Policy-as-code — Declare tool-use invariants in policy.yaml (deterministic operators, not vibes).
Behavioral fuzzing — Evolutionary search mutates attack conversations against your live agent path.
Proof on traces — Violations are evaluated in code on tool-call JSON — AI proposes, code proves.
Minimize + regress — Smallest reproduction, saved under .mutiny/tests/, replayed with mutiny test.
Adapter-first Core — Framework-independent engine; OpenAI Agents SDK adapter shipped; more via contributions.
CLI-first DX — mutiny init / run / test into your project (not “host our demo agent” as the product).
Optional Hosted UI — Observe campaign lineage and evidence in a browser when you sync with mutiny run --hosted (./scripts/dev.sh).

Screenshots & demo

Live Hosted captures from mutiny-sable.vercel.app plus a CLI run frame. Assets live under docs/assets/.

Demo loop

Static storyboard (fallback):

  mutiny init  →  mutiny run  →  violation (code proves)
                                      │
                                      ▼
                               minimize + save
                               .mutiny/tests/
                                      │
                                      ▼
                               mutiny test  →  PASS (after fix)

Hosted & CLI frames

View	Capture
Hero / landing
Campaigns
Violation / evolution graph
Policy
Tests
CLI `mutiny run`

The sample project runs offline without an API key. Hosted lineage (optional): ./scripts/dev.sh, then mutiny run --hosted.

Install

For your agent project:

pip install mutiny-ai
mutiny --help

Console script is mutiny. Do not use bare pip install mutiny or mutiny-sdk — those are unrelated projects.

Optional: install from git / source

pip install \
  "mutiny-core @ git+https://github.com/CodewithJha/mutiny.git#subdirectory=packages/mutiny_core" \
  "mutiny-openai-agents @ git+https://github.com/CodewithJha/mutiny.git#subdirectory=packages/mutiny_openai_agents" \
  "mutiny-ai @ git+https://github.com/CodewithJha/mutiny.git#subdirectory=packages/mutiny_cli"

Prerequisites

Tool	Required for	Notes
Python ≥ 3.11	Everything	Check with `python3 --version`
pip (or uv)	Agent-project install	`pip install mutiny-ai`
uv	Contributing / Hosted	Workspace sync + `uv run mutiny …`
git	Clone / optional git install
Node.js ≥ 20 + npm	Optional Hosted UI	Only if you run `apps/web` / `./scripts/dev.sh`

Contributors: clone + uv workspace

git clone https://github.com/CodewithJha/mutiny.git
cd mutiny
uv sync --extra dev
uv run mutiny --help

Contributors: editable `pip` (per package)

Root pip install -e . does not install the user CLI wheel (workspace root is not published). Install siblings together so deps resolve:

python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

pip install -e packages/mutiny_core -e packages/mutiny_openai_agents -e packages/mutiny_cli

mutiny --help

Optional Hosted (API + UI)

One command from the repo root:

./scripts/dev.sh
# API  http://127.0.0.1:8000
# UI   http://127.0.0.1:3000
# Health: GET http://127.0.0.1:8000/api/health

Or manually:

# terminal A — API
uv run python -m mutiny_api --host 127.0.0.1 --port 8000

# terminal B — UI (Node required)
cd apps/web && npm install && npm run dev

CLI campaigns default to local Core. Use mutiny run --hosted (or --hosted-url) for optional Hosted lineage (local execute + Hosted observe/sync — not remote adapter execution).

Hosted safety (M-PR8E): The Hosted API will not load/execute a customer .mutiny/adapter.py via project_path (returns 410 hosted_customer_execution_removed). Prefer mutiny run / mutiny run --hosted for customer projects. The bundled in_process_demo harness still works in Hosted as a labeled demo. MUTINY_ALLOW_PROJECT_EXEC is ignored.

Hosted auth (M-PR7 / P0-1 / P0-4): Loopback (127.0.0.1 / localhost / ::1) may omit MUTINY_API_TOKEN for local demo. Non-loopback binds require a non-empty token or python -m mutiny_api refuses to start. When set, protected routes require Authorization: Bearer (same value on the web process for UI rewrites). Auth does not sandbox customer Python. See SECURITY.md.

Hosted FS / rate limits / backup: Customer project_path is opaque on Hosted (no server-side project tree access — P0-3). The API applies in-process rate limits when auth is configured (P1-2; not distributed). Operators back up Hosted lineage with mutiny db backup / mutiny db restore (P2-6; not an HTTP API) — see SECURITY.md.

Quick start

Sample project (recommended first run)

cd examples/openai_support_agent
uv run mutiny init    # scaffolds .mutiny/adapter.py, policy.yaml, mutiny.yaml
uv run mutiny run     # local by default
uv run mutiny test    # replay saved regressions after a finding is saved

The sample agent uses a scripted offline model when OPENAI_API_KEY is unset — good for local smoke and CI. More detail: examples/openai_support_agent/.

Your own OpenAI Agents SDK project

cd /path/to/your-agent
uv run --directory /path/to/mutiny mutiny init --path .
# edit .mutiny/adapter.py  → AGENT_REF + POLICY_CONTEXT
# edit policy.yaml         → your tool names and rules
uv run --directory /path/to/mutiny mutiny run --path .
uv run --directory /path/to/mutiny mutiny test --path .

mutiny init writes:

File	Role
`.mutiny/adapter.py`	Wires Adapter #1 to your agent export
`policy.yaml`	Deterministic tool-use invariants
`mutiny.yaml`	Campaign defaults

With Hosted running (./scripts/dev.sh), use mutiny run --hosted for the API path (never selected from config alone).

Docs: docs/ — start at docs/README.md.

Troubleshooting

Symptom	What to check
`mutiny: command not found`	Run via `uv run mutiny …` from the repo after `uv sync --extra dev`, or activate the venv where you `pip install -e`’d the packages
`pip install -e .` at repo root fails	Expected — use `uv sync --extra dev` or the per-package editable installs above
Hosted UI won’t start	Install Node ≥ 20, then `cd apps/web && npm install`
Port already in use	Free 8000 (API) and 3000 (UI), or change the ports in `scripts/dev.sh` / Next
API looks down	`curl -sf http://127.0.0.1:8000/api/health` — expect JSON with a healthy status
Campaign can’t reach Hosted	Start `./scripts/dev.sh`, then `mutiny run --hosted`; or use plain `mutiny run` for local-only
Sample agent needs a live model	Set `OPENAI_API_KEY`; leave unset (or `MUTINY_SAMPLE_OFFLINE=1`) for the offline scripted model

Cold-start checklist: docs/COLD_START.md.

How it works

1. Connect your agent     → adapter (Adapter #1: OpenAI Agents SDK)
2. Declare invariants     → policy.yaml (deterministic tool rules)
3. Search                 → evolutionary campaign mutates attack conversations
4. Prove                  → code evaluates tool calls on the trace (not an LLM judge)
5. Minimize               → smallest reproduction that still violates
6. Freeze                 → permanent regression under .mutiny/tests/

Your agent project
        ↑
OpenAI Agents SDK Adapter   ← Adapter #1 (shipped)
        ↑
   Adapter Layer            ← future adapters (contributions welcome)
        ↑
   Mutiny Core              ← framework-independent engine
        ↑
   CLI (init / run / test)  ← primary
   Hosted API + UI          ← secondary (lineage / ops)

Architecture

Short version: one Core, thin adapters, CLI-first.

Layer	Location	Role
Core	`packages/mutiny_core/`	Policy oracle, campaign, fitness, minimize, regression
Adapter #1	`packages/mutiny_openai_agents/`	OpenAI Agents SDK → your local agent
CLI	`packages/mutiny_cli/`	`mutiny init` / `run` / `test`
Sample	`examples/openai_support_agent/`	Customer-style reference target
Hosted	`apps/api/`, `apps/web/`	Optional lineage / ops

Core must not import framework SDKs. New frameworks = new adapters on the same TargetAdapter port — Core stays put.

Deep dive: docs/ARCHITECTURE.md · docs/SYSTEM_DESIGN.md.

Adapters

Adapter	Status
OpenAI Agents SDK	Shipped (Adapter #1)
LangGraph	Wanted — roadmap / good first epic
CrewAI	Wanted
PydanticAI	Wanted
AutoGen	Wanted (later)
Localhost OpenAI-compatible HTTP	Wanted

Building an adapter? Start from packages/mutiny_core’s TargetAdapter port and open an issue so we can align on the interface. See CONTRIBUTING.md.

Commands

Command	What it does
`mutiny init [--path] [--force]`	Scaffold `.mutiny/adapter.py` + `policy.yaml` + `mutiny.yaml`
`mutiny run [--path] [--hosted] [--hosted-url] [--no-hosted] [--attestation]`	Local campaign by default; Hosted only with `--hosted` / `--hosted-url`
`mutiny test [id] [--path] [--failed] [--json] [--no-report]`	Replay `.mutiny/tests/` (PASS / FAIL / SKIPPED); optional single id
`mutiny db backup --out … [--db] [--overwrite]`	Consistent Hosted SQLite backup (operator; WAL-aware)
`mutiny db restore --from … [--db] [--force]`	Destructive restore into an explicit DB path (requires `--force` if dest exists)

Full flag reference: docs/CLI.md (from mutiny --help). Quick check: mutiny <cmd> --help.

Roadmap

Phase	Focus
Current scope	Engine-first Core · Adapter #1 (OpenAI Agents SDK) · CLI · minimize / regress · optional Hosted · PyPI `mutiny-ai`
Next	Contributor onboarding · more adapters · keep demo assets current
Beta	LangGraph / CrewAI / PydanticAI / HTTP adapters · policy packs · exportable reports
v1	Stable contracts · CI GitHub Action for regression replay · authenticated Hosted

Full detail: docs/ROADMAP.md · tracked as 🚀 Roadmap to v1.0.

Wanted contributor areas: framework adapters, policy packs, tests, Hosted a11y/UX polish, CI/DX.

Contributing

PRs welcome — especially adapters, policy packs, tests, docs, and a11y. Newcomers: start with a good first issue (fork → PR in about 30 minutes).

uv sync --extra dev
uv run pytest tests/unit -q

PR CI also runs offline integration + reliability (3.11/3.12), sample local CLI smoke, web typecheck/build, and package builds — see CONTRIBUTING.md.

Guide (30-min path): CONTRIBUTING.md
Good first issues: open labeled issues · catalog docs/GOOD_FIRST_ISSUES.md
Ideas & brainstorming: Discussions
Changelog / releases: CHANGELOG.md
Conduct / Security / Support: CODE_OF_CONDUCT.md · SECURITY.md · SUPPORT.md
Docs hub: docs/README.md
Issues: github.com/CodewithJha/mutiny/issues

If this engine is useful, a star helps other agent builders find it. Fork the repo and open a PR — we’re glad to help in the comments.

Contributors

Thanks to everyone who has contributed. New faces welcome — pick up a good first issue.

_CodewithJha

_Mr-Neutr0n

_kkkhs

Full graph: github.com/CodewithJha/mutiny/graphs/contributors.

FAQ

Is Mutiny on PyPI?
Yes — install with pip install mutiny-ai (PyPI). CLI command is mutiny. Do not use bare pip install mutiny or mutiny-sdk — those are other projects.

How do I run on Windows?
Create a venv, then activate with PowerShell .\.venv\Scripts\Activate.ps1 (or cmd activate.bat). Prefer WSL for contributor/uv workflows. Short FAQ: SUPPORT.md § Windows.

Which agent frameworks work today?
Adapter #1: OpenAI Agents SDK only. Others (LangGraph, CrewAI, PydanticAI, AutoGen, HTTP) are roadmap / contribution targets on the same TargetAdapter port.

Do I need an OpenAI API key?
No for the sample project — it uses an offline scripted model when the key is unset. Your own live agent may need whatever keys that agent already uses.

Is the Hosted UI required?
No. CLI is primary. mutiny run is local by default. Use mutiny run --hosted when you want Hosted to observe lineage in a browser (local execute + sync).

Is the bundled demo agent the product?
No. It’s a reference harness for docs and reliability. The product story is: install Mutiny into your agent project.

Can I point Mutiny at random websites?
No. Authorized testing only — local / in-process / localhost. Not an open-internet attack proxy.

Where do I ask for help?
SUPPORT.md. Security → SECURITY.md.

Known limitations

One shipped adapter (OpenAI Agents SDK); multi-framework support is intentional future work, not missing polish of a single kernel.
Published on PyPI as mutiny-ai (pip install mutiny-ai) — see docs/PUBLISHING.md.
Policy expressiveness is deliberately narrow (deterministic operators on tool calls) — not NL policies judged by an LLM.
Hosted is optional and secondary: observe-only for customer runs; non-loopback requires Bearer auth; in-process rate limits when auth is configured; multi-tenant cloud / SSO are not current scope.
Sample and reference demo agents use mock tools; they are not production payment or email systems.
See CHANGELOG.md for Unreleased / initial release notes.

Honest scope detail: docs/ROADMAP.md § Current scope.

Safety

Authorized testing only. Current targets are local projects / in-process or localhost with sandboxed mock tools. Mutiny is not an open-internet attack proxy.

Navigation Menu

Repository files navigation

Mutiny

Table of contents

Why Mutiny

Features

Screenshots & demo

Demo loop

Hosted & CLI frames

Install

Prerequisites

Contributors: clone + uv workspace

Contributors: editable `pip` (per package)

Optional Hosted (API + UI)

Quick start

Sample project (recommended first run)

Your own OpenAI Agents SDK project

Troubleshooting

How it works

Architecture

Adapters

Commands

Roadmap

Contributing

Contributors

FAQ

Known limitations

Safety

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
apps		apps
config		config
docs		docs
examples		examples
packages		packages
scripts		scripts
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
DESIGN.md		DESIGN.md
LICENSE		LICENSE
PRODUCT.md		PRODUCT.md
README.md		README.md
SECURITY.md		SECURITY.md
SUPPORT.md		SUPPORT.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
railway.toml		railway.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

Mutiny

Table of contents

Why Mutiny

Features

Screenshots & demo

Demo loop

Hosted & CLI frames

Install

Prerequisites

Contributors: clone + uv workspace

Contributors: editable pip (per package)

Optional Hosted (API + UI)

Quick start

Sample project (recommended first run)

Your own OpenAI Agents SDK project

Troubleshooting

How it works

Architecture

Adapters

Commands

Roadmap

Contributing

Contributors

FAQ

Known limitations

Safety

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Contributors: editable `pip` (per package)