whyfail

Evidence-based diagnostics for Python failures.

Python exceptions tell you what failed:

KeyError: 'user'

whyfail tells you why it most likely happened — by analyzing the actual failure: the traceback, the inspected runtime values, and the source around the failing line. No AI, no cloud, no network. It reads the failed program like a careful debugger would, records only what it can observe, and says "I don't have enough evidence" instead of guessing when it cannot tell.

Runtime values can contain credentials, so whyfail v2 routes every value that enters a diagnostic through SecretShield — pattern- and entropy-based secret detection — before anything is rendered. Useful crash diagnostics, without turning them into credential leaks.

KeyError: 'user'
================

What failed
-----------
response["user"] raised KeyError.

Execution path
--------------
  <module> (app.py:1)
    ↓
  fetch_user() (app.py:10)
    ↓
  load() (app.py:4)   ← raised here

Value history
-------------
  - `response` was assigned the return value of `fetch_user()` at line 3
    (`fetch_user()` returns the expression `payload` at line 10 — a static
    observation, not a runtime claim).

Broken assumption
-----------------
  - the code assumes the mapping accessed by `response` contains the key
    'user'
    Runtime evidence against it: the available keys are: 'account', 'status'

Likely cause
------------
The mapping does not contain the key 'user'.

Runtime evidence
----------------
  - the subscripted value is a dict.
  - value: {'account': {'id': 7}, 'status': 'active'}
  - available keys (2): 'account', 'status'
  - the failing subscript targeted key 'user'.

Expected vs actual
------------------
  Expected:
    the mapping accessed by `response` contains the key 'user'
  Actual:
    the available keys are: 'account', 'status'

Failure location
----------------
  app/users.py:4 in load()

  2 | def load():
  3 |     response = {"account": {"id": 7}, "status": "active"}
  4 |     return response["user"]
    |            ^^^^^^^^^^^^^^^^
What to investigate
-------------------
  • Verify what `response` really contains when this code runs — its
    producer does not include the key 'user'.
  • If the key can legitimately be absent, guard the access (e.g. an `in`
    check or `.get(...)` default) so the missing key is handled explicitly.

Confidence: high

That is not generated text. Every statement in it was observed: the value's type, its keys, the requested key, the exact source span, the frames that called the failing code, and the assignments that produced the failing value. The narrative is assembled from that evidence — nothing is invented to make the story read better.

Why it exists

Most error messages describe the immediate operation ("key not found"), not the reason it happened. Humans debug by looking at the runtime values around the crash — and so does whyfail, automatically, at the moment of failure while the values still exist.

whyfail is built around one rule, above all others:

It does not guess what happened. It analyzes what the failed program can actually tell us.

A mediocre tool says "your API probably changed". whyfail says:

The mapping does not contain the key 'user'.

I cannot determine why the key is missing from the available runtime evidence.

A conservative diagnosis is better than a confident but incorrect one.

Features

One engine, three interfaces — the CLI, the Python API, and the pytest plugin all share the same diagnostic engine.
Deep, but bounded, evidence — exception type/message, full chains, source context with AST analysis, safe inspection of runtime locals, function arguments, mapping keys, sequence lengths, dataclass fields and public attributes.
v3 root-cause model — each diagnosis separates the immediate failure (the operation that raised) from the likely root cause, states the assumption the code acted on, and pairs Expected vs Actual using only observed evidence.
Value provenance — where the failing value came from: assignment, parameter, call-site argument, import — traced through source and the live frames, bounded to a handful of hops and never quoting raw values.
Call-chain analysis — a compact outermost-first execution path to the failure, with the raise site marked, propagated frames distinguished, and recursive runs folded into an observed repeat count.
Execution story — an ordered, numbered narrative of how the program reached the failure.
Machine-readable output — whyfail run --format json python app.py emits the structured diagnostic as JSON for editors, CI, and scripts.
Per-exception diagnostics for KeyError, IndexError, TypeError, AttributeError, NameError/UnboundLocalError, ZeroDivisionError, ValueError, ImportError/ModuleNotFoundError, and AssertionError.
Exception chains — raise X from Y and implicit context are followed to the underlying failure that matters.
Honest source highlighting — the caret is derived from real AST + bytecode column information; when the failing expression cannot be pinned down, no misleading caret is drawn.
Confidence levels — high, medium, low, and explicit insufficient evidence statements. Speculation is labelled as speculation ("Possible explanations"), never as fact.
SecretShield-powered redaction (v2) — every runtime value that enters a diagnostic passes through SecretShield's pattern + entropy detection before it reaches any renderer. SecretShield is installed automatically as a dependency of whyfail — you never install or configure it separately.
Redaction by default — local values whose names suggest secrets (password, token, api_key, authorization, private_key, cookies, ...) and values that look like credentials (sk-…, ghp_…, BEGIN ... PRIVATE KEY, JWTs, Bearer …) are never printed; high-entropy tokens with innocuous names are caught by SecretShield.
Fully local, deterministic, offline — no telemetry, no network, ever.

Installation

pip install whyfail

Requires Python 3.10+. Installing whyfail also installs SecretShield automatically — it is a declared dependency, so you never run pip install secretshield yourself.

Releases

whyfail 3.0 — runtime root-cause analysis

whyfail 3.0
───────────

• Structured root-cause model: immediate failure vs. likely root cause
• Expected vs. actual, generated from runtime evidence
• Broken-assumption detection with the contradicting observation
• Value provenance (assignment / parameter / call-site history)
• Compact call-chain analysis with recursion folding
• Execution-story narrative
• Evidence-based "what to investigate" suggestions per failure
• JSON diagnostics for tooling (--format json)
• Bounded analysis: size/depth/frame budgets with explicit notes
• All v2 diagnostics, CLI, API, pytest integration and redaction preserved

whyfail 2.0 — SecretShield protection

whyfail 2.0
───────────

• SecretShield-powered runtime protection
• Automatic sensitive-value redaction
• Secure nested runtime inspection
• Safer diagnostic evidence
• Existing v1 diagnostics preserved
• CLI preserved
• Python API preserved
• pytest integration preserved

(Requires Python 3.10+ — SecretShield itself requires 3.10+.)

How to read a diagnosis: observed, inferred, possible, suggested

Every section in a whyfail diagnostic belongs to exactly one of four categories, and the layout tells you which is which:

Observed — statements that were inspected directly: the exception message, the runtime type of a value, a mapping's key list, a sequence length, the source snippet, the frame records. What failed, Runtime evidence, Expected/Actual rows, Value history lines, and the Execution path are observed (value-history lines that come from source are labelled "static observation, not a runtime claim").
Inferred — the engine's evidence-backed answer. The Likely cause and Broken assumption sections are inferences, and every one carries a confidence (high / medium / low).
Possible — genuinely plausible explanations the evidence cannot prove, under Possible explanations (for example "this may be a typo" only when a similarly named key/attribute actually exists).
Suggested — advice for what to investigate or change, under What to investigate. Suggestions are tied to the specific failure; there is no generic boilerplate advice.

When the evidence cannot support even an inference, whyfail says "Insufficient runtime evidence…" at low confidence instead of inventing a cause. Never reward sounding confident — reward being correct.

Quick start

1. The Python API

import whyfail

try:
    run_request()
except Exception as exc:          # note: the except block is where frames live
    diagnostic = whyfail.explain(exc)
    print(whyfail.format_diagnostic(diagnostic))
    # or machine-readable:
    diagnostic.to_dict()
    diagnostic.to_json()

whyfail.explain(exception) returns a structured Diagnostic — exception chain, failure location, observed facts, immediate-failure statement, expected vs actual rows, broken assumptions, provenance, execution story, evidence-based suggestions, confidence, redaction summary, and the call chain. Rendering is separate, so JSON/editor integrations need no engine changes.

2. The CLI

Run any Python program (or pytest) and get a diagnosis of the unhandled failure, after the program's own output:

whyfail run python app.py
whyfail run python -m mypackage
whyfail run pytest
whyfail run --format json python app.py   # machine-readable diagnostics

$ whyfail run python app.py
Traceback (most recent call last):      # ← unchanged original output
  ...
KeyError: 'user'

====================================
whyfail diagnosis
====================================
KeyError: 'user'
What failed
-----------
response["user"] raised KeyError.
...
Confidence: high

The child process runs as normally as possible: stdout, stderr, arguments, environment, and exit code are preserved. whyfail only observes; it never patches the running program. A failing program exits with its own exit code — a successful diagnosis never turns a failure into a success.

whyfail --help
whyfail --version

Machine-readable output (`--format json`)

whyfail run --format json python app.py

prints one JSON document to stdout — an array of {"context", "diagnostic"} entries (one per diagnosed failure, e.g. one per failed pytest test):

[
  {
    "context": null,
    "diagnostic": {
      "whyfail": 1,
      "schema": 2,
      "exception_type": "KeyError",
      "diagnosis": {
        "analyzer": "KeyError",
        "failure": "data[\"user\"] raised KeyError.",
        "expected": ["..."],
        "actual": ["..."],
        "broken_assumptions": [{"assumption": "...", "runtime": "..."}],
        "suggestions": ["..."],
        "provenance": [{"variable": "...", "origin": "...", ...}],
        "story": ["..."],
        "facts": [...],
        "cause": {"text": "...", "confidence": "high", ...},
        "notes": [...]
      },
      "call_chain": [{"function": "...", "role": "call", ...}],
      "location": { ... },
      "chain": [ ... ],
      "redacted_count": 0
    }
  }
]

Notes for tooling:

The --format option belongs to whyfail and must come before the command: whyfail run --format json python app.py (the spec-style ... --format json after the program path is passed to the child).
In JSON mode the only thing written to stdout is the JSON document. The child program's own stdout/stderr is captured and not forwarded, so jq/parsers always see pure JSON. Use plain whyfail run (text mode) when you want the program's output inline.
Exit status still mirrors the child: 0 = the program succeeded, non-zero = it failed. A successful run prints [].
The document is the exact structured model the engine produces (Diagnostic.to_dict()), already redacted and bounded — the CLI adds no extra fields.
The Python API offers the same data in-process via whyfail.explain(exc).to_dict() / .to_json().

3. The pytest plugin

No test rewrites needed:

pytest --whyfail

When a test fails, its diagnosis is printed alongside the normal failure output, with the failing test as context:

test_api.py::test_returns_user [call failed]
KeyError: 'user'

Failure analysis happens at the moment the exception is raised — inside pytest's own reporting hooks — so test frames and locals are still alive when the engine inspects them.

Supported diagnostics

Exception	What whyfail shows (from evidence)
`KeyError`	requested key, subject type, its keys, similar-key typo suggestions (only when a similar key actually exists)
`IndexError`	attempted index, sequence length, valid index range
`TypeError`	unsupported operands (from the message types + inspected operands), calling non-callables, subscripting non-subscriptables, argument-count mismatches, iterating non-iterables, `None` operands
`AttributeError`	object type, requested attribute, attributes that do exist, naming-mistake suggestions backed by a similar real attribute
`NameError` / `UnboundLocalError`	the missing name, scope, what is bound in that scope, bindings earlier/later in the source
`ZeroDivisionError`	the division expression and the runtime divisor when resolvable
`ValueError`	failed `int()`/`float()` conversions with the actual literal and runtime argument
`ImportError` / `ModuleNotFoundError`	missing module vs. missing symbol, importable parent prefixes, local shadowing files, circular-import wording
`AssertionError`	the asserted condition and the operand values that made it false; deliberate `raise AssertionError` is reported as such
anything else	an honest generic diagnosis — facts about the location, and an explicit insufficient evidence statement

Example output

IndexError: list index out of range
===================================

What failed
-----------
items[10] raised IndexError.

Value history
-------------
  - `items` was assigned a literal List value at line 2.

Broken assumption
-----------------
  - the code assumes `items` contains at least 11 element(s)
    Runtime evidence against it: observed length: 3

Likely cause
------------
Index 10 is out of range: the sequence has 3 element(s), so valid indexes are 0..2.

Expected vs actual
------------------
  Expected:
    index 10 falls inside the bounds of `items`
  Actual:
    the sequence has 3 element(s); with index 10, valid indexes would be 0..2

Failure location
----------------
  app/main.py:6 in main()

  4 | def main():
  5 |     items = ["a", "b", "c"]
  6 |     return items[10]
    |            ^^^^^^^^^
What to investigate
-------------------
  • Check how many elements `items` really holds when this code runs (it had
    3 at failure time).
  • If the length is not guaranteed, guard the access with a `len()` check
    or iterate instead of indexing blindly.

Confidence: high

With insufficient evidence, the output says so instead:

Likely cause
------------
Insufficient runtime evidence to determine the root cause from the available
local evidence.

At low confidence the narrative sections that would need runtime data are simply absent — the tool never fills them with guesses.

Why did this happen? — three questions

Every v3 diagnosis attempts to answer, in order:

What failed?          items[10] raised IndexError.
Why did it fail?      index 10 is outside a 3-element sequence.
Why was that          `items` was built with only 3 elements at line 2;
possible?             nothing re-populated it before the access.

The third level (the why was that possible layer) comes from provenance and call-chain evidence. When the evidence cannot support it, that layer is omitted rather than invented.

Architecture

Runtime failure
       │
       ▼
Failure Capture          capture.py   — exception chains + frames (live)
       │
       ▼
Evidence Collection      runtime.py   — bounded, guarded value inspection
       │                  redact.py    — conservative secret redaction
       ▼
Context Analysis         source.py    — source reading, AST ops, carets
       │
       ▼
Diagnosis Engine         engine.py + per-exception analyzers
       │                  ├─ analyzers (keyerror, indexerror, ...) fill
       │                  │    expected/actual/broken assumption/suggestions
       │                  ├─ provenance.py — bounded value history
       │                  ├─ callchain.py  — execution path + recursion folding
       │                  └─ story assembly
       ▼
Security Boundary        shield.py    — SecretShield pass over every string
       ▼
Diagnostic Model         models.py    — structured data, never raw values
       │
       ├── CLI renderer      renderer.py / cli.py (text + --format json)
       ├── Python API        api.py
       └── Pytest renderer   pytest_plugin.py

The pipeline runs inside the failing process — an except block, an installed sys.excepthook (CLI child), or pytest's reporting hooks — so the exception's frames and locals are alive when inspected. Rendered diagnostics are plain text built from the structured model.

Offline capture (`whyfail run`)

For whyfail run python app.py, the CLI launches the program with an observing sys.excepthook. On an unhandled failure the child itself runs the engine while frames are alive, redacts, and writes the structured result to a temporary JSON sidecar that only the parent CLI reads and renders. Normal behavior — output, traceback, exit code — is untouched.

For whyfail run pytest, the CLI launches pytest with the whyfail plugin loaded; the plugin writes the same sidecar protocol and the CLI renders it. Running pytest --whyfail directly renders inline instead.

Runtime Privacy

whyfail analyzes local runtime information (traceback frames, local variables, function arguments, inspected objects) to explain failures.

Because runtime variables can contain credentials — passwords, API keys, tokens, cookies, authorization headers — whyfail integrates SecretShield to detect and redact sensitive values before diagnostic information is rendered. The sanitization boundary sits inside the engine: every string that enters the structured Diagnostic passes through SecretShield first, so the CLI, the Python API, and the pytest plugin all receive the same already-safe evidence — there is no per-renderer redaction to forget.

SecretShield is installed automatically as part of whyfail. It is a declared dependency; users never install or configure it separately, and security is on by default with no opt-in flag.
No runtime data is sent to a remote service. Detection is fully local; whyfail and SecretShield perform no network I/O.
Structural evidence is preserved. Redaction hides sensitive values, not the shape of the data: response is a dict, its key list, sequence lengths, and types are still reported while values are protected.
Fail-closed. If SecretShield is unavailable or errors, whyfail falls back to its own conservative rules (sensitive variable names and unmistakable credential shapes) and notes the fallback in the diagnostic — it never emits raw values just because the primary layer failed.
Redaction cannot guarantee detection of every possible secret. Both layers are heuristic; a novel secret format with an innocuous variable name may evade detection. Treat diagnostics the way you treat tracebacks.
Source lines are displayed like tracebacks. Context lines that contain credential literals are masked; the failing source line is shown verbatim, exactly as Python's own tracebacks show it.

Privacy and local-only guarantees

No network. Ever. The engine performs no I/O beyond reading local files (source) and the local import path. There is no telemetry, no external service, no analytics.
No raw values leave the process. Values are summarized and redacted before being recorded; the sidecar contains only rendered text.
whyfail never alters process behavior. Importing whyfail does not wrap or filter your program's stdout/stderr: SecretShield's stream guardians, which it installs on import, are disabled again by whyfail so the traced program's output stays byte-for-byte identical. Protection happens on the diagnostic data itself. (If you want SecretShield's stream-level protection for your own prints, import secretshield directly.)

Limitations

whyfail provides evidence-based likely explanations, not guaranteed root-cause analysis. The cause section states what the evidence supports, at a stated confidence; speculation appears only under "Possible explanations".
Provenance is static and best-effort. The value history comes from reading assignments, parameters and call sites in the source plus the live frame records; it is deliberately bounded (a handful of hops) and is not a full dataflow or debugger trace. When it cannot establish an origin it says nothing rather than guessing.
Only failing programs are analyzed. v3 analyzes live runtime failures (API, CLI, pytest). A static "explain this source file" mode and automatic code-fixing remain future work — the model already carries structured fix suggestions.
Traceback frames are capped (500) and the call chain is folded and summarized past 24 entries; a folded recursion shows an observed repeat count, and truncation is stated in the diagnostic notes.
Analysis happens at the moment the exception is alive. If a program overrides sys.excepthook, whyfail run cannot capture that failure.
The interpreter must be able to import whyfail (the CLI runs children with the interpreter it is installed into).
A pathological object whose __repr__ loops forever without raising cannot be interrupted from inside the same process (standard for any diagnostic tool); all raising/misbehaving cases are fully guarded.
Source files are read from disk at analysis time, so source edits made between a crash and the analysis can make code context stale.
Secret redaction is heuristic and not perfect (see Runtime Privacy). whyfail hides sensitive values that SecretShield's patterns and entropy detection — plus whyfail's own name rules — recognize; it cannot guarantee that an unrecognized secret format never appears.
Python 3.10+ is required (whyfail 2 depends on SecretShield, which requires 3.10+).

Development

git clone https://github.com/Sam3360/whyfail
cd whyfail
pip install -e ".[dev]"
python -m pytest          # full suite incl. golden + limits tests

Project layout

pyproject.toml           packaging / metadata / entry points
src/whyfail/             the package (engine, analyzers, renderers, CLI, plugin)
tests/                   unit + integration + false-positive tests
  golden/                realistic failing programs, asserted structurally
benchmarks/              overhead comparison script (dev only)
.github/workflows/ci.yml CI (Linux, Python 3.10–3.14, build check)

Testing

The suite covers every supported exception type, runtime inspection, source extraction and highlighting, exception chaining, the v3 narrative layers (expected/actual, broken assumptions, provenance, call chains, recursion folding, JSON output), golden programs asserting structured fields, resource limits (huge mappings, deep nesting, cyclic values, giant tracebacks), confidence, redaction, hostile/recursive/huge values, the CLI (exit codes, output preservation, -m, --format json and pytest invocations), the pytest plugin, and — importantly — false-positive tests proving the engine refuses to invent causes when the evidence does not support them.

Performance

The engine only runs when a diagnostic is requested, so normal program performance is unaffected. Each analysis runs under hard budgets — frame capture (≤500), call-chain entries (≤24, middle summarized), provenance hops (≤5), rendered value size/depth/width, per-snippet line length, and total traceback walking — so a pathological input yields a short diagnosis with an explicit note, never a hang or an unbounded dump. python benchmarks/bench_diagnose.py compares plain execution against a diagnose-then-continue loop for a rough overhead feel.

Releasing

python -m build
twine upload dist/*     # after review

License

MIT — see LICENSE.

Navigation Menu

Repository files navigation

whyfail

Why it exists

Features

Installation

Releases

whyfail 3.0 — runtime root-cause analysis

whyfail 2.0 — SecretShield protection

How to read a diagnosis: observed, inferred, possible, suggested

Quick start

1. The Python API

2. The CLI

Machine-readable output (`--format json`)

3. The pytest plugin

Supported diagnostics

Example output

Why did this happen? — three questions

Architecture

Offline capture (`whyfail run`)

Runtime Privacy

Privacy and local-only guarantees

Limitations

Development

Project layout

Testing

Performance

Releasing

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
benchmarks		benchmarks
src/whyfail		src/whyfail
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
setup.py		setup.py

Latest commit

History

Folders and files

Repository files navigation

whyfail

Why it exists

Features

Installation

Releases

whyfail 3.0 — runtime root-cause analysis

whyfail 2.0 — SecretShield protection

How to read a diagnosis: observed, inferred, possible, suggested

Quick start

1. The Python API

2. The CLI

Machine-readable output (--format json)

3. The pytest plugin

Supported diagnostics

Example output

Why did this happen? — three questions

Architecture

Offline capture (whyfail run)

Runtime Privacy

Privacy and local-only guarantees

Limitations

Development

Project layout

Testing

Performance

Releasing

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Machine-readable output (`--format json`)

Offline capture (`whyfail run`)