Skip to content

feat(testing): add a testing kit and reference adapters (ENG-506) - #16

Merged
folathecoder merged 2 commits into
mainfrom
folathecoder/eng-506-020-3-testing-kit-reference-adapters-and-docs
Nov 3, 2026
Merged

folathecoder merged 2 commits into
mainfrom
folathecoder/eng-506-020-3-testing-kit-reference-adapters-and-docs

Conversation

@folathecoder

@folathecoder folathecoder commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

agentling no longer ships a model adapter, so "plug in any provider" needed a way to prove it. This adds agentling.testing: a ScriptedModel for running agents offline and a check_model that runs an adapter through a fixed script and reports which contract rule it broke. Three reference adapters (OpenAI-compatible, Anthropic Messages, native Ollama) ship as copy-paste files with fixtures that pass check_model in CI. The README, a new MIGRATING.md, the ROADMAP and the CHANGELOG are rewritten for 0.2.0, and the README and package description now present agentling as an agent engine.

Ticket

ENG-506. Closes ENG-506. Dev plan: ENG-519. Test plan: ENG-520.

Changes

Testing kit (src/agentling/testing.py, imported as agentling.testing)

  • ScriptedModel(turns) replays one turn per stream() call: a ChatMessage (streamed in pieces, as the loop sees a real model), a list of Deltas, or an exception. It records calls and tools, and raises AssertionError("ScriptedModel has no more turns") when the script runs out.
  • check_model(factory, *, recorded=None, live=False, expects_tool_calls=True) runs a text turn and an echo-tool turn, plus a third turn that must raise when the fixture has one. It returns a ConformanceReport with eight rules: text reassembles, finish_reason exactly once, usage present, tool call reassembles, only ModelError escapes, inputs not mutated, provider_data serialisable, and no empty Delta. raise_for_failures() lists every failed rule.
  • RecordingTransport, write_fixture and read_fixture record HTTP exchanges one JSON object per line and replay them line by line, so an adapter sees the framing it saw live. authorization, x-api-key and api-key headers are redacted as they're captured.
  • import agentling.testing works without httpx; recorded mode and RecordingTransport raise ImportError("... pip install httpx") without it. httpx stays a dev dependency.

Reference adapters (examples/providers/, not in the wheel)

  • anthropic_messages.py: lifts the system prompt, merges tool results and following user text into one user message, reads usage from message_start and message_delta (cache reads and writes counted in the input total), and keeps thinking blocks in provider_data["anthropic"]["blocks"] so they're replayed in order on the next turn.
  • ollama_native.py: reads NDJSON from /api/chat, re-encodes object arguments, recovers tool_name for tool results from the earlier assistant turn, and reports tool_calls as the finish reason when Ollama says "stop" after a tool call.
  • openai_chat.py gains extra_body, merged into the request last.
  • All three map HTTP status to ModelError subclasses, never retry, and raise ModelServerError on a malformed stream.
  • record.py records an adapter against a live backend into a fixture. fixtures/ holds one fixture per adapter plus an LM Studio one with empty tool call ids, and a README saying where every line came from.

Examples and tests

  • examples/custom_model_generate_only.py wraps a backend that returns whole replies with deltas_from_message.
  • The six copied fake models (FakeModel in test_agent.py, _ScriptedModel in test_examples.py, and four examples) are replaced by ScriptedModel.
  • New: test_testing.py, test_providers_anthropic.py, test_providers_ollama.py, test_packaging.py (the wheel holds exactly the package; fixtures hold no key markers) and test_readme.py (runs every README python block, and checks the adapter section's rules and error table against the code).

Docs

  • README: agent engine positioning, a quickstart that runs with no API key, a "Write a model adapter" section (fields, the seven contract rules, an error table with retryable defaults, provider_data, check_model and record.py), and an agentling.testing reference. The "around 800 lines" claim is replaced by a link to agent.py.
  • MIGRATING.md: the 0.1 code and its 0.2 replacement for each of 18 removed or changed APIs.
  • ROADMAP: the "More models" section now says agentling ships no providers. CHANGELOG: entries under [0.2.0] - Unreleased.
  • pyproject.toml: description and keywords say "agent engine".

How it was tested

  • uv run pytest: 538 passed, 12 skipped, in 2.2 s on Python 3.11.15, offline. The 12 skips are README blocks marked # not executed with a reason: 4 need an API key, 7 are fragments that use agent or session from earlier examples, and 1 shows the protocol's shape.

  • uv run ruff check src tests examples, ruff format --check, uv run mypy src tests examples and uv run mypy --strict src: clean.

  • No test lost: 325 test functions on main, 402 now, every old name still present.

  • Fourteen behaviours were broken on purpose, one at a time (rule checks, redaction, replay headers, Anthropic thinking blocks and cache tokens, Ollama finish reason and tool_name, extra_body, malformed-stream handling, string turns); a test failed every time.

  • A probe fed both new adapters malformed events and chunks; every case now raises a ModelError.

  • Test plan ENG-520: TC-01 to TC-77 automated and passing.

  • Live run (TC-78): examples/providers/record.py against Ollama 0.40.14 with qwen3:8b on macOS, 3 Nov 2026. All eight rules passed:

    rule 1 (text reassembles): passed
    rule 2 (finish_reason once): passed
    rule 3 (usage present): passed
    rule 4 (tool call reassembles): passed
    rule 5 (only ModelError escapes): passed
    rule 6 (inputs not mutated): passed
    rule 7 (provider_data serialisable): passed
    rule 8 (no empty Delta): passed
    

    The recording replaces lines 1 and 2 of ollama_native.jsonl; line 3, the 404, stays hand-written. It confirmed two adapter choices against a real server: Ollama sends id and function.index on tool calls, and reports done_reason: "stop" after a tool call, which the adapter turns into tool_calls.

  • OpenAI-compatible adapter, live: record.py openai_chat against the same Ollama server's /v1 endpoint with qwen3:8b: all eight rules passed. That recording was a check only and isn't committed.

  • Not run (TC-79): live OpenAI, Anthropic and LM Studio recordings; no key or install was available. Those fixtures stay hand-written from the documented shapes, and examples/providers/fixtures/README.md labels every line.

Screenshots / recordings

None: no UI change.

Notes for reviewers

  • Suggested reading order: src/agentling/testing.py, then tests/test_testing.py, then one adapter with its test file.
  • testing.py is 634 lines against a planned ~350; the eight rule checks and the fixture validation are most of it. It's one module because the release NFR lists testing.py as the only new file in the wheel.
  • A turn that raises fails every rule that needed its output, whatever the exception type; the test plan's TC-12 case h was updated to match.
  • RecordingTransport redacts keys in headers, not in request bodies. A backend that takes its key in the body would leave it in the fixture; the docstring and fixtures README say to read a fixture before committing it, and the suite scans for common key prefixes.
  • Scope addition: the README opening and the PyPI description now use the "agent engine" positioning. That string is permanent on PyPI, so it lands here rather than with the docs site rewrite (ENG-521).
  • Follow-ups: add uv run mypy --strict src to CI; ENG-507 (release); ENG-508 (Aldwin replaces tests/fakes/model.py with ScriptedModel); ENG-521 (docs site).

agentling ships no model adapter, so a developer plugging one in needs
a way to prove it follows the contract, a worked example to copy, and
documentation of the contract itself. This adds all three.

- agentling.testing: ScriptedModel replays scripted turns so an agent
  runs offline; check_model runs an adapter through a text turn and a
  tool turn and reports which of eight contract rules it broke;
  RecordingTransport, write_fixture and read_fixture record a backend
  once, with API keys redacted from headers, and replay it in tests.
  Importing it doesn't need httpx; recorded mode does.
- Reference adapters in examples/providers/ for the Anthropic
  Messages API and native Ollama, alongside the OpenAI-compatible one,
  which gains extra_body. Each has a fixture and passes check_model.
  Malformed streams raise ModelServerError rather than escaping as
  KeyError or AttributeError.
- examples/providers/record.py records an adapter against a live
  backend; examples/custom_model_generate_only.py wraps a backend that
  can't stream.
- The six copied fake models in tests and examples now use
  ScriptedModel.
- The README describes agentling as an agent engine, its quickstart
  runs with no API key, and a new "Write a model adapter" section is
  checked against the code by tests. Every README python block runs
  in the suite or says why it can't.
- MIGRATING.md gives the 0.1 code and its 0.2 replacement for every
  removed or changed API. ROADMAP and CHANGELOG are updated.
Lines 1 and 2 of ollama_native.jsonl are now a live recording against Ollama 0.40.14 with qwen3:8b. Line 3, the 404, stays hand-written. The fixtures README says which lines were recorded and which were written by hand.
@folathecoder
folathecoder marked this pull request as ready for review November 3, 2026 12:22
@folathecoder
folathecoder merged commit 642fc17 into main Nov 3, 2026
2 checks passed
@folathecoder
folathecoder deleted the folathecoder/eng-506-020-3-testing-kit-reference-adapters-and-docs branch November 3, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant