Repository navigation
feat(testing): add a testing kit and reference adapters (ENG-506) - #16
Merged
folathecoder merged 2 commits intoNov 3, 2026
Conversation
agentling ships no model adapter, so a developer plugging one in needs a way to prove it follows the contract, a worked example to copy, and documentation of the contract itself. This adds all three. - agentling.testing: ScriptedModel replays scripted turns so an agent runs offline; check_model runs an adapter through a text turn and a tool turn and reports which of eight contract rules it broke; RecordingTransport, write_fixture and read_fixture record a backend once, with API keys redacted from headers, and replay it in tests. Importing it doesn't need httpx; recorded mode does. - Reference adapters in examples/providers/ for the Anthropic Messages API and native Ollama, alongside the OpenAI-compatible one, which gains extra_body. Each has a fixture and passes check_model. Malformed streams raise ModelServerError rather than escaping as KeyError or AttributeError. - examples/providers/record.py records an adapter against a live backend; examples/custom_model_generate_only.py wraps a backend that can't stream. - The six copied fake models in tests and examples now use ScriptedModel. - The README describes agentling as an agent engine, its quickstart runs with no API key, and a new "Write a model adapter" section is checked against the code by tests. Every README python block runs in the suite or says why it can't. - MIGRATING.md gives the 0.1 code and its 0.2 replacement for every removed or changed API. ROADMAP and CHANGELOG are updated.
Lines 1 and 2 of ollama_native.jsonl are now a live recording against Ollama 0.40.14 with qwen3:8b. Line 3, the 404, stays hand-written. The fixtures README says which lines were recorded and which were written by hand.
folathecoder
marked this pull request as ready for review
November 3, 2026 12:22
folathecoder
deleted the
folathecoder/eng-506-020-3-testing-kit-reference-adapters-and-docs
branch
November 3, 2026 12:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agentling no longer ships a model adapter, so "plug in any provider" needed a way to prove it. This adds
agentling.testing: aScriptedModelfor running agents offline and acheck_modelthat runs an adapter through a fixed script and reports which contract rule it broke. Three reference adapters (OpenAI-compatible, Anthropic Messages, native Ollama) ship as copy-paste files with fixtures that passcheck_modelin CI. The README, a newMIGRATING.md, the ROADMAP and the CHANGELOG are rewritten for 0.2.0, and the README and package description now present agentling as an agent engine.Ticket
ENG-506. Closes ENG-506. Dev plan: ENG-519. Test plan: ENG-520.
Changes
Testing kit (
src/agentling/testing.py, imported asagentling.testing)ScriptedModel(turns)replays one turn perstream()call: aChatMessage(streamed in pieces, as the loop sees a real model), a list ofDeltas, or an exception. It recordscallsandtools, and raisesAssertionError("ScriptedModel has no more turns")when the script runs out.check_model(factory, *, recorded=None, live=False, expects_tool_calls=True)runs a text turn and an echo-tool turn, plus a third turn that must raise when the fixture has one. It returns aConformanceReportwith eight rules: text reassembles,finish_reasonexactly once, usage present, tool call reassembles, onlyModelErrorescapes, inputs not mutated,provider_dataserialisable, and no emptyDelta.raise_for_failures()lists every failed rule.RecordingTransport,write_fixtureandread_fixturerecord HTTP exchanges one JSON object per line and replay them line by line, so an adapter sees the framing it saw live.authorization,x-api-keyandapi-keyheaders are redacted as they're captured.import agentling.testingworks withouthttpx; recorded mode andRecordingTransportraiseImportError("... pip install httpx")without it.httpxstays a dev dependency.Reference adapters (
examples/providers/, not in the wheel)anthropic_messages.py: lifts the system prompt, merges tool results and following user text into one user message, reads usage frommessage_startandmessage_delta(cache reads and writes counted in the input total), and keeps thinking blocks inprovider_data["anthropic"]["blocks"]so they're replayed in order on the next turn.ollama_native.py: reads NDJSON from/api/chat, re-encodes object arguments, recoverstool_namefor tool results from the earlier assistant turn, and reportstool_callsas the finish reason when Ollama says "stop" after a tool call.openai_chat.pygainsextra_body, merged into the request last.ModelErrorsubclasses, never retry, and raiseModelServerErroron a malformed stream.record.pyrecords an adapter against a live backend into a fixture.fixtures/holds one fixture per adapter plus an LM Studio one with empty tool call ids, and a README saying where every line came from.Examples and tests
examples/custom_model_generate_only.pywraps a backend that returns whole replies withdeltas_from_message.FakeModelintest_agent.py,_ScriptedModelintest_examples.py, and four examples) are replaced byScriptedModel.test_testing.py,test_providers_anthropic.py,test_providers_ollama.py,test_packaging.py(the wheel holds exactly the package; fixtures hold no key markers) andtest_readme.py(runs every README python block, and checks the adapter section's rules and error table against the code).Docs
retryabledefaults,provider_data,check_modelandrecord.py), and anagentling.testingreference. The "around 800 lines" claim is replaced by a link toagent.py.MIGRATING.md: the 0.1 code and its 0.2 replacement for each of 18 removed or changed APIs.[0.2.0] - Unreleased.pyproject.toml: description and keywords say "agent engine".How it was tested
uv run pytest: 538 passed, 12 skipped, in 2.2 s on Python 3.11.15, offline. The 12 skips are README blocks marked# not executedwith a reason: 4 need an API key, 7 are fragments that useagentorsessionfrom earlier examples, and 1 shows the protocol's shape.uv run ruff check src tests examples,ruff format --check,uv run mypy src tests examplesanduv run mypy --strict src: clean.No test lost: 325 test functions on
main, 402 now, every old name still present.Fourteen behaviours were broken on purpose, one at a time (rule checks, redaction, replay headers, Anthropic thinking blocks and cache tokens, Ollama finish reason and
tool_name,extra_body, malformed-stream handling, string turns); a test failed every time.A probe fed both new adapters malformed events and chunks; every case now raises a
ModelError.Test plan ENG-520: TC-01 to TC-77 automated and passing.
Live run (TC-78):
examples/providers/record.pyagainst Ollama 0.40.14 withqwen3:8bon macOS, 3 Nov 2026. All eight rules passed:The recording replaces lines 1 and 2 of
ollama_native.jsonl; line 3, the 404, stays hand-written. It confirmed two adapter choices against a real server: Ollama sendsidandfunction.indexon tool calls, and reportsdone_reason: "stop"after a tool call, which the adapter turns intotool_calls.OpenAI-compatible adapter, live:
record.py openai_chatagainst the same Ollama server's/v1endpoint withqwen3:8b: all eight rules passed. That recording was a check only and isn't committed.Not run (TC-79): live OpenAI, Anthropic and LM Studio recordings; no key or install was available. Those fixtures stay hand-written from the documented shapes, and
examples/providers/fixtures/README.mdlabels every line.Screenshots / recordings
None: no UI change.
Notes for reviewers
src/agentling/testing.py, thentests/test_testing.py, then one adapter with its test file.testing.pyis 634 lines against a planned ~350; the eight rule checks and the fixture validation are most of it. It's one module because the release NFR liststesting.pyas the only new file in the wheel.RecordingTransportredacts keys in headers, not in request bodies. A backend that takes its key in the body would leave it in the fixture; the docstring and fixtures README say to read a fixture before committing it, and the suite scans for common key prefixes.uv run mypy --strict srcto CI; ENG-507 (release); ENG-508 (Aldwin replacestests/fakes/model.pywithScriptedModel); ENG-521 (docs site).