End-to-end testing for OVOS skills.
OvoScope runs a full OVOS Core pipeline in-process with a FakeBus. It needs no server, no
audio stack, and no network. Load real skill plugins, send a test utterance, and check every
bus message that comes back: type, data, routing context, session state, and message order.
| Full pipeline | Runs real intent pipeline plugins (Adapt, Padatious, Fallback, Converse, Common Query) |
| Isolated | Config isolation strips user preferences, and the deterministic DEFAULT_TEST_PIPELINE excludes AI, persona, and OCP stages |
| Ordered assertions | Checks message type, data keys, routing context, and session state in order |
| Recording mode | Captures a live message sequence and saves it as a JSON fixture. No manual construction needed |
| Multi-turn | Pass a list of utterances to test full conversational flows |
| pytest fixture | The minicroft class-scoped fixture is auto-discovered through the pytest11 entry point |
| Inject skills | Use extra_skills={id: SkillClass} to load inline test skills without a PyPI entry point |
| Inject messages | Use MiniCroft.inject_message() to trigger non-utterance handlers (GUI events, timers, API calls) |
| Typed models | The optional ovoscope[pydantic] bridge adds schema-validated messages through ovos-pydantic-models |
pip install ovoscopeTo add typed message model support:
pip install ovoscope[pydantic]import unittest
from ovos_bus_client.message import Message
from ovos_bus_client.session import Session
from ovoscope import End2EndTest
SKILL_ID = "ovos-skill-hello-world.openvoiceos"
session = Session("test-session")
utterance = Message(
"recognizer_loop:utterance",
{"utterances": ["hello world"], "lang": "en-US"},
{"session": session.serialize(), "source": "A", "destination": "B"},
)
class TestHelloWorld(unittest.TestCase):
def test_intent_match(self):
End2EndTest(
skill_ids=[SKILL_ID],
source_message=utterance,
expected_messages=[
utterance,
Message(f"{SKILL_ID}.activate", context={"skill_id": SKILL_ID}),
Message(f"{SKILL_ID}:HelloWorldIntent",
data={"utterance": "hello world"}, context={"skill_id": SKILL_ID}),
Message("mycroft.skill.handler.start", context={"skill_id": SKILL_ID}),
Message("speak", data={"lang": "en-US"}, context={"skill_id": SKILL_ID}),
Message("mycroft.skill.handler.complete", context={"skill_id": SKILL_ID}),
Message("ovos.utterance.handled", context={"skill_id": SKILL_ID}),
],
).execute(timeout=10)OvoScope checks only the keys you list in expected.data and expected.context. It ignores
extra keys in the received message.
If you do not know the exact message sequence yet, record it from a live run:
from ovoscope import End2EndTest
test = End2EndTest.from_message(
message=utterance,
skill_ids=[SKILL_ID],
timeout=20,
)
test.save("tests/fixtures/hello_world.json") # anonymizes location data by defaultReplay the fixture in CI:
End2EndTest.from_path("tests/fixtures/hello_world.json").execute(timeout=10)OvoScope auto-registers the minicroft class-scoped fixture on install. You do not need
setUp/tearDown boilerplate:
class TestMySkill:
skill_ids = ["my-skill.author"]
def test_something(self, minicroft):
End2EndTest(
minicroft=minicroft,
skill_ids=self.skill_ids,
source_message=utterance,
expected_messages=[...],
).execute(timeout=10)OvoScope exposes composable pipeline stage lists so tests stay deterministic regardless of which AI plugins are installed on the host:
from ovoscope import ADAPT_PIPELINE, PADATIOUS_PIPELINE, FALLBACK_PIPELINE, PERSONA_PIPELINE
# Adapt only: fastest
mc = get_minicroft([SKILL_ID], default_pipeline=ADAPT_PIPELINE)
# Full intent chain
mc = get_minicroft([SKILL_ID],
default_pipeline=ADAPT_PIPELINE + PADATIOUS_PIPELINE + FALLBACK_PIPELINE)
# Opt in to persona for AI testing
mc = get_minicroft([SKILL_ID], default_pipeline=DEFAULT_TEST_PIPELINE + PERSONA_PIPELINE)DEFAULT_TEST_PIPELINE is the default when isolate_config=True. It includes all standard
built-in stages and leaves out persona, Ollama, OCP, and m2v plugins.
| Document | |
|---|---|
| docs/usage-guide.md | Start here: 8 test patterns with full worked examples |
| docs/ci-integration.md | Wiring OvoScope into GitHub Actions |
| docs/minicroft.md | MiniCroft and get_minicroft() reference |
| docs/capture-session.md | CaptureSession internals |
| docs/end2end-test.md | End2EndTest full parameter reference |
| docs/e2e-pipeline-harness.md | E2EPipelineHarness — testing a single pipeline plugin against raw bus messages |
| docs/intent-cases.md | File-based intent test cases (.intent.test) via register_intent_case_tests |
| docs/pydantic-integration.md | Typed message models with ovos-pydantic-models |
| docs/cli.md | ovoscope CLI — record/run/diff/validate/coverage/bus-coverage, plus ovoscope-setup |
| FAQ.md | Common questions and gotchas |
OvoScope is part of the OpenVoiceOS tooling suite:
- ovos-core: the OVOS assistant core that OvoScope tests skills against.
- ovos-workshop: the skill base classes that OvoScope loads and drives.
- ovos-bus-client: the message bus client behind
FakeBusandMessage. - ovos-test-harness: a companion test harness for OVOS components.
Developed by TigreGótico for OpenVoiceOS.
This project was funded through the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101135429.
PRs are welcome. See CONTRIBUTING.md for guidelines.
Parts of this project — including code, tests, and documentation — are developed with the assistance of AI coding agents, under human review before merge. Commit messages and pull request descriptions in the git history and CHANGELOG.md note when a change originated from an AI-assisted session, so contributors and users can see where AI assistance has been applied.