Repository navigation
feat(orchestrator): pick task skills by project traits, in shadow - #1497
Draft
edwinyjlim wants to merge 2 commits into
Draft
edwinyjlim wants to merge 2 commits into
edwinyjlim wants to merge 2 commits into
Conversation
Orchestrator runs resolve each task's skills two ways. The exact-match pick (the id itself, else its group's variant for the run's framework) decides. The criteria resolver in `@shared/skill-resolution` picks beside it from menu tags and the run's project traits, and each disagreement is logged as `orchestrator skill resolver disagreement`. With the `wizard-agent-dag` flag on, the criteria pick decides and the exact match is its fallback. `@services/project-traits` holds the run's project traits from static detection: the framework, and `callsLlmApi` when an LLM SDK is a dependency. `traitsToSkillTags` maps the framework to its `framework:<id>` tag, and the program runner hands the tags to the orchestrator through `RunConfig.traitTags`. Skill-menu entries carry `title`, `description` and `tags`, and bundle variants inherit the bundle's title and description. Every event after the menu loads carries `context_mill_version`, and the terminal `setup wizard finished` event carries the run's wizard flags. `fixtures/context-mill/` holds a context-mill build's skill menu, agent menu and 41 agent prompts; `pnpm record:context-mill-fixtures` re-records them from a release. The equivalence test checks that both resolvers pick the same skill for every prompt skill and framework. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
Flows in the program contract read project traits, and the contract imports no service, so the vocabulary types live there and `@services/project-traits` produces their values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Orchestrator runs pick each task's skill two ways: the exact-match pick used today, and a criteria pick from the skill menu's tags and the project's traits. With
wizard-agent-dagoff, the exact match decides and the criteria pick runs in shadow. With it on, the criteria pick decides and the exact match is its fallback. The flag does not exist in PostHog yet, so every run takes the off path.Stack
The four PRs, in merge order:
wizard-agent-dagflag that decides which pick installs.main. Reads C1's tags but works without them. D1 builds on its traits service, and W2 on its traits and flag.wizard-agent-dagon, runs use them instead of the fetched prompts and the planner's graph.W1, D1 and W2 merge in that order and ship together in one wizard release, with
wizard-agent-dagoff. A later PR, W3, removes the flag and the paths it replaced after a week at 100%.How a task's skill is picked
flowchart TD task["Task's skill id<br/>from its prompt's frontmatter"] --> legacy & criteria menu["Skill menu<br/>tags, group, framework, default"] --> legacy & criteria traits["Project traits as tags,<br/>such as framework:nextjs"] --> criteria legacy["Exact-match pick<br/>the id, else its group's variant<br/>for the run's framework"] criteria["Criteria pick<br/>the id, else skills carrying its tag<br/>and the framework tag, ranked by<br/>default, then menu order"] legacy & criteria --> flag{"wizard-agent-dag"} legacy & criteria -. "when they differ" .-> event(["orchestrator skill resolver<br/>disagreement event"]) flag -- off --> off["Installs the exact match"] flag -- on --> on["Installs the criteria pick,<br/>else the exact match"]The framework's reference skill (
integration-<variant>) goes through the same picker. A menu without tags, such as v1.59.0, resolves the same: each entry'sgroup,frameworkanddefaultcount as tags.Project traits
@services/project-traitsreads the traits from static detection.traitsToSkillTagsturns them into the tags the criteria pick uses, and the program runner passes those to the orchestrator throughRunConfig.traitTags.frameworknextjsframework:nextjscallsLlmApiyeswhen an LLM SDK is a dependency; absent otherwiseMenu and fixtures
SkillEntrycarriestitle,descriptionandtags. Bundle variants inherit the bundle's title and description.fixtures/context-mill/holds a context-mill build's skill menu, agent menu and 41 agent prompts.pnpm record:context-mill-fixtures [base-url]re-records them from a release.Telemetry
orchestrator skill resolver disagreementcontext_mill_version$feature/<key>setup wizard finished, one per wizard flag, so run outcomes split by flagTest plan
pnpm typecheckpnpm lint: no errors, and no new warnings in changed filespnpm test: 258 files, 3,822 testsCONTEXT_MILL_MENU=<path>), which has no tagsbuildVersion: dev)wizard-agent-daggoes to 50%, where it holds until a judgment call takes it to 100%🤖 Generated with Claude Code