Repository navigation
feat(ENG-016): Kappa-style consensus over implementation directions (Phase 1) - #13
Merged
Merged
Conversation
…Phase 1)
Implements PRD-001: at the architecture/system-design stage the senior
technical roles independently rate candidate directions; a chance-corrected
inter-rater agreement coefficient gates adopt-vs-tie-break. Phase 1 ships
advisory (records the round + score, doesn't block) behind CONSENSUS_ENABLED.
- core (pure, heavily tested):
- agreement coefficients — Krippendorff's alpha (primary; nominal/ordinal/
interval over a units×raters matrix), Fleiss' kappa (single-pick) and
Kendall's W (ranked). Degenerate cases (no variation / unanimity) map to 1,
never NaN.
- `decide()` — aggregate winner (vote tally / Borda) gated on
`coefficient ≥ threshold` with a strict no-tie check; the coefficient is a
confidence signal feeding the rule, not a hard inference (PRD caveat).
- zod schemas: ConsensusCandidate/Rating/ConsensusConfig/Round/Outcome.
- agents: `proposeDirections` (2–4 distinct candidates) + `rateDirections`
(one rater, independent, structured output constrained to candidate ids).
- server: durable `directionConsensus` workflow — generate → rate the roles
**in parallel (barrier)** → decide → branch; tie-break = human escalation via
the `architecture_decision` gate (signal + client fn + API endpoint mirroring
approveRoadmap). Budget-enforced per rater, audited, and emitted to the
decision log (ENG-014). No-op when <2 viable candidates or runtime/budget
unavailable. Config-driven (rubric/mode/threshold/tie-breaker), per unit.
Tests: agreement fixtures (perfect/none/negative/partial/degenerate) for all
three coefficients; decide() pick + ranked paths; rater output parsing; and
workflow-branch tests (consensus-reached · tie-break→gate · advisory · no-op)
via the time-skipping harness. typecheck/lint/test (144)/build all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes ENG-016 (Wave 4), implementing PRD-001. (Supersedes #12, which GitHub auto-closed when the stacked base branch was deleted on #11's merge — same commit, now based on
main.)At the architecture/system-design stage the senior technical roles independently rate candidate implementation directions; a chance-corrected inter-rater agreement coefficient gates whether the aggregate winner is adopted or sent to a tie-breaker. Phase 1 ships advisory (records the round + agreement score, doesn't block) behind
CONSENSUS_ENABLED.What's added
decide()= aggregate winner (vote tally / Borda) gated oncoefficient ≥ thresholdwith a strict no-tie check. zod schemas for candidate/rating/config/round/outcome.proposeDirections(2–4 distinct candidates) +rateDirections(one independent rater, structured output constrained to candidate ids — ENG-009).directionConsensusworkflow: generate → rate the three roles in parallel (barrier) → decide → branch. Tie-break = human escalation via thearchitecture_decisiongate (signal +approveArchitecture+POST /api/epics/:id/approve-architecture, mirroringapproveRoadmap). Budget-enforced per rater, audited, emitted to the decision log (ENG-014), traceable to the originating request. No-op when <2 viable candidates / runtime unavailable. Per-unit config (rubric/mode/threshold/tie-breaker).Acceptance criteria — all met
core schemas ✓ · pure agreement fn (Krippendorff + Fleiss + Kendall, perfect/none/partial/degenerate) ✓ · pure decide() ✓ · independent structured raters + 2–4 candidate gen ✓ · durable parallel-rater step branching on threshold, budget-enforced + audited ✓ ·
architecture_decisiontie-break with deciding identity ✓ · round/ratings/outcome in the decision log ✓ · config-driven + no-op path ✓ · workflow tests for consensus-reached and tie-break→gate (plus advisory + no-op) ✓Tests
pnpm typecheck·lint·test(144) ·buildall green.🤖 Generated with Claude Code