Skip to content

Review fixtures: three samples trip sibling articles, so recall and clean rate 1.0 are out of reach for a faithful run #201

Description

Where: evaluation/review-fixtures.json cases built from three samples (main at 07e324d), scored by tools/Test-ReviewFixtures.ps1 at minimumExpectedRecall 1.0 and minimumCleanRate 1.0.

Run: BCQuality main at 07e324d, Claude Code 2.1.283, model claude-opus-5-5. All 108 cases were run, each reviewed only by its domain's leaf skill, as the per-case request files prepared by Test-ReviewFixtures.ps1 -PrepareDirectory describe.

The run scored recall 53/54 and clean rate 52/54. All three misses are findings that the leaf skills and articles themselves call for:

  1. data-modeling/check-blocked-in-referencing-code-not-in-master.bad.al (positive case). Only that article was expected. The leaf also cited:
    • owning-table-must-delete-dependents-in-ondelete.md: "Loyalty Point Entry"."Member No." has a TableRelation to "Loyalty Member", and "Loyalty Member" has no OnDelete. That is the article's own detection signal: a table declares a TableRelation to table X, and table X has no OnDelete trigger.
    • master-table-no-from-number-series-in-oninsert.md (minor): a custom master with a No. Code[20] key, no No. Series field and no OnInsert number assignment. al-data-modeling-review names this cue: a custom master table changes its primary key, No./No. Series fields or OnInsert without assigning a blank No. from setup through a number series.
  2. data-modeling/check-blocked-in-referencing-code-not-in-master.good.al (clean control): the same two articles, for the same reasons.
  3. upgrade/initvalue-does-not-update-existing-rows.good.al (clean control): it cites register-upgrade-tags-with-subscribers.md. The upgrade step is guarded by HasUpgradeTag and set with SetUpgradeTag, but the tag is never added to OnGetPerCompanyUpgradeTags, which is that article's anti-pattern.

In all three cases, the per-case request's candidateArticles include the sibling article. The scorer counts any finding id outside expected, so a leaf that follows its skill cannot score 1.0 on these cases.

Suggested fix: make the samples comply with their sibling articles. For example, give "Loyalty Member" an OnDelete for its point entries (or a DeleteAllowed/refusal the article accepts) plus a No. Series assignment, and register the upgrade tag in the initvalue sample. The alternative is to let a case declare "also acceptable" ids for sibling concerns that its sample does not isolate.

Related open pull requests that also edit evaluation/review-fixtures.json: #181, #196, #198.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions