Skip to content

feat(evolution): carry the observed seat in the exported dataset record - #1210

Open
HugoXK wants to merge 1 commit into
mainfrom
dev4/1194-observed-seat
Open

HugoXK wants to merge 1 commit into
mainfrom
dev4/1194-observed-seat

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-4

Fixes #1194

The defect

An exported evolution dataset record carried attribution only — the claim — and nothing about what the instance observed. src/lib/evolution/export.ts selected the job row without attribution_ground_json, and src/lib/evolution/dataset.ts redacted attribution but had no ground field, so a consumer of the dataset could not tell a claim from an observation: the exact distinction the job page already draws through groundAttribution and the legacy reconstruction from #1186.

What changed

  • src/lib/teamStats.ts: the legacy reconstruction half of the page's attributionGroundFor moves here as reconstructAttributionGround(recorded, observedOnly); the job page and the export now call one rule. The page's behaviour is unchanged (its existing grounding tests pass untouched).
  • src/lib/evolution/export.ts: selects j.attribution_ground_json; each row gains attributionGround = storedGround ?? reconstructAttributionGround(parsed claim, team_stats_json !== null). attribution stays parse(attribution_json), exactly as before — the claim is not transformed.
  • src/lib/evolution/dataset.ts: EvolutionRow.attributionGround is typed AttributionGround | null and redacted through the existing redactValue; DATASET_SCHEMA_VERSION moves to 2; attribution_ground_json joins REQUIRED_JOB_COLUMNS.
  • Docs: docs/decisions/0003-agentdescent-evolution.md and the dataset paragraph in docs/architecture.md name both fields and version 2. One changelog.d/1194.added.md.

No schema migration (#1186 already added the column and its ALTER TABLE), no change to the split, the hash rule, the consent filter or the page.

What the record looks like at this tip

A probe against a Store with one stored-ground row and one pre-#1186 row (claim plus team stats), exported through loadEvolutionRows + buildDataset:

{
  "schemaVersion": 2,
  "records": [
    { "jobId": "job_legacy", "attribution": { "writtenBy": "Dev-1" },
      "attributionGround": { "steps": [{ "key": "writtenBy", "claimed": null, "observed": "Dev-1", "state": "grounded" }] } },
    { "jobId": "job_stored", "attribution": { "writtenBy": "Dev-2", "reviewedBy": "Reviewer" },
      "attributionGround": { "steps": [{ "key": "writtenBy", "claimed": "Dev-2", "observed": "Dev-1", "state": "unobserved" }] } }
  ]
}

The legacy row is reconstructed by the page's rule (observation, no claim); the stored row keeps its claim and carries the observation beside it. A row with nothing recorded carries attributionGround: null — the loader test pins that, and the unit test pins null for an input row without a ground record.

Acceptance, item by item

  1. Both fields, or null: test/evolutionDataset.test.ts — "a record carries the observed ground beside the claim…" (claim unchanged, stored ground travels through, no-record row is null) and "the loader reads the stored ground and reconstructs a legacy row, never guessing grounded".
  2. Version 2, and version 1 still verifies: the version travels inside the hashed projection. test/evolutionDatasetVerify.test.ts loads test/fixtures/evolution-dataset-v1.json — a real dataset produced by the pre-change builder at 044c7515 (ds_c0f66c8fe1cda136) — and verifyDataset returns ok at schemaVersion: 1.
  3. The refusal names the column: assertExportable(['kind','evidence_json','team_stats_json'], 38) throws naming attribution_ground_json; REQUIRED_JOB_COLUMNS.length is 4.
  4. Docs: both files above.
  5. Red then green: below.

Red at the base

At 044c7515, npx tsx --test test/evolutionDataset.test.ts:

  • a record carries the observed ground… → expected: 2, actual: 1 (schema version).
  • the frozen schema refusal is gone… → Missing expected exception (no attribution_ground_json requirement).
  • the loader reads the stored ground… → the loaded record had no attributionGround.

At this tip, the same three suites together: 28/28 pass.

Ran:

  • npm run check — pass (lint baseline 11 warnings in 8 entries, typecheck, 3296/3296 tests, build)
  • node scripts/version.mjs check — version: 0.4.5 ok
  • npm run mock:check — ok (14 files)
  • node scripts/changelog.mjs check — 4 fragment(s) ok
  • npx tsx --test test/evolutionDataset.test.ts test/evolutionDatasetVerify.test.ts test/attributionGrounding.test.ts — 28/28
    Not run: Docker image, real SMTP, a production export of the live database.

Uncertainty and residual risk

  • The reconstruction is the page's rule, so it reads team_stats_json !== null as "this column holds an observation". A row whose team stats are malformed JSON takes that branch and reconstructs from whatever parseAttributionJson returns; that is the page's current answer, not a new one.
  • A stored ground value that fails parseAttributionGround falls back to reconstruction rather than exporting the malformed object — same as the page.
  • redactValue runs over the ground so role names pass the one recogniser; the shape is cast back to AttributionGround because redaction preserves strings.
  • No export is rewritten or re-registered; a version-1 file on disk stays version 1 and verifies as before.
  • The MISSING_JOB_COLUMNS message tells the operator to start the current Fledge once, which applies the ALTER TABLE; that is the existing migration path, not a new one.

Alternatives considered

  • Compute the ground in buildDataset from attribution + teamStats: rejected — the builder only sees rows, not whether the stored ground column had a value, and the loader is the one place that reads it.
  • Duplicate the legacy reconstruction in the exporter: rejected — the card names fix(jobs): a claimed seat is not an observed seat #1186's rule as the one to reuse, and two copies drift.
  • Keep DATASET_SCHEMA_VERSION at 1: rejected — the record shape is a data contract; a consumer must know which fields to expect.

…rd (#1194)

Every exported record now carries `attributionGround` beside the stored
`attribution` claim: the per-key claimed/observed/state record the job page
already reads, reconstructed by the same rule for rows written before #1186 and
null when nothing was recorded. The rule moves to `teamStats.ts` so the page
and the export share it. The column joins REQUIRED_JOB_COLUMNS, so a pre-#1186
database is refused by name, and DATASET_SCHEMA_VERSION moves to 2 while a
version-1 export still verifies against its own manifest.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at 5b59219

  • Shape and version are one change: DATASET_SCHEMA_VERSION = 2 (src/lib/evolution/dataset.ts:20), attributionGround: AttributionGround | null on EvolutionRow (:44) and redacted in redactRow (:166); contentHashOf hashes the whole records array (:255-259), so the ground sits inside the content hash and the version bump is required, not cosmetic.
  • One read exports both halves: j.attribution_ground_json joins the SELECT (src/lib/evolution/export.ts:24), the claim stays the stored object (:54), and storedGround ?? reconstructAttributionGround(attribution, team_stats_json !== null) (:55) uses the single rule at src/lib/teamStats.ts:55-69, which the page now calls (src/views/pages.ts:513) instead of keeping its own copy.
  • Back-compat reproduced, not read: I rebuilt test/fixtures/evolution-dataset-v1.json with the base builder at 044c7515 from its own records and got byte-equal manifest and records, id ds_c0f66c8fe1cda136; at this tip verifyDataset recomputes over manifest.schemaVersion (dataset.ts:369), so v1 returns ok and a tip-built dataset verifies at 2.
  • The refusal is real: attribution_ground_json joins REQUIRED_JOB_COLUMNS (dataset.ts:27), assertExportable names the missing column (:434-440), and the CLI calls it before loadEvolutionRows (scripts/evolution-shadow.ts:185), so the new SELECT cannot hit a pre-#1186 database.
  • Red then green: 28/28 at the tip; at base 044c7515 with only the committed tests, 18 pass / 3 fail naming version 1≠2, the missing refusal and the loaded record with no attributionGround. No UI surface — only the helper moved and the rendered string is unchanged. Non-blocking: the fixture's rows are synthetic (finishedAt: ""), so "a real dataset produced at 044c751" proves the version path and its bytes, not a production export.

Ran: npx tsx --test test/evolutionDataset.test.ts test/evolutionDatasetVerify.test.ts test/attributionGrounding.test.ts (28/28 in a tree built as base 044c7515 + this diff, all 10 blobs equal to head); npx tsx --test test/evolutionDataset.test.ts test/evolutionDatasetVerify.test.ts with only the committed test files at base (18 pass, 3 fail as quoted); npx tsx probe_fixture.ts (base builder rebuild of the v1 fixture, manifest and records byte-equal, id ds_c0f66c8fe1cda136).
Not run: full npm run check locally (CI ci-ok 136933907113 success at this tip); Docker, SMTP and an export of a live database.
Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:approved Issue reviewed and approved by the Fledge Platform Reviewer label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved Issue reviewed and approved by the Fledge Platform Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(evolution): the dataset record carries the observed seat beside the claimed one

2 participants