Skip to content

P1: Turn research memos into portable rerunnable experiment bundles #32

Description

@OnePunchMonk

Outcome

Make the output useful enough that researchers share a report and collaborators can inspect or extend the experiment.

Current evidence

Reviewed main 38091c8c on 2026-09-13. Source: src/agent/research_memo.py.

The memo generator already exists and preserves unavailable fields. README says existing-manifest replay remains deferred until episode results are persisted as their own artifact.

Source review only: no new training run, benchmark or external pilot was performed for this issue.

Scope and dependencies

Delivery slice of #28 P3; extend the existing research_memo/episode_report implementation.

Work

  1. Persist episode outcomes and references to all attempted candidates, configs, memory snapshots, dataset/split hashes and software identities in a versioned bundle.
  2. Support report regeneration from saved outcomes without rerunning agents; keep this distinct from fresh execution. Include a rerun command and declared data-access requirements.
  3. Add one user-data ingestion recipe with schema validation, timestamp/order checks and explicit cost assumptions. Keep private/raw data out of shared bundles unless explicitly included.
  4. Export a self-contained HTML/Markdown decision report: hypothesis, evidence, baseline/candidate diff, unsuccessful candidates, outcome coverage and limitations; share existing renderer semantics.

Acceptance

  • A second clean checkout regenerates the report from the bundle without model/network calls.
  • An optional fresh execution validates data identity before running; absent private data is actionable rather than silently replaced.
  • Two user-owned corpora/data fixtures exercise ingestion; malformed timestamps or missing fields fail before research.
  • Every displayed measured number resolves to stored evidence; changing a report's presentation does not consume the final holdout.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions