Skip to content

feat(evidence): retain opt-in test execution receipts and dependency observations - #358

Open
dinakarjs wants to merge 9 commits into
agentrust-io:mainfrom
dinakarjs:feat/execution-evidence-receipts
Open

dinakarjs wants to merge 9 commits into
agentrust-io:mainfrom
dinakarjs:feat/execution-evidence-receipts

Conversation

@dinakarjs

@dinakarjs dinakarjs commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem and resulting behavior

The execution receipt could be paired with an edited command while retaining the same log, and its documentation overstated dependency discovery. Version 2 adds a hashed evidence.json payload binding command argv, the retained log digest and explicit extra-package observations. Validation checks both its digest and its agreement with the receipt. The LangChain workflow explicitly observes langchain-core and langgraph separately from the schema-limited tested_against block.

Documentation now states exact declared-version matching versus compatibility floors, the limited observation scope, the retention of command-line arguments and the requirement to keep secrets out of argv. Version 1 receipts remain historical and require a fresh capture for this checker. These artifacts remain unsigned: an operator can rewrite all three consistently, and hashes do not authenticate execution.

Validation — 2 November 2026

Head: 9a69ab5; tree: 97ed261d1a6b692c90ce2cc80f189b7d3a262bb8.

  • All 15 receipt tests pass, including real subprocess success/failure/timeout; command substitution, log substitution with an updated receipt hash, changed payload, requested missing packages and boolean return codes are refused.
  • Isolated command/payload-binding, payload-digest and extra-metadata guard weakenings produce 3, 1 and 1 assertion failures respectively; restored tests pass.
  • A real released LangGraph interoperability test passed under the recorder with langchain-core 1.6.3 and langgraph 1.2.12. The receipt records both framework versions and TRACE 0.11.0; consistency validation and pip check pass. Framework dependencies came from the existing sdk-langchain hash lock, with the released TRACE model/validation dependencies installed separately; no complete environment-lock claim.
  • Targeted Ruff E4/E7/E9/F and whitespace checks pass. Current-head hosted validation and independent re-review remain required; no full repository suite rerun is claimed.

Opt-in local/CI observations only: no adversarial attestation, automatic dependency discovery, tier promotion or independent execution acceptance. Prepared with OpenAI Codex; DCO-signed.

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Oct 21, 2026
…ce receipt

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
…ce receipt

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Preserve contribution history and evidence boundaries; validate the final merged tree.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com>
@dinakarjs
dinakarjs marked this pull request as ready for review October 31, 2026 03:43

Copy link
Copy Markdown
Contributor Author

@carloshvp Please review the opt-in execution-receipt recorder, consistency checker and LangChain workflow integration at current head c6e976e0dfe1d4813e545c960df049f61969e832. This PR is now ready for independent review, and the existing reviewer requests are populated. All returned current-head technical workflows succeed (validation evidence); the PR description records the prior local tests and their revisions.

Unsigned observations only; no adversarial attestation, complete dependency discovery or automatic tier promotion.

Please identify any remaining change required or disposition this bounded contribution. This request does not claim independent approval, a new full-suite rerun, or merge readiness. Prepared with OpenAI Codex.

@rajnisht7 rajnisht7 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the overall changes looks good, a few things needs to be looked on:

  • Docs overclaim - docs/test-evidence.md says undeclared dependencies fail the check. They don’t. The recorder only observes packages listed in tested_against. kindly fix the wording to match the actual behaviour.
  • Command is not bound to the log - Validator only checks that command is a non-empty list. one can change the command in receipt.json and keep the same log; check still returns consistent. it would be nice to bind the command (e.g. command_sha256 or include it in a hashed evidence payload).
  • Secrets in the command - full command argv is stored in receipt.json and then uploaded as an artifact. Docs only warn about logs. Please document that secrets must never be passed as command-line arguments.
  • Incomplete dependency observation for LangChain - Receipt only records agentrust-trace. The interop test actually runs against langchain-core==1.6.3 and langgraph==1.2.12. Schema currently forbids putting those in tested_against, so either document clearly that framework versions are out of scope for this receipt, or give the recorder an explicit way to observe extra packages.

also: tested_against is treated as a version floor elsewhere in the repo, but this checker requires exact equality. Worth being explicit about that.

Version the unsigned receipt payload and check command/log/extra-package agreement. Add explicit package observation to the LangChain workflow, document exact-version and secret-argv boundaries, and retain malformed/substitution controls.

Prepared with OpenAI Codex.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>

Copy link
Copy Markdown
Contributor Author

@rajnisht7 Your receipt review is addressed at 9a69ab5. Version 2 binds command/log digest/extra observations through a hashed evidence.json payload. The LangChain workflow explicitly observes langchain-core and langgraph; docs correct discovery/exact-version claims and prohibit secrets in retained command argv. All 15 receipt tests pass; a recorded real LangGraph interop test passes with core 1.6.3/graph 1.2.12 and a consistent receipt. Isolated binding/digest/metadata guard weakenings fail the new tests. The description records evidence and limits: unsigned artifacts can still be rewritten together and do not authenticate execution. Please re-review the current head. Prepared with OpenAI Codex.

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 9a69ab5e948de7f136465e815bfc4060b6d34b56.

The version-2 payload is the right repair for the substitution that was left open. evidence.json carries the command, the log digest, and the extra-package map. The receipt stores the digest of that file. validate requires the payload to equal those receipt fields and the file digest to match. Changing the command in the receipt alone fails. Substituting the log and updating only the receipt hash fails. A boolean returncode fails because the check uses type(...) is int, so False is not success. Version 1 receipts are refused rather than interpreted. That is the correct compatibility cut: a historical file stays historical, and a current check requires a fresh capture.

What this does not do, and what docs/test-evidence.md now says plainly, is authenticate the run. An operator who rewrites the receipt, the payload, and the log together, and updates both hashes, still gets consistent. A digest catches an accidental substitution. It does not catch a coordinated rewrite. draft-durga-agenttrust-trace-verification-00 draws that line: a canonical hash binds bytes to a claim, and fail-closed delegation attestation is a separate object. TR-2026-05 (10.5281/zenodo.23776217) is the identity version of the same limit. An unsigned local receipt is not an on-behalf-of attestation of the runner, and the tool does not pretend otherwise. Keeping secrets out of argv is stated because the argv is retained in both JSON files. Exact equality with tested_against is stated as a check of this run, not a change to the repository's compatibility floor. langchain-core and langgraph are observed with --observe-package, outside the schema-limited block, which is the honest place for them. The LangChain job installs the no-framework lock, which pins agentrust-trace==0.11.0, before the framework lock, so the declared version is present for the checker without being smuggled in as a framework pin.

Python 3.11.9, python -m pytest tests/test_test_evidence.py -q on this head: 15 passed, 0 failed, including real subprocess success, failure, and timeout. I did not rerun the LangGraph interoperability test or the full repository suite. Approve, within that unsigned boundary.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants