Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
cf77277
Add trusted native skill comparison subaction [skip ci]
ronniegeraghty Nov 12, 2026
47f26e5
Bound generated comparison projections [skip ci]
ronniegeraghty Nov 12, 2026
0083ebd
Fix comparison provenance, drift and reporting defects [skip ci]
ronniegeraghty Nov 13, 2026
323678c
Restore registry-portable locked resolutions [skip ci]
ronniegeraghty Nov 13, 2026
3c8cdb7
Preserve literal untracked filename detection [skip ci]
ronniegeraghty Nov 13, 2026
77cd381
Preserve native verdicts and measurement integrity [skip ci]
ronniegeraghty Nov 13, 2026
6665de1
Scope trusted report ordering and clarify safe validation [skip ci]
ronniegeraghty Nov 13, 2026
99b488d
Guard raw measurement drift and final merge freshness [skip ci]
ronniegeraghty Nov 13, 2026
6011d30
Reject post-arm drift and align report byte budgets [skip ci]
ronniegeraghty Nov 13, 2026
6010699
Measure resolved file symlinks and clarify missing artifacts [skip ci]
ronniegeraghty Nov 13, 2026
5446748
Disable Git replacements and reject empty untracked state [skip ci]
ronniegeraghty Nov 13, 2026
2a30ab7
Display model-less native evidence and clarify failure icon [skip ci]
ronniegeraghty Nov 13, 2026
3fd266b
Reuse native eval names and avoid repeated failure details [skip ci]
ronniegeraghty Nov 13, 2026
e926c2c
Expose fixed safe comparison setup failure codes [skip ci]
ronniegeraghty Nov 13, 2026
339bc23
Reject Git filters before comparison source restoration [skip ci]
ronniegeraghty Nov 13, 2026
5be803c
Bind comparison worktrees and preserve source path transitions [skip ci]
ronniegeraghty Nov 13, 2026
750ff67
Classify shallow checkouts and missing measurement artifacts [skip ci]
ronniegeraghty Nov 13, 2026
82e3c32
Disable Git callbacks and classify malformed selection entries [skip ci]
ronniegeraghty Nov 13, 2026
55fd6b7
Use scenario-local indicators without hiding native scoring failures …
ronniegeraghty Nov 13, 2026
8350082
Simplify changed-skill comparison and integrate current main [skip ci]
ronniegeraghty Nov 13, 2026
6f8fb17
Fix local result rendering and ordinary long labels [skip ci]
ronniegeraghty Nov 14, 2026
9e78eb4
Check projection IDs and bound credential token prefixes [skip ci]
ronniegeraghty Nov 14, 2026
a2823c4
Describe both Action bundles in license notices [skip ci]
ronniegeraghty Nov 14, 2026
c00931f
Reject reports with unknown stored run order [skip ci]
ronniegeraghty Nov 14, 2026
bcbe1c1
Reject selected-skill measurement aliases before execution [skip ci]
ronniegeraghty Nov 14, 2026
c029d4d
Exercise external symlink target bytes in regression [skip ci]
ronniegeraghty Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ sticky PR comment** with the suite, eval names, verdicts, and evidence links.
Vally owns selection, execution and grading; the Action handles installation
and GitHub integration.

For approval-gated Before/After skill comparisons from a trusted `workflow_run`,
use the separate [comparison subaction](comparison/README.md). The root Action's
event and checkout policy is unchanged.

Use `github/evals-action@v0` for the latest 0.x pilot release
(currently [v0.1.0](https://github.com/github/evals-action/releases/tag/v0.1.0)).
Use `@v0.1.0` to stay on that specific Action release. The `v0` tag advances as
Expand Down
330 changes: 330 additions & 0 deletions comparison/README.md

Large diffs are not rendered by default.

47 changes: 47 additions & 0 deletions comparison/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Compare skill evals
description: Authenticate a source PR, run native Before/After arms, or publish bounded comparison projections.
inputs:
command:
description: context, run, or report. Call each in its separately permissioned job.
required: true
source-workflow:
description: Exact trusted source workflow path, for example .github/workflows/skill-evals.yml.
required: true
github-token:
description: Job-scoped GitHub token for authenticated provenance and, only in report, commenting.
default: ${{ github.token }}
copilot-token:
description: Explicit execution credential, only in the approval-gated run job.
context:
description: Exact context output from the trusted selection job, never from an artifact.
selection:
description: Exact schema documented in comparison/README.md, from trusted selection job output.
producing-attempt:
description: Selection job's producing attempt. Required on run/report to reject partial paid retries and bind publisher retries.
comparison-id:
description: Selected include entry id for this run job.
working-directory:
description: Candidate merge checkout relative to GITHUB_WORKSPACE, for run only.
vally-version:
description: Exact stable native Vally version for run; latest is deliberately unsupported here.
default: '0.21.0'
artifact-directory:
description: Absolute downloaded projection directory, for report only. Contents are untrusted JSON data.
outputs:
context:
description: Authenticated source/producer/PR/trusted-workflow/merge identity from context.
context-path:
description: Private local context.json path for the trusted repository selector; not an artifact authority.
merge:
description: Current verified PR merge SHA from context.
projection-directory:
description: Run's private output directory; do not upload recursively.
before-path:
description: Exact allowlisted Before JSON projection path.
after-path:
description: Exact allowlisted After JSON projection path.
outcome:
description: passed/failed for run, posted/stale for report. Posting never changes a native failure.
runs:
using: node24
main: ../dist/comparison-index.cjs
4 changes: 2 additions & 2 deletions dist/THIRD-PARTY-NOTICES.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
THIRD-PARTY NOTICES
GitHub Evals Action

This inventory describes packages contributing code to dist/index.cjs.
This inventory describes packages contributing code to dist/index.cjs and dist/comparison-index.cjs.
License text inclusion is evidence, not release approval. Missing terms remain unresolved.
Inline third-party comments are retained separately in index.cjs.LEGAL.txt.
Inline third-party comments are retained separately in index.cjs.LEGAL.txt and comparison-index.cjs.LEGAL.txt.

========================================================================
Package: @actions/artifact
Expand Down
1 change: 1 addition & 0 deletions dist/comparison-index.cjs

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions dist/comparison-index.cjs.LEGAL.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Bundled license information:

undici/lib/web/fetch/body.js:
/*! formdata-polyfill. MIT License. Jimmy Wärting <https://jimmy.warting.se/opensource> */

undici/lib/web/websocket/frame.js:
/*! ws. MIT License. Einar Otto Stangvik <einaros@gmail.com> */
2 changes: 1 addition & 1 deletion dist/index.cjs

Large diffs are not rendered by default.

Loading
Loading