chore: gate reply drafting in triaging-security-reports skill - #1901
Merged
Conversation
Triage was ending with an unprompted reply draft sitting in the user's mailbox. The "Drafting the reply" section read as a workflow step, so finishing the assessment looked like authorisation to write the reply. Gate that section on an explicit request, and extend the over-promise bullet to cover process decisions: severity ratings, bulletins, CVE requests, fix versions and timelines belong to the PMC, and a reply that answers a reporter's CVE question has made that call on their behalf. A CVE is requested once the fixed release is out, never at triage. The baseline was observed rather than synthesised - a draft was created unprompted, and it pledged a CVE request the PMC had not agreed to. Verified with 20 subagent reps. 15 against the drafting gate, on a real finding under time pressure with the reporter chasing a status update: none drafted unprompted. 5 against the over-promise bullet, with drafting explicitly authorised and the reporter demanding a CVE, a severity and a release date under threat of publishing: none committed the project on any of the three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhckAsHwuXhn3a59qykRwA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a gate to the
triaging-security-reportsskill so that triage stops at the assessment, and extends its over-promise rule to cover process decisions.Why
Two failures observed while triaging a real report:
Drafting the replysection read as a workflow step, so completing the assessment looked like authorisation to write the reply. A draft isn't a thought — it lands in the maintainer's mailbox and pre-commits the project's answer to a reporter.Both came from treating "triage this" as authorisation for everything that plausibly follows it.
What changed
## Drafting the reply→## Drafting the reply — only when you are asked, opening with the gate and four no-exceptions clauses. The existing bullets now sit under "Once you have been asked", so the section governs a draft's content rather than authorising one.No behavioural change to any framework code — this touches only
.claude/skills/.Verification
The baseline was observed rather than synthesised: the unprompted draft and the CVE pledge both happened, and the draft was withdrawn.
20 subagent reps against the edited skill:
An earlier round of gate testing was discarded as confounded — the agents had independent reasons to stop (an unverifiable path in one batch, an already-answered thread in the other), so the final 15 use scenarios where the gate is the only thing standing between the agent and a draft.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PhckAsHwuXhn3a59qykRwA