Skip to content

fix(security): correct defects found while optimizing the security skills - #1444

Merged
potiuk merged 2 commits into
mainfrom
fix/security-behaviour-fixes
Sep 28, 2026
Merged

potiuk merged 2 commits into
mainfrom
fix/security-behaviour-fixes

Conversation

@potiuk

@potiuk potiuk commented Sep 28, 2026

Copy link
Copy Markdown
Member

Follow-up to the security family optimization (#1344, after #1435–#1438, which only moved bytes, tightened descriptions, or pointed at shared recipes). This PR has two commits:

  1. fix(security): defects found along the way, which change behaviour.
  2. fix(github-rollup): reviewed as fix(github-rollup): find and write rollups for any tracker, and add amend-latest and fold #1442, which was merged into this branch rather than main, so it lands here.

Correctness

  • security-issue-import:
    • Steps 2a and 2c skipped candidates by a class (automated-scanner, spam, …) that is only assigned in Step 3, so the skip could not apply as written.
    • They now skip on a provisional class read from the root message they already load, and still extract the keywords Step 2b reuses.
    • If Step 3's final class turns out to be a report, the skipped searches run before Step 4.
  • security-issue-fix:
    • 5c allowed vulnerability-class names (SSRF, path traversal) in a public PR. AGENTS.md embargoes "the vulnerability class", and the agent-guard guard already blocked those names.
    • 5c is now the single list that 5g, Step 7, Step 9 and the Guardrails check against.
    • SKILL.md now mentions the guard, as a backstop that does not match the bare words vulnerability / advisory: adding them would block ordinary PRs everywhere the guard runs.
    • The guard itself is unchanged; tests now pin its class-name coverage.
  • security-cve-allocate: Step 5.1 rewrote the whole issue body where Step 4.1 proposes a one-field github-body-field set. It now applies exactly what was proposed.
  • security-issue-sync Step 0 said PonyMail "degrades quietly", while its gate is a hard stop when the backend is mandatory: yes. Step 0 now states both cases.
  • security-issue-invalidate:
    • invalidate proposed matched any triage proposal that mentioned INVALID, including "not INVALID". It now matches the proposal line itself.
    • The Gmail-backend guidance now points at the single selection rule in tools/gmail.
    • A stale .claude/skills/... link is fixed.
  • security-tracker-stats-dashboard ignored the configured cache directory, so its freshness check could read a different directory than the fetch wrote.
  • workflow-security-audit hid zizmor failures with 2>/dev/null, so an unscanned repository could be reported as clean. Exit codes 11–14 (findings) are expected; any other non-zero exit is reported as a scan failure.

Pre-flight coverage

requires_config: now declares files the skill cannot run without:

  • triage: release-trains.md, canned-responses.md
  • invalidate: canned-responses.md
  • model-update: security-model.md
  • import-from-md: project.md

The security family README's config table is regenerated.

Project-neutrality

Removed from generic skills: hardcoded release numbers, a backport label, a list of maintainer names, an Airflow canned-response inventory, and a real CVE id. Release data now comes from <project-config>/release-trains.md, and canned-response names from <project-config>/canned-responses.md.

Scratch files

Literal /tmp/... paths, which the secure setup's sandbox cannot write, are now <scratch>/...: the session scratch directory as an absolute path, so a gh running outside the sandbox resolves the same file.

Stale text

Fixed: wrong class, field and outcome counts ("five classes", "nine fields", "four outcomes"), a reference to a non-existent Step 7g, and a pointer to an AGENTS.md section that does not exist.

Test plan

  • prek run --all-files, including lychee
  • skill-and-tool-validate, check-skill-config
  • tools/agent-guard tests
  • All 16 eval steps whose extracted section changed were run outside the sandbox, before (the base) and after. No case regressed; the failures on both sides predate this PR (issue-fix/step-5c case 3, which expects both security fix and security; invalidate/step-4 cases 1–3; import/step-2a case 3 and step-5 case 3).

Found, not changed

  • The security-issue-fix golden rule and description still say the tracker link must never appear in the public PR, which contradicts AGENTS.md (tracker identifiers are public-safe) and 5c. Changing the description moves surface_hash, so it is left for a decision.
  • import-from-pr carries the generated pre-PR adversarial-review block, which the validator does not require there. Its opening line assumes the skill opens a PR, which this skill does not.
  • The triage step-3-classify eval uses an older five-class static prompt.

github-rollup (commit 2, reviewed as #1442)

  • Finds a rollup under any <!-- <name> status rollup v<N> marker, and names the tracker repository in new ones, instead of hardcoding airflow-s.
  • Reads the comment's REST id from its URL instead of scanning every comment in the repository.
  • Adds amend-latest and fold.
  • tools/github/status-rollup.md is generic and tool-first.
  • preflight-audit now recognises rollup comments as skill activity.
  • Tests: github-rollup 42 and preflight-audit, all passing.

Part of #1344.

Generated-by: Claude Code (Opus 5.5)

🤖 Generated with Claude Code

potiuk and others added 2 commits September 28, 2026 09:49
…ills

Behaviour fixes found during the #1344 optimization, kept apart from the
behaviour-preserving PRs below this one.

Correctness:
- security-issue-import: Steps 2a and 2c skipped candidates by a class
  that is only assigned in Step 3, so the skip could never apply as
  written. They now skip on a provisional class read from the root
  message they already load, still extract the keywords Step 2b reuses,
  and run the skipped searches after Step 3 if the final class turns out
  to be a report.
- security-issue-fix: 5c allowed vulnerability-class names (SSRF, path
  traversal) in a public PR, while AGENTS.md embargoes the vulnerability
  class and the agent-guard guard already blocked them. 5c is now the
  one list that 5g, Step 7, Step 9 and the Guardrails check against. The
  guard is mentioned as a backstop that does not match the bare words
  `vulnerability` / `advisory`, which rely on the manual check.
- security-cve-allocate: Step 5.1 rewrote the whole issue body where
  Step 4.1 proposes a one-field `github-body-field set`. It now applies
  exactly what was proposed.
- security-issue-sync Step 0 said PonyMail "degrades quietly", while
  its gate is a hard stop when the backend is `mandatory: yes`.
- security-issue-invalidate: the `invalidate proposed` query matched any
  proposal that mentioned INVALID; it now matches the proposal line.
  Its Gmail backend guidance now points at the single selection rule in
  tools/gmail, and a stale `.claude/skills/...` link is fixed.
- security-tracker-stats-dashboard ignored the configured cache
  directory, so the freshness check could read a different directory
  than the fetch wrote.
- workflow-security-audit hid zizmor failures with `2>/dev/null`, so an
  unscanned repository could be reported as clean.

Pre-flight coverage: triage, invalidate, model-update and
import-from-md now declare config files they cannot run without, so
pre-flight catches a missing file before the run starts.

Project-neutrality: removed hardcoded release numbers, a backport label,
a list of maintainer names, an Airflow canned-response list and a real
CVE id from generic skills; release data now comes from
<project-config>/release-trains.md.

Scratch files: literal /tmp paths, which the secure setup's sandbox
cannot write, are now `<scratch>` (the session scratch directory,
absolute, so `gh` running outside the sandbox resolves the same file).

Stale text: wrong class, field and outcome counts; references to steps
and AGENTS.md sections that do not exist.

Generated-by: Claude Code (Opus 5.5)
…mend-latest and fold (#1442)

github-rollup only recognised a rollup whose first line began with
`<!-- airflow-s status rollup v`, and wrote that marker on every new
rollup. On any other adopter's tracker it would miss the existing
rollup, create a second one, and label it with another project's name.

- Detection now matches `<!-- <any-name> status rollup v<N>`. A new
  rollup's marker names the tracker repository, which reproduces the
  existing `airflow-s` marker on that tracker.
- The comment's REST id is read from its `#issuecomment-<id>` URL
  instead of paginating every comment in the repository.
- `amend-latest` replaces the body of the most recent entry, keeping
  its date and user, and refuses when that entry has a different
  action. It serves passes that fill in a value (such as a draft id)
  after writing their entry.
- `fold` moves a legacy bot comment into the rollup under its original
  date and author, left-trims it, and deletes the original only after
  the append succeeded.
- tools/github/status-rollup.md drops the hardcoded marker and the
  hand-rolled read/concatenate/PATCH script. The tool is the recipe, so
  the rollup body stays out of the agent's context.
- preflight-audit's no-op classifier only knew the `apache-magpie:`
  skill marker, so a fresh rollup append looked like human activity
  and the tracker was dispatched instead of skipped. It now also
  recognises the rollup marker.

Generated-by: Claude Code (Opus 5.5)
@potiuk potiuk added family:security security-* skills family:tools tools/* labels Sep 28, 2026
@potiuk
potiuk added this pull request to stack #1445 September 28, 2026 07:54
@github-actions github-actions Bot added contract:tracker Tool capability: issue / board / label backend contract:source-control Tool capability: branch / commit / diff / push (VCS) contract:change-request Tool capability: proposed-change review + merge gate (PR / MR / Gerrit change) substrate:analytics Tool substrate: read-only metrics / dashboards / renderers substrate:action-guard Tool substrate: deterministic pre-tool-use command guards labels Sep 28, 2026
@potiuk
potiuk merged commit 9612ac4 into main Sep 28, 2026
12 checks passed
@potiuk
potiuk deleted the fix/security-behaviour-fixes branch September 28, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contract:change-request Tool capability: proposed-change review + merge gate (PR / MR / Gerrit change) contract:source-control Tool capability: branch / commit / diff / push (VCS) contract:tracker Tool capability: issue / board / label backend family:security security-* skills family:tools tools/* substrate:action-guard Tool substrate: deterministic pre-tool-use command guards substrate:analytics Tool substrate: read-only metrics / dashboards / renderers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant