Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/pr-management/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ says which file is missing.
|---|---|---|
| [`mentoring-config.md`](../../plugins/magpie-setup/templates/mentoring-config.md) | Tone knobs and hand-off protocol for the thread-level mentoring skill. | `mentor` |
| [`pr-management-triage-ci-check-map.md`](../../plugins/magpie-setup/templates/pr-management-triage-ci-check-map.md) | CI-check name pattern → category name + doc-URL mapping for the violations comment. | `pr-triage` |
| [`privacy-llm.md`](../../plugins/magpie-setup/templates/privacy-llm.md) | Which model tier may see which class of content, for projects routing foundation-private information away from third-party models. | `reviewer-routing` |
| [`privacy-llm.md`](../../plugins/magpie-setup/templates/privacy-llm.md) | Which model tier may see which class of content, for projects routing foundation-private information away from third-party models. | `pr-triage`, `reviewer-routing` |
| [`release-trains.md`](../../plugins/magpie-setup/templates/release-trains.md) | Active release branches, release-manager attribution per cut, rotation rosters, security-team roster. | `code-review`, `reviewer-routing` |
| [`stale-sweep-config.md`](../../plugins/magpie-setup/templates/stale-sweep-config.md) | Grace windows and exemption labels for stale sweeps. Absent, the framework defaults apply. | `pr-stale-sweep` |

Expand Down
8 changes: 5 additions & 3 deletions plugins/magpie-pr-management/skills/pr-triage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ when_to_use: |
already triaged or in its grace window.
argument-hint: "[pr:N] [label:LBL] [author:LOGIN] [review-for-me] [stale] [repo:owner/name]"
capability: capability:triage
surface_hash: sha256:5c92df54aab39ad7
surface_hash: sha256:27129b96ac38f34d
license: Apache-2.0
measured_tokens: 5027
measured_tokens: 5079
---
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
Expand Down Expand Up @@ -322,7 +322,9 @@ Selector semantics (`triage pr:<N>` / `label:<LBL>` / `author:<LOGIN>` / `review
[`classify-and-act.md`](classify-and-act.md), once — the pre-filters
(F1–F5c), the first-match-wins decision table, the Real-CI guard on
`passing` rows, and the single-pass output contract are specified
there.
there. When `enable_typed_decision_prefilter` is enabled, an advisory
Comment thread
potiuk marked this conversation as resolved.
Resolved
shadow pre-filter runs alongside post-guard classification to record
telemetry without altering decisions (see [`classify-and-act.md`](classify-and-act.md) Step 2.4).

---

Expand Down
47 changes: 44 additions & 3 deletions plugins/magpie-pr-management/skills/pr-triage/classify-and-act.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,10 @@ short; it is the only one the skill needs at decision time.

Classification + action selection is a **pure function of state**
populated by the single batched GraphQL query in
[`fetch-and-batch.md`](fetch-and-batch.md). No network calls, no
prompts, no writes.
[`fetch-and-batch.md`](fetch-and-batch.md). The decision table
itself makes no network calls, prompts or writes; the optional
shadow pass in step 4 calls the typed-decision provider and
appends a telemetry line.

## Step 2 — Classify the entire fetched set

Expand All @@ -50,9 +52,48 @@ Run **every PR fetched in Step 1** through
20), the [Real-CI guard](classify-and-act.md#real-ci-guard)
must pass — otherwise re-route to `pending_workflow_approval`
(row 1) or `rebase` (row 16).
4. **Opt-in typed-decision shadow pre-filter (advisory):**
When enabled via `enable_typed_decision_prefilter: true`
(default `false`) with threshold `typed_decision_confidence_threshold`
(default `0.85`), run the helper alongside the post-guard classification to evaluate classifier accuracy.
Write the PR state to a scratch file and invoke:
```bash
uv run --project <framework>/tools/typed-decision python3 <framework>/skills/pr-management-triage/scripts/typed_decision_prefilter.py \
--file <scratch>/pr-<N>.json \
--table-classification <label>
```
- **Contract:**
The decision table and Real-CI guard result remains authoritative in all cases.
The shadow pass records predictions alongside the final table classification for telemetry.
- **`--file` JSON schema:**
The input file provides a JSON object containing:
`number` (int/str), `author` (str or `{"login": str}`), `authorAssociation` (str), `statusCheckRollup` (str), `failed_checks` (list of str), `recent_main_failures` (list of str), `mergeable` (str), `unresolved_threads` (int), `isDraft` (bool), `commits_behind` (int), `real_ci_ran` (bool), `labels` (list of str), `title` (str), `body` (str), `commit_messages` (list of str).
Missing fields default to `UNKNOWN` to avoid biasing prompts toward `passing`.
- **Outcomes:**
- `high_confidence`:
Confidence meets or exceeds threshold and predicted label is valid.
Logs `{pr, table_classification, predicted_label, confidence, latency_ms, match, outcome: "high_confidence"}`.
- `low_confidence`:
Confidence below threshold or unrecognised label.
Logs `{pr, table_classification, predicted_label, confidence, latency_ms, match, outcome: "low_confidence"}`.
- `fell_through`:
Flag disabled, provider unavailable, or network error.
Logs `{pr, outcome: "fell_through"}` with the specific reason (or skips logging if disabled).
Triage proceeds unaffected.
- **Third-party LLM endpoint and privacy prerequisites:**
- Endpoint: `https://api.typesafe.ai/v1/systemone`
- Credentials: `TYPESAFE_API_KEY` (or fallback `JEV_API_KEY`) or `~/.config/apache-magpie/typesafe.key`.
- Privacy-LLM approval: Requires an opt-in entry in `<project-config>/privacy-llm.md`
with non-empty `Data-residency contract` and maintainer sign-off
before enabling outbound classification.
- Contributor title, body, and commits are fenced
inside `<untrusted-external-data>` with tags escaped as data only.
- **Telemetry:**
Records are appended to `.apache-magpie-local/logs/pr-triage-typed-decision.jsonl`.

Classification + action selection is a pure function of the data
already fetched in Step 1. No extra network calls. No prompts.
already fetched in Step 1.
The decision table itself makes no network calls; the optional shadow pass in step 4 does.
The full-set classification runs in a single pass over the
in-memory list assembled in Step 1 — no pagination, no chunking.

Expand Down
12 changes: 12 additions & 0 deletions plugins/magpie-pr-management/skills/pr-triage/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,18 @@ landed in `gh` 2.20+; earlier versions need the REST call from

---

## 6. Typed-decision shadow pre-filter prerequisites (when enabled)

When `enable_typed_decision_prefilter: true` is configured in `<project-config>/pr-management-config.md` or overrides:

1. **Third-party endpoint:** The provider calls `https://api.typesafe.ai/v1/systemone` to classify PR states.
2. **Credentials:** `TYPESAFE_API_KEY` (or fallback `JEV_API_KEY`) environment variable or `~/.config/apache-magpie/typesafe.key` must be present.
3. **Privacy-LLM opt-in:** Requires an approved entry in `<project-config>/privacy-llm.md` with a non-empty `Data-residency contract` and valid maintainer `Approved-by` sign-offs.
4. **Prompt injection defense:** Contributor title, body, and commits are enclosed in `<untrusted-external-data>` as data only.
5. **Fail-open contract:** If any credential, module import, or privacy approval is missing, or on network error/timeout, the pre-filter logs `fell_through` and triage proceeds normally without interrupting the maintainer.

---

## What to do when a prerequisite fails mid-session

If step 1 or 2 passes at start but a later mutation fails with a
Expand Down
Loading