Develop SP approval runbook

[BigLep]

Done Criteria

We have a runbook for SP approvers that covers:

1. How often to check if approved SPs are not meeting approval criteria
2. What to do when they aren't?
   • What channel to message in or where to keep an issue tracking the action
   • How long to wait before unapproving them
   • Commands to run to unapprove them

Why Important

• This supports having more bus factor to this critical area that @TippyFlitsUK has been handling by default.

User/Customer

Notes

1. We can give a time window (e.g., 1–4 hours) to diagnose & see if they recover.
   • Reasons to give a window:
     • Avoid flapping the approved list on minor or transient issues.
     • Allow James to account for known short maintenance, etc.
   • But, from a risk standpoint:
     • If SP is failing for more than that window, better to unapprove to protect users.
2. We'll get automated alerting with Automated "alerting" if an SP should get approved or unapproved dealbot#280, but when the alarm goes off, we still need a runbook for how we handle it.
3. Before automated alerting, we're expecting to manually check twice per day.
4. We need to get a couple of people onboarded onto this process (e.g., Beck, Orjan)

[rjan90]

We have a runbook for how to add/remove SP from the approval list here: https://www.notion.so/filecoindev/FOC-Operational-Excellence-2b7dc41950c1802aa432fff8ecb801cc?source=copy_link#31bdc41950c180c486deef64503debd0

What is lacking is writing down these:

• How often to check if approved SPs are not meeting approval criteria
• What to do when they aren't?
  • What channel to message in or where to keep an issue tracking the action
  • How long to wait before unapproving them

@TippyFlitsUK can you take on this? I have created a section for this here: https://www.notion.so/filecoindev/FOC-Operational-Excellence-2b7dc41950c1802aa432fff8ecb801cc?source=copy_link#325dc41950c1807ab15cc8e4023ba3bf

[TippyFlitsUK]

Hey folks 👋

I’ve gone through this and drafted the missing process layer for approved SP reviews. The existing add/remove runbook already covers the operational steps, so this fills in the gap Oran called out around how often we check, what triggers review, how we communicate, what channel we use, how long we wait, and how we track action.

Proposed approach

• approved SPs should be reviewed at least twice per day until automated alerting is in place
• a review starts when an approved SP drops below any of the published rolling thresholds outside the maintenance windows
• that starts an investigation window of up to 4 hours
• we should not immediately unapprove on the first dip if it looks transient
• approvers can act sooner if the issue is clearly severe, broad, ongoing, user-impacting, or the SP directly confirms downtime / maintenance overrun
• if the SP is an endorsed SP, communication should go to #pdp-endorsed-sp
• otherwise, communication should go to #fil-pdp-mainnet-launch
• DM should only be used as a secondary follow-up path if needed
• every review gets a new GitHub issue in FilOzone/tpm-utils for consistency and audit trail
• if the issue turns out to be a Dealbot/tooling problem, we should link a separate dealbot issue
• if the SP recovers within the investigation window, they stay approved and the issue gets closed as transient
• if they are still below threshold at the end of the window, they should be unapproved immediately using the existing runbook procedure

Current approvers

• James (@TippyFlits)
• Beck (@beck)
• Oran (@Phi-rjan)
• Steve (@biglep)

I’ve also added a short future-improvements note pointing to:

• dealbot#280 for automated approval alerting
• tpm-utils#20 for paging / escalation

https://www.notion.so/filecoindev/FOC-Operational-Excellence-2b7dc41950c1802aa432fff8ecb801cc?source=copy_link#325dc41950c1807ab15cc8e4023ba3bf

[BigLep]

@TippyFlitsUK : sweet - thanks.
I made some edits and left some comments and suggestions in https://www.notion.so/filecoindev/FWSS-Approved-List-327dc41950c1805c9802f70e4877c9b7

[BigLep]

For when we look at this again to close it out, reminder that there is https://filecoinproject.slack.com/archives/C08TVNKJV7C/p1773765824853769 and Beck has created a template we should review (example issues in https://github.com/beck-8/test/issues).

[TippyFlitsUK]

Updates added today - https://www.notion.so/filecoindev/FWSS-Approved-List-327dc41950c1805c9802f70e4877c9b7

[BigLep]

@TippyFlitsUK : thanks for getting updates in Notion and in #19 for review. I don't know if I will get a chance to review before I leave town. I'm good with @rjan90 giving the review/approval.