Skip to content

Deployment readiness audit: production go-live scan before first real users (host undecided) #244

New issue
New issue
Closed
Closed
Deployment readiness audit: production go-live scan before first real users (host undecided)#244
Labels
enhancementNew feature or requestepic-1.5priority:tier1Gates first real userssliceThin vertical work item
Milestone
Phase 1.5 — Hardening

Description

@mforce
mforce
opened on Jul 28, 2026

This was generated by AI during triage.

Context

First real users are ready to start using the app. Hosting target is undecided, and this repo stays host-portable — so the audit separates portable app-hardening (lands in this repo) from host selection + provider config (lands in a separate deployment/ops repo — see the boundary below). Host selection is one of its outputs.

Repo boundary (host-agnostic)

Per AGENTS.md → Host-agnostic repo (deployment boundary): this repo carries no provider-specific config. The checklist below spans two homes:

Already hardened — verify still true under prod config, don't redo

Forwarded headers / HSTS / CSP / pinned AllowedHosts (#144), auth rate limiting (#143), HttpOnly refresh cookie + in-memory access token (#145), ReadOnly PII gating (#127), account lockout (#128), CI vulnerability + SAST gates (#146), telemetry pipeline + metrics (#214, #215), static-asset caching / CDN posture (#141).

Audit checklist — each failing item becomes a linked gap issue

  • Secrets & config: production secret delivery for PEM JWT keys, Seed:* credentials, DB connection string — no user-secrets in prod; per-host env-var story (PemKey.Normalize handles escaped \n).
  • Database: managed Postgres vs self-hosted; automated backups + a restore drill. Epic EPIC: Phase 1.5 — Egg product hardening #15's "Backup/recovery workflow" is still open — decide whether it blocks go-live.
  • Migrations: Database:MigrateOnStartup posture for prod (fine single-instance; multi-instance needs a migration gate). Pre-deploy squash to a single InitialCreate tracked in Squash the 30 EF migrations into a single InitialCreate before first production deploy #245.
  • TLS + domain: cert story, a CDN / edge in front, health-check endpoint for the host's probes.
  • Observability in prod: where OTLP traces/metrics/logs land (hosted Grafana / OTel backend); alerting on error rate and container resources.
  • Provisioning: first-account setup flow for a real farm (seeded admin credentials from Seed:*, not demo data); password rotation after first login.
  • Capacity: instance sizing taken from the day-in-the-life simulation (Release rehearsal: day-in-the-life simulation of 10 users (all roles) with container-resource + HTTP-traffic monitoring #243).
  • Runbook: deploy, rollback, restore-from-backup, JWT key rotation, refresh-token invalidation.
  • Host selection: compare 2–3 candidates (a managed PaaS vs a VPS, …) against the items above; recommend one. → deployment repo.

Additional checks — review-fleet pass (2026-07-28, codex + 2 grounded agents; details in the comment below)

Each item below is a filed, linked gap issue. BLOCKERs must clear before first real users.

BLOCKER

RECOMMENDED

NICE / footguns (not filed separately — see comment): Npgsql pool vs Postgres max_connections (fold into #243) · compose CLUCKWORK_HOST vs CLUCKWORK_ALLOWED_HOSTS name drift · seeded admin's Admin role re-asserted every boot · anonymous /health/ready DB round-trip per hit · no committed prod-config source of truth (a host deploy manifest).

Deliverable

Follow-up security scan — 2026-07-31

A second Graphify-guided code, dependency, secret, container, and deployment review validated the controls already listed above and opened the residual gaps below. No source change is implied by checking an item here; each checkbox closes only when its linked issue acceptance criteria pass.

New app-side gaps

Private deploy/infra follow-up

Provider-specific findings, concrete configuration, topology, private issue identifiers, and remediation status are intentionally tracked in a private deploy/infra tracker and are not enumerated in this public issue.

Existing app issues strengthened by this scan

PostgreSQL baseline follow-up — 2026-07-31

  • PostgreSQL 18 pre-launch baseline (Upgrade pre-launch PostgreSQL baseline from 16 to 18 #321): update disposable dev/CI/simulation/reference databases, use the PG18 /var/lib/postgresql volume target, require the current patched 18.x minor and reviewed digest, and verify migrations, checksums, SCRAM, persistence, health, and pg_stat_statements. This is explicitly not a live production database migration.

Activity

  1. mforce
    added this to the Phase 1.5 — Hardening milestone on Jul 28, 2026
  2. mforce
    added
    enhancementNew feature or request
    sliceThin vertical work item
    on Jul 28, 2026

  3. mforce commented on Jul 28, 2026

    @mforce
    mforce
    on Jul 28, 2026
    OwnerAuthor

    Additional go-live checks — review-fleet pass (codex + 2 grounded Claude agents)

    A three-reviewer sweep of the codebase (codex + 2 repo-grounded Claude agents; the usual pi reviewer's endpoint was down) against this checklist surfaced concerns not yet listed. Deduped and ranked below; the parenthetical (3/3) etc. is how many independent reviewers flagged it — a confidence signal, not a priority. Every item is grounded in a verified file:line. Items already on the checklist or in the "already hardened" list are excluded.

    Each BLOCKER + RECOMMENDED item is filed as a linked gap issue (#260–#274), all under milestone Phase 1.5 and added to epic #15:

    # Item Sev
    #260 Proxy-trust chain (HSTS + rate-limit bucket) BLOCKER
    #261 Railway DATABASE_URL unparseable → crash-loop BLOCKER
    #262 Enforce DB TLS (sslmode) BLOCKER
    #263 Migration/owner role ≠ runtime role BLOCKER
    #264 Farm timezone provisioning + tzdata image dep BLOCKER
    #265 Sole-Owner account recovery (break-glass) BLOCKER
    #266 CI-built/tested image + readiness gate + smoke test BLOCKER
    #267 Container/image hardening (non-root, digests, scan) RECOMMENDED
    #268 JWT multi-key rotation ring + kid RECOMMENDED
    #269 EnableRetryOnFailure on request path RECOMMENDED
    #270 Purge expired refresh_tokens (idempotency = #259) RECOMMENDED
    #271 Worker single-runner guarantee RECOMMENDED
    #272 PII export/erasure/retention (GDPR + PH DPA) RECOMMENDED
    #273 Log redaction/retention/cost + log sink RECOMMENDED
    #274 PWA registration.update() (stale shell) RECOMMENDED

    The NICE-to-have / footgun items below were not filed separately — they stay here for whoever works the audit. The backup strengthening (encryption/retention/PITR/RPO/RTO) folds into this milestone's existing Backup/recovery workflow item rather than a new issue.

    BLOCKER — resolve before first real users

    • Production proxy trust is unsolved — silently kills Security: forwarded headers, HSTS, CSP + security headers, pinned AllowedHosts #144 HSTS and the Security: rate limiting on anonymous auth endpoints (login/refresh) #143 per-IP login limiter (3/3). ForwardedHeaders clears the framework defaults and trusts only RateLimiting:TrustedProxies networks (Program.cs:243-247), but the Railway draft tells operators to leave it empty (deploy-railway-cloudflare.md:159). Result behind Railway/Cloudflare: X-Forwarded-Proto is ignored → Request.IsHttps==false → UseHsts() never emits the header and UseHttpsRedirection() is a no-op; and the login/refresh limiter keys on the proxy's single IP (Program.cs:297) → every user shares one 10-per-15-min bucket (one attacker locks out the farm). Fix per host: pin the exact immediate-proxy CIDR (or platform forwarded-headers), then verify HSTS is emitted, spoofed XFF is rejected, and two clients get two buckets. Own gap issue.

    • Railway DATABASE_URL is a postgresql:// URI Npgsql cannot parse → first deploy crash-loops (1/3, concrete). The draft sets ConnectionStrings__Default=$(DATABASE_URL) and claims Npgsql parses the URL natively (deploy-railway-cloudflare.md:69-75); it does not — Npgsql takes key-value form only, and MigrateAsync() runs eagerly at boot (Program.cs:523-524) so the process dies before listening. No URL→key-value shim exists (repo-wide grep). Fix: map Railway's discrete PG* vars, or add a postgresql://→NpgsqlConnectionStringBuilder shim. Blocks adopting the draft as-is. Own gap issue.

    • App↔DB connection has no enforced TLS (sslmode) (2/3). Provider passes the string straight to UseNpgsql (PostgresDbContextConfigurator.cs:12); compose uses bare Host=db;… (docker-compose.yml:12), draft forwards $(DATABASE_URL) unchecked. Npgsql defaults to SSL Mode=Prefer → silent plaintext fallback, no cert validation → DB password + farm/customer PII exposed over any non-loopback hop. Fix: require SSL Mode=Require (ideally VerifyFull + root cert); fail deploy if the effective mode permits plaintext. Own gap issue.

    • Migration/owner DB role not separated from the runtime role (2/3, codex=BLOCKER). One connection string serves every request and runs boot-time DDL with MigrateOnStartup default-true (Program.cs:523, docker-compose.yml:12 uses POSTGRES_USER for both). An app-tier RCE/injection gets DDL (DROP/ALTER), not just DML. Fix: migrate from a pre-deploy job with an owner credential, set Database:MigrateOnStartup=false in prod, grant the runtime role DML-only. Pairs with the Squash the 30 EF migrations into a single InitialCreate before first production deploy #245 squash. Own gap issue.

    • Seeded farm hardcodes timezone "UTC"; IFarmClock fails closed (2/3). DatabaseSeeder.cs:156 seeds the account at UTC, and FarmClock throws rather than substituting a zone (FarmClock.cs:29-41) — so until an admin sets the real IANA zone, every "today" boundary, the 7-day auto-lock sweep, and FIFO lot availability run on UTC. Also a latent image constraint: the Debian base ships tzdata/ICU, but nothing pins it — an Alpine/chiseled image or InvariantGlobalization=true breaks every farm's date logic at once. Fix: make timezone confirmation a cutover gate (or seed from Seed:TimeZoneId); document "image must contain tzdata"; add a boot smoke assertion that a representative zone resolves. Own gap issue.

    • Sole-Owner account recovery has no supported path (2/3). No email/SMTP anywhere; self-service change needs the current password; an Owner cannot reset their own via the users route (UserEndpoints.cs:32,139); re-seeding never resets an existing admin's password (DatabaseSeeder.cs:193-211). A single-Owner farm that loses the password → DB surgery only. Fix: a tested break-glass CLI (reset user, revoke refresh tokens, rotate security stamp, write an audit row), or require two Owner accounts before cutover. Own gap issue.

    • CI is not a deploy gate; the Docker image is never built or tested in CI (2/3). ci.yml builds/tests source but never builds the multi-stage image (npm ci, dist→wwwroot copy, sourcemap move) — a Dockerfile break first appears at deploy; Railway auto-deploys on push independent of Actions unless "Wait for CI" is enabled. Fix: add a docker build + boot-against-throwaway-Postgres + /health/ready job; require CI before promotion; add a post-deploy smoke test (live/ready, SPA loads, login, refresh cookie, one authed read) through the real Cloudflare domain. Own gap issue.

    Also strengthen the existing backup item: codex notes "daily backups + restore drill" omits encryption, retention, PITR window, off-platform copy, and measured RPO/RTO — capture these under the item / epic #15.

    RECOMMENDED — before or immediately after launch

    • Container runs as root; base images float; final image never vuln-scanned (3/3). No USER in the Dockerfile (Dockerfile:29-37; base ships a non-root app user unused); node:22-alpine/sdk:10.0/aspnet:10.0 by floating tag; Dependabot covers Actions/npm/NuGet but not Docker. Fix: USER $APP_UID, pin bases by digest, add Trivy/Grype on the built image in CI, schedule base-image rebuilds.
    • JWT signing-key rotation has no multi-key/kid support (2/3). One IssuerSigningKey (Program.cs:330), tokens carry no kid (JwtTokenService.cs). The runbook's "JWT key rotation" isn't mechanically possible without 401-ing every live access token. Fix: validation key ring (IssuerSigningKeys) + kid + keep the old public key one access-token lifetime.
    • No transient-DB resilience on the request path (2/3). EnableRetryOnFailure absent (PostgresDbContextConfigurator.cs). The Hardening: DurableJobWorker crash takes down the whole API on a transient DB outage #65 worker recovers from DB blips; HTTP requests throw straight to 500 on a managed-PG failover. Fix: bounded Npgsql retry strategy; wrap the export's explicit RepeatableRead transactions in CreateExecutionStrategy().ExecuteAsync.
    • No Railway/compose readiness health-check wired (1/3). App exposes /health/live + /health/ready (DB+migrations), but there's no railway.json (healthcheckPath) and the compose app service has no healthcheck / the Dockerfile no HEALTHCHECK — so an instance with /health/ready==503 still gets traffic and deploys aren't readiness-gated. Fix: commit railway.json healthcheckPath=/health/ready (+ timeout covering boot migration); add a compose healthcheck.
    • refresh_tokens and idempotency_records grow unbounded — no purge (2/3). Only sweep is DailyEntryLockSweep; nothing deletes expired refresh tokens or old idempotency rows (which store the full response body forever, IdempotencyMiddleware.cs:109). Months → index bloat + indefinite payload retention. Fix: a periodic worker sweep with retention windows (idempotency past the client-retry window; refresh lineage past reuse-detection need).
    • Background worker has no single-runner guarantee (2/3). Every instance registers DurableJobWorker (Program.cs:500); job poll + DailyEntryLockSweep claim no rows (no FOR UPDATE SKIP LOCKED/lease). Two instances double-run. Fix: document "single API instance" as a deploy invariant now; add advisory-lock/SKIP LOCKED claiming before any horizontal scale. (Distinct from the checklist's multi-instance migration gate — this is runtime jobs.)
    • No personal-data export / erasure / retention policy (2/3, market-dependent). Stores customer name/phone/email/address + free-text notes (Customer.cs) and user emails; only an admin farm-wide CSV export (F24: CSV export / manual backup — per-dataset CSVs, full account export (zip), self-host dump instructions #95), no per-subject DSAR or erasure; AuditEvent pins ActorEmail immutably. No retention rule. Relevant to EU/UK GDPR and the PH Data Privacy Act (the es/tl locales point at PH). Fix: classify fields, set retention, define export + erasure-vs-pseudonymization with a documented audit-trail exception, propagate deletes into backups/telemetry.
    • Production log redaction / retention / cost undefined (2/3). Serilog is Console-only (appsettings.json) — no OTLP log sink, so structured logs (Observability: structured logs at domain state transitions #216) land only in the host's console with host-default retention. The anonymous /client-errors endpoint writes caller-controlled messages + full stacks (ClientErrorEndpoints.cs:96) and boot logs real admin emails (DatabaseSeeder.cs:193). Fix: define forbidden fields + redaction, retention, access control, and a volume/cost budget for the chosen sink.
    • PWA never re-checks for updates on a long-lived session (1/3). SW config is correct (prompt, no skipWaiting) but nothing calls registration.update() — a barn phone left open all day in standalone mode generates no navigation, so updatefound never fires and the UpdatePrompt never appears; the client runs the old shell against the new /api/v1. Fix: periodic registration.update() + one on visibilitychange→visible; keep prompt-not-auto behavior.

    NICE-TO-HAVE / footguns

    • Npgsql pool size vs Postgres max_connections undocumented (2/3) — default 100/instance; fold into the capacity note (Release rehearsal: day-in-the-life simulation of 10 users (all roles) with container-resource + HTTP-traffic monitoring #243) so a future 2nd instance can't exhaust connections.
    • Compose prod env-var name drift (1/3) — docker-compose.yml/.env.example read CLUCKWORK_HOST, but deploy/.env sets CLUCKWORK_ALLOWED_HOSTS → --profile prod deploy gets empty AllowedHosts → 400s on the real domain. Reconcile the name.
    • Seeded admin's Admin role re-asserted every boot (1/3) — DatabaseSeeder re-adds the role to the existing admin each start; a deliberate demotion is undone by a restart. Stop re-asserting for existing users.
    • Anonymous, unthrottled /health/ready does a DB round-trip per hit (1/3) — minor load amplifier; cache the readiness result a few seconds or accept it behind Cloudflare.
    • No committed prod-config source of truth (1/3) — all Railway config lives in the dashboard; a committed railway.json + env-parity table would close the drift (already seen in TrustedProxies). Folds into the deploy-gate item.

    Verified-clean — do NOT re-open (reviewer consensus)

    CORS (N/A — same-origin SPA from wwwroot) · prod error leakage (generic Results.Problem, no dev page) · /client-errors (CRLF-sanitized, byte-capped) · session invalidation on password change (all refresh tokens revoked in-transaction; documented residual: an access token stays valid ≤15 min after revocation, by design) · refresh-token theft (hash-only, rotation, reuse-detection + family revoke, xmin CAS) · #65 worker/DB-outage crash (fixed) · SW skipWaiting hazard (not present — only the update-detection gap above) · auth cookie Secure (env-based, safe behind the plain-HTTP proxy hop) · demo data leaking to prod (gated on Seed:Demo=false + empty catalog).

  4. mforce
    added 4 commits that reference this issue on Jul 30, 2026

    feat(api): farm timezone provisioning + tzdata/ICU boot guard (#264) (#…

    418df65

    feat(api): migrate command + prod migration split (#263) + extract CL…

    7e8f9df

    feat(api): fail Production boot when RateLimiting:TrustedProxies is e…

    a3c2fb8

    docs(agents): drop the private deploy repo's name from the host-agnos…

    67be827
  5. mforce
    added
    epic-15
    epic-1.5
    and removed
    epic-15
    on Aug 12, 2026

  6. mforce commented on Aug 16, 2026

    @mforce
    mforce
    on Aug 16, 2026
    OwnerAuthor

    Checklist item Provisioning: first-account setup flow for a real farm is superseded by #533 (slice 3 of epic #530, Phase 1.6 — Multi-farm tenancy).

    #533 adds provision-account — a run-then-exit CLI verb that creates an account, its per-account reference data (egg grades, packed-unit conversions) and its Owner with a freshly generated password printed to stdout only, all in one transaction and attributed to a system actor per #500. That is the real answer to "first-account setup flow for a real farm", and it generalises: farm #2 onward gets the same path rather than a hand-written INSERT.

    Note the checklist item as written still references Seed:* credentials, which #283 retired along with the runtime seeder. The current answer is bootstrap-admin for the default account and provision-account for every farm after it.

    Tick this item when #533 merges.

    Two other items on this audit are worth re-reading in light of #530:

  7. mforce
    added
    priority:tier1Gates first real users
    on Aug 31, 2026

  8. mforce commented on Sep 13, 2026

    @mforce
    mforce
    on Sep 13, 2026
    OwnerAuthor

    Closing — the gate this issue describes has already been passed

    Closed on 2026-09-13 during the issue cleanup, with the status recorded rather than assumed.

    This was written on 2026-07-28 as a pre-go-live scan: "production go-live scan before first
    real users."     The app is now in production. A gate that has been passed cannot be a gate; leaving
    it open as tier1 made the four-issue tier1 tier read as "blocking launch" when launch happened.

    The 15 gap issues it spawned — verified 2026-09-13, not recalled

    13 closed:

    #
    #260 proxy-trust boot guard
    #261 URI-form connection strings
    #262 Production Postgres TLS floor
    #263 migration/runtime DB role split + migrate verb
    #264 farm timezone provisioning, tzdata/ICU
    #265 break-glass recover-admin
    #266 CI-built image + readiness health check
    #267 container hardening — non-root, pinned digests, Trivy
    #269 transient-DB retry boundary
    #270 expired refresh-token purge sweep
    #271 single-runner leader gate
    #273 log redaction, retention, sink
    #274 PWA registration.update()

    Each carries a decision record in docs/decisions/.

    2 open, and they do not need this tracker:

    Both are self-contained and reviewed on their own terms.

    What genuinely remains is not in this repo

    The rest of this issue's cross-references point at cluckwork-deploy: trusted-proxy CIDRs (#5),
    deploy manifest with migrate-before-serve and the readiness gate (#6), post-deploy smoke test
    through the production URL (#7), CDN/DNS/TLS edge (#8), backups — encryption, retention, PITR,
    declared RPO/RTO, restore drill (#9), and the operational runbook (#12).

    Per AGENTS.md's host-agnostic boundary, provider manifests, IaC, edge config and concrete
    environment values belong in that repo, not this one. Tracking them from a Cluckwork issue was
    always going to double-count them.

    The one worth flagging out loud: cluckwork-deploy#9, backups and a restore drill. An app in
    production without a verified restore is the highest-severity item anywhere in this cleanup, and it
    is not a Cluckwork issue.

    Not claimed

    This close-out does not claim the deployment is fully hardened. It claims the audit is
    finished and its findings are dispositioned. A fresh post-go-live review — backups verified,
    restore drill executed, runbook current — is worth doing and would be a new issue with today's
    evidence, not a two-month-old pre-launch scan.

  9. mforce
    closed this as completedon Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestepic-1.5priority:tier1Gates first real userssliceThin vertical work item

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions