diff --git a/README.md b/README.md index c5065b60160b0..5dc403353838d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,27 @@ +# T3 Turbo — Self-Hosted Infrastructure (`infra/t3turbo-relay`) + +> **This branch is the T3 Turbo operator repo.** It carries the complete, versioned +> kit for standing up the self-hosted stack: Cloudflare relay + tunnel, Clerk auth, +> and self-hosted Supabase Postgres — no Axiom, no APNs (Android-only operator). + +## Start here + +| What | Where | +| ------------------------------------------------------------------------------ | ---------------------------------------------------------------- | +| Master runbook (bring-up order, architecture, threat model, secrets inventory) | [`infra/README.md`](infra/README.md) | +| Rebase seam — upstream files we modify + conflict guidance | [`SEAM.md`](SEAM.md) | +| Supabase: schema, seed, RLS, setup rules | [`infra/supabase/`](infra/supabase/) | +| Cloudflare Tunnel configs + systemd unit | [`infra/cloudflared/`](infra/cloudflared/) | +| GitHub/Cloudflare vars & secrets checklist (names only) | [`infra/cloudflare/CHECKLIST.md`](infra/cloudflare/CHECKLIST.md) | +| Relay deploy guide (Hyperdrive, Clerk, verification) | [`infra/relay/DEPLOY.md`](infra/relay/DEPLOY.md) | + +**Hard rule:** no secret values in this repository — ever. Placeholders in `` +are resolved from the operator's private vault at deploy time. + +--- + +_Upstream T3 Code README follows below (kept for rebase context)._ + # T3 Code T3 Code is an "agent harness control surface". It enables control of the agents on your machine with a best-in-class mobile app ([iOS](https://apps.apple.com/us/app/t3-code-remote-claude-more/id6787819824), [Android](https://play.google.com/store/apps/details?id=com.t3tools.t3code)), [web app](https://app.t3.codes) and [Electron-based desktop app](https://t3.codes). diff --git a/SEAM.md b/SEAM.md new file mode 100644 index 0000000000000..4efb1422e365c --- /dev/null +++ b/SEAM.md @@ -0,0 +1,39 @@ +# Upstream seam + +`git show --name-status 774c53df b4904491` identifies these 16 unique upstream-owned paths. The +label is the nature of this fork's change. + +- **Gated** `.github/workflows/deploy-relay.yml` — fork-safe credential detection, manual dispatch, + and external-database/optional-service inputs. +- **Gated** `.github/workflows/release.yml` — release artifacts remain buildable when Connect is not + configured. +- **Gated** `docs/operations/release.md` — documents the optional Connect release path. +- **Additive** `docs/operations/self-host-relay.md` — self-hosting runbook. +- **Optional** `infra/relay/.env.example` — external PostgreSQL settings and optional Axiom/APNs + groups. +- **Optional** `infra/relay/alchemy.run.ts` — provider layers and tracing outputs exist only when + configured. +- **Optional** `infra/relay/scripts/deploy.test.ts` — covers deployments without tracing outputs. +- **Optional** `infra/relay/scripts/deploy.ts` — accepts absent tracing outputs while retaining the + relay URL requirement. +- **Optional** `infra/relay/src/Config.ts` — APNs is a complete optional group. +- **Optional** `infra/relay/src/agentActivity/ApnsDeliveries.test.ts` — adapts delivery tests to the + optional APNs type. +- **Optional** `infra/relay/src/agentActivity/ApnsDeliveries.ts` — supplies the disabled APNs layer. +- **Additive** `infra/relay/src/agentActivity/ApnsDisabled.test.ts` — proves disabled delivery is a + no-op. +- **Gated** `infra/relay/src/db.ts` — a complete `DATABASE_*` group selects external PostgreSQL; + otherwise the upstream PlanetScale path remains. +- **Additive** `infra/relay/src/deploymentConfiguration.test.ts` — covers complete and absent + provider groups. +- **Optional** `infra/relay/src/observability.ts` — Axiom resources require the complete pair. +- **Optional** `infra/relay/src/worker.ts` — APNs queues and tracing layers are conditional. + +## Nightly sync conflicts + +Resolve against the new upstream file first, then reapply only the behavior above; never take the +fork's whole file over a newer upstream implementation. Drop a fork hunk when upstream now provides +equivalent gating or optionality. Preserve the PlanetScale fallback, the Cloudflare credential +gates, and the all-or-nothing `DATABASE_*`, Axiom, and APNs groups. Re-run the focused relay tests +and compare workflow variable/secret names after every conflict. Never use a credential, generated +state, or production output to resolve a conflict. diff --git a/infra/README.md b/infra/README.md new file mode 100644 index 0000000000000..06d5bead426f0 --- /dev/null +++ b/infra/README.md @@ -0,0 +1,412 @@ +# T3 Turbo self-hosted infrastructure + +This directory is the operator runbook for `t3turbo.pro`. It covers the locked deployment: + +- Cloudflare DNS, a relay Worker, Hyperdrive, and Cloudflare Tunnel +- Clerk authentication +- self-hosted Supabase Docker with PostgreSQL 17 and persistent host storage +- `relay.t3turbo.pro` for the relay API and `app.t3turbo.pro` for the hosted T3 surface +- an Android-only operator deployment, with no APNs and no Axiom + +No secret value belongs in this repository. Values shown between angle brackets are replacement +markers and must remain outside Git history after they are resolved. + +## Supported minimum stack + +The relay source on this branch directly supports the locked deployment: + +- A complete `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, and + `DATABASE_PASSWORD` set selects external PostgreSQL and causes Alchemy to create Hyperdrive for + that origin. It takes precedence over the legacy PlanetScale fallback. +- Omitting the complete Axiom variable/secret pair disables Axiom resources and trace export. +- Omitting every APNs variable/secret disables APNs queues and delivery at deployment and runtime. + +For T3 Turbo, provide the complete `DATABASE_*` set and leave every PlanetScale, Axiom, and APNs +setting unset. An incomplete database set is not a partial configuration: the relay falls back to +the legacy PlanetScale path and the deploy fails without its credentials. + +## Architecture + +```text + Clerk production instance + JWT template: t3-relay + aud: t3-code-relay + ^ + | token verification + | + Android / web / desktop clients | + | | + +---- https://relay.t3turbo.pro --------+---- Cloudflare Worker + | | + | | Hyperdrive + | v + | db.t3turbo.pro:15432 + | public TLS, Cloudflare IP ACL + | | + | v + | Supabase PostgreSQL 17 + | /srv/t3turbo/supabase/volumes + | + +---- https://app.t3turbo.pro ---- Cloudflare edge + | + | outbound tunnel + v + cloudflared on Ubuntu + | + v + T3 server / hosted app + + Normal environment API and WebSocket traffic goes directly through each linked environment's + managed tunnel. The relay is a control plane, not a terminal or filesystem traffic proxy. +``` + +`relay.t3turbo.pro` is a Worker custom domain. `app.t3turbo.pro` is a Cloudflare Tunnel hostname. +They must not be assigned to the same DNS record or tunnel ingress rule. + +## Threat model + +TLS certificate issuance records hostnames in public Certificate Transparency logs. Treat +`relay.t3turbo.pro`, `app.t3turbo.pro`, and `db.t3turbo.pro` as discoverable; unguessable DNS names +are not a security control. + +- Put deny-by-default Cloudflare Access (Zero Trust) applications and policies in front of both + the relay and app hostnames before production use. The relay policy must use an authentication + path that every intended web, CLI, and Android client can satisfy, such as enrolled Cloudflare + One devices; never embed an Access service token in a public client build. +- Keep application origin IPs out of public DNS and behind the outbound-only Cloudflare Tunnel. + Bind the local T3 service to loopback. The DNS-only PostgreSQL endpoint is the explicit exception. +- Restrict the PostgreSQL listener to Cloudflare's complete, current published IPv4 and IPv6 ranges + plus explicitly approved operator addresses, with default-deny firewalling everywhere else. +- Put no secret in DNS, source code, examples, comments, issue text, workflow output, or logs. + Hostnames, account IDs, and public client identifiers must never be treated as credentials. + +## Components + +| Component | Responsibility | Durable state | +| ----------------- | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | +| Cloudflare DNS | Hosts `t3turbo.pro` records and Worker/tunnel hostnames. | Cloudflare account | +| Cloudflare Access | Enforces deny-by-default Zero Trust policy before requests reach the relay or hosted app. | Cloudflare account | +| Relay Worker | Authenticates clients, links environments, issues short-lived credentials, and manages endpoint metadata. | Worker configuration plus PostgreSQL | +| Hyperdrive | Pools Worker-to-PostgreSQL connections. Query caching is disabled by the current relay source. | Cloudflare account | +| Clerk | Provides web/mobile sessions, the relay JWT template, and public CLI OAuth. | Clerk production instance | +| Supabase Docker | Runs PostgreSQL 17 and the supporting self-hosted Supabase services. | Host bind mounts and Docker named volumes | +| `cloudflared` | Provides outbound-only connectivity for `app.t3turbo.pro` and optional protected operator services. | Tunnel record in Cloudflare and scoped credential on Ubuntu | +| GitHub Actions | Builds releases and reconciles the Worker from the locked external-database mode. | Repository/environment variables and secrets | + +## Bring-up order + +### Phase 0: establish custody and recovery + +1. Assign one password manager vault for infrastructure secrets and an independent encrypted backup + destination. +2. Record the Ubuntu host owner, Cloudflare account owner, Clerk owner, GitHub environment + reviewers, and recovery contacts. +3. Confirm the Ubuntu host has at least 4 CPU cores, 8 GB RAM, an SSD-backed persistent filesystem, + Docker Engine, Docker Compose v2, Git, OpenSSL, `curl`, and PostgreSQL 17 client tools. +4. Patch Ubuntu and configure time synchronization before issuing certificates or tokens. + +Verification: + +```sh +docker version +docker compose version +openssl version +psql --version +timedatectl status +``` + +Rollback: no service state exists yet. Remove only packages installed for this phase; do not erase +the server if it contains unrelated workloads. + +### Phase 1: activate Cloudflare and reserve names + +1. Add `t3turbo.pro` to the intended Cloudflare account and complete nameserver activation. +2. Reserve these non-conflicting names: + - `relay.t3turbo.pro`: Worker custom domain; no pre-existing CNAME. + - `app.t3turbo.pro`: Cloudflare Tunnel DNS route. + - `db.t3turbo.pro`: DNS-only database TLS endpoint when using the locked public-endpoint path. +3. Create the least-privilege deployment token described in + [cloudflare/CHECKLIST.md](./cloudflare/CHECKLIST.md). Store it only in the password manager and + GitHub Actions. +4. Create self-hosted Cloudflare Access applications for `relay.t3turbo.pro` and + `app.t3turbo.pro`. Attach deny-by-default policies for the intended operators and enrolled + devices. Prove the chosen relay policy works with every client surface before deployment. +5. Decide the managed endpoint zone. The current code requires `RELAY_TUNNEL_ZONE_NAME`; using + `t3turbo.pro` is valid, though a delegated child zone reduces blast radius. + +Verification: + +```sh +dig +short NS t3turbo.pro +dig +short relay.t3turbo.pro +dig +short app.t3turbo.pro +``` + +The nameservers must be Cloudflare-owned. Relay and app may remain unresolved until later phases. + +Rollback: delete only records created for this deployment. Do not delete the zone if it hosts other +services. + +### Phase 2: install self-hosted Supabase + +Follow [supabase/README.md](./supabase/README.md) from a persistent directory such as +`/srv/t3turbo/supabase`. Pin the Supabase Git commit, generate every `.env` secret, set +`KONG_HTTPS_PORT=18443`, and start the stack. Fresh current Supabase Docker deployments use +PostgreSQL 17. + +Verification: + +```sh +cd /srv/t3turbo/supabase +docker compose ps +docker compose exec db pg_isready -U postgres +docker compose exec db psql -U postgres -d postgres -Atc 'show server_version' +``` + +Every required container must be healthy, `pg_isready` must succeed, and the server version must +start with `17`. + +Rollback: use `docker compose down` to stop containers while retaining state. Never use +`reset.sh`, `docker compose down -v`, or delete `volumes/` during a routine rollback. + +### Phase 3: install the relay schema and RLS + +From the Supabase project directory, apply the checked-in SQL in order: + +```sh +docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/schema.sql +docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/seed.sql +docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/rls.sql +``` + +Create a dedicated login role, set its password interactively, and grant it membership in +`service_role`: + +```sh +docker compose exec db psql -U postgres -d postgres +``` + +Then run in `psql`: + +```sql +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'relay_runtime') THEN + CREATE ROLE relay_runtime LOGIN; + END IF; +END +$$; +\password relay_runtime +GRANT service_role TO relay_runtime; +``` + +Verification: + +```sh +docker compose exec db psql -U postgres -d postgres -c \ + "select count(*) as relay_tables from pg_tables where schemaname = 'public' and tablename like 'relay_%';" +docker compose exec db psql -U postgres -d postgres -c \ + "select tablename, rowsecurity from pg_tables where schemaname = 'public' and tablename like 'relay_%' order by tablename;" +``` + +The table count must be `9`, and `rowsecurity` must be true for all nine tables. + +Rollback: restore a pre-schema logical backup or drop only the nine `relay_*` tables after proving +they contain no required data. Do not reset the Supabase database. + +### Phase 4: configure database TLS, firewalling, and Hyperdrive + +Follow [supabase/SETUP-RULES.md](./supabase/SETUP-RULES.md). The locked path is a public PostgreSQL +TLS endpoint whose firewall admits the current Cloudflare IP ranges and explicitly approved +operator addresses only. Use a WebPKI certificate whose SAN includes `db.t3turbo.pro`, and connect +as `relay_runtime`, never as `postgres`. The current relay +creates Hyperdrive with `sslmode=require`, caching disabled, and an origin connection limit of 20. + +Cloudflare documents three supported network paths: unrestricted public access, public access with +a Cloudflare IP ACL, and a private database via Cloudflare Tunnel. The second is the locked path. +The third is the preferred alternative when the deployment policy is later allowed to change. + +Verification: + +- A non-allowlisted network cannot open the database port. +- A TLS client receives the expected certificate and hostname. +- An allowlisted operator can authenticate as `relay_runtime` and query the relay tables. + +After Phase 7, Hyperdrive's connection must succeed and `pg_stat_activity` must show +`application_name = 'Cloudflare Hyperdrive'` after a relay query. + +Rollback: close the public database firewall rule and keep the database running locally. Do not +delete the database or revoke the current login until any deployed Worker is confirmed quiescent. + +### Phase 5: configure Clerk + +1. Create or select a Clerk production instance for `t3turbo.pro`. +2. Create the `t3-relay` JWT template with claims `{ "aud": "t3-code-relay" }`. +3. Create a public CLI OAuth application using PKCE with `openid`, `profile`, and `email` scopes. +4. Allow both `http://127.0.0.1:34338/callback` and + `https://app.t3turbo.pro/connect/callback`. +5. Configure the Android/native application and restrict production origins and sign-ups to the + intended operator population. +6. Enroll intended Android devices in the Cloudflare One/Access path selected in Phase 1 and prove + that native relay requests pass the outer Access policy. +7. Store the publishable key as a GitHub variable and the secret key as a GitHub secret. Never put + the secret key in a client build. + +Verification: obtain a token with template `t3-relay`, decode it locally without logging it, and +confirm `aud` is `t3-code-relay`, `iss` is the production Clerk issuer, and the token is unexpired. + +Rollback: disable the OAuth application and template only after clients have been pointed away from +the relay. Rotating the Clerk secret key is safer than deleting the production instance. + +### Phase 6: install the application tunnel + +Follow [cloudflared/README.md](./cloudflared/README.md). Create a tunnel for the Ubuntu-hosted T3 +surface, route `app.t3turbo.pro`, install the systemd service, and place Cloudflare Access in front +of any optional SSH or raw TCP hostname. + +Do not route `relay.t3turbo.pro` to this tunnel while the relay Worker owns that custom domain. Put +the required Access application and policy in place before routing `app.t3turbo.pro`. The example +configuration includes a clearly marked, mutually exclusive relay-origin fallback only for a future +non-Worker deployment. + +Verification: + +```sh +systemctl is-active cloudflared +cloudflared tunnel --config /etc/cloudflared/config.yml ingress validate +cloudflared access curl https://app.t3turbo.pro/ +``` + +Rollback: remove the `app.t3turbo.pro` tunnel DNS route or stop `cloudflared`. Existing local T3 +service data remains intact. + +### Phase 7: deploy the relay Worker + +Use [relay/DEPLOY.md](./relay/DEPLOY.md). Configure all four `DATABASE_*` variables and the +`DATABASE_PASSWORD` secret. Alchemy creates Hyperdrive with caching disabled and an origin +connection limit of 20. Leave PlanetScale, Axiom, and APNs settings unset; the deployment report +must state that tracing and mobile push are disabled. + +Verification: + +```sh +cloudflared access curl https://relay.t3turbo.pro/health +cloudflared access curl \ + https://relay.t3turbo.pro/.well-known/oauth-authorization-server +cloudflared access curl \ + https://relay.t3turbo.pro/.well-known/oauth-protected-resource +``` + +The health response must be `{"ok":true,"service":"relay"}`. + +Rollback: redeploy the last known-good Worker version or detach its custom domain. Do not roll back +the database schema unless the previous Worker is schema-incompatible and a tested backup exists. + +### Phase 8: configure and verify releases + +Populate the exact locked-stack variables and secrets in +[cloudflare/CHECKLIST.md](./cloudflare/CHECKLIST.md). Build clients with: + +- `T3CODE_CLERK_PUBLISHABLE_KEY` +- `T3CODE_CLERK_JWT_TEMPLATE=t3-relay` +- `T3CODE_CLERK_CLI_OAUTH_CLIENT_ID` +- `T3CODE_RELAY_URL=https://relay.t3turbo.pro` + +Verification: + +1. The Release relay-public-config job succeeds rather than skipping. +2. A built Android client signs in and lists linked environments. +3. `t3 connect status --json` shows the intended environment. +4. Normal API and WebSocket traffic reaches the environment tunnel, not the relay Worker. + +Rollback: ship a client build with Connect public configuration disabled or point a new build at the +last known-good relay. Build-time public values cannot be changed inside an already shipped binary. + +## Routine verification + +Run these checks after upgrades and at least weekly: + +- `docker compose ps` and `pg_isready` on the Ubuntu host. +- PostgreSQL backup age, size, encryption, and a scheduled restore test. +- TLS expiry for the database endpoint and the two public HTTPS hostnames. +- Access-authenticated `GET https://relay.t3turbo.pro/health`, plus a denied request from an + unmanaged client. +- Cloudflare Tunnel connector health and systemd restart count. +- Hyperdrive origin health and connection count. +- Clerk sign-in, JWT audience, and CLI OAuth callback. +- GitHub production-environment secret review and stale credential removal. + +## Rebase safety and state ownership + +The branch is safe to rebase only when code and operator state remain separate. + +Lives in Git: + +- relay source, migrations, and these runbooks; +- SQL schema, empty seed contract, and RLS policy definitions; +- example tunnel and systemd files containing placeholders only; +- workflow _names_ and variable/secret _names_. + +Lives in Cloudflare: + +- zones, DNS records, Worker versions, custom domains, queues, Hyperdrive configuration; +- tunnels, tunnel credentials, Access policies, certificates, and Alchemy state. + +Lives in GitHub variables or the `production` environment: + +- public account IDs, domains, Clerk publishable identifiers, template name, audience, and public + OAuth client ID. + +Lives in GitHub secrets, the server, or the password manager: + +- API tokens, Clerk secret, database passwords, Supabase `.env`, TLS private keys, tunnel + credentials, and backup-encryption material. + +Never resolve a rebase conflict by committing `.env`, Alchemy state, a Wrangler state directory, +tunnel JSON, `cert.pem`, a database dump, a private key, or a rendered connection string. Recheck +workflow input names against the workflow files after every upstream rebase. A rebase must not +delete or recreate retained Cloudflare or PostgreSQL state. + +## Secrets inventory + +This inventory names secrets; it never records their values. + +| Secret name | Authoritative location | Consumers | +| ------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| `CLOUDFLARE_API_TOKEN` | GitHub Actions secret and operator password manager | Worker/Hyperdrive/DNS deployment | +| `CLOUDFLARE_ACME_DNS_TOKEN` | `/root/.secrets/certbot/cloudflare.ini` mode `0600` and password manager | DNS-01 renewal for `db.t3turbo.pro` | +| `CLOUDFLARE_TUNNEL_TOKEN` | `/etc/cloudflared/tunnel.env` mode `0600`, or password manager | Remotely managed `cloudflared` service | +| `CLOUDFLARE_TUNNEL_CREDENTIALS` | `/etc/cloudflared/.json`, root-owned and mode `0640` for the dedicated `cloudflared` group | Locally managed tunnel runtime | +| `CLOUDFLARE_ORIGIN_CERT` | `/etc/cloudflared/cert.pem` mode `0600`; remove from runtime host when no longer needed | Locally managed tunnel creation/administration only | +| `CLERK_SECRET_KEY` | GitHub `production` environment secret and Clerk | Relay Worker backend verification | +| `POSTGRES_PASSWORD` | `/srv/t3turbo/supabase/.env` mode `0600` and password manager | Supabase services and administration | +| `DATABASE_PASSWORD` | GitHub `production` environment secret, password manager, and PostgreSQL role verifier | Hyperdrive origin login for `relay_runtime` | +| `JWT_SECRET` | Supabase `.env` and password manager | Legacy Supabase HS256 signing/verification | +| `ANON_KEY` | Supabase `.env`; treat as public-but-rotatable credential | Supabase `anon` API role | +| `SERVICE_ROLE_KEY` | Supabase `.env` and password manager | Supabase privileged API role | +| `SUPABASE_SECRET_KEY` | Supabase `.env` and password manager when asymmetric keys are enabled | Server-side Supabase API | +| `JWT_KEYS` | Supabase `.env` and password manager when asymmetric keys are enabled | Supabase Auth signing keys | +| `DASHBOARD_PASSWORD` | Supabase `.env` and password manager | Supabase Studio basic authentication | +| `SECRET_KEY_BASE` | Supabase `.env` and password manager | Realtime/Supavisor cryptography | +| `REALTIME_DB_ENC_KEY` | Supabase `.env` and password manager | Realtime sensitive database fields | +| `VAULT_ENC_KEY` | Supabase `.env` and password manager | Supavisor configuration encryption | +| `PG_META_CRYPTO_KEY` | Supabase `.env` and password manager | Studio/postgres-meta connection-string encryption | +| `LOGFLARE_PUBLIC_ACCESS_TOKEN` | Supabase `.env` and password manager | Supabase local logs override, if enabled; not Axiom | +| `LOGFLARE_PRIVATE_ACCESS_TOKEN` | Supabase `.env` and password manager | Supabase local logs override, if enabled; not Axiom | +| `S3_PROTOCOL_ACCESS_KEY_ID` | Supabase `.env` and password manager | Supabase Storage S3 protocol | +| `S3_PROTOCOL_ACCESS_KEY_SECRET` | Supabase `.env` and password manager | Supabase Storage S3 protocol | +| `POSTGRES_TLS_PRIVATE_KEY` | PostgreSQL `db-config` volume, ACME source directory, and encrypted backup | Native PostgreSQL TLS | +| `BACKUP_ENCRYPTION_KEY` | Offline/password-manager custody, separate from backup storage | Database and configuration backups | +| `CloudMintKeyPair.privateKey` | Alchemy/Cloudflare encrypted state | Relay-issued cloud credentials | +| `ApnsDeliveryJobSigningSecret` | Alchemy/Cloudflare encrypted state only when APNs is enabled | Optional APNs queue signing; not created for the locked target | +| `PLANETSCALE_API_TOKEN_ID` | GitHub Actions secret only for the legacy PlanetScale fallback | Optional legacy database deployment; unset for the locked target | +| `PLANETSCALE_API_TOKEN` | GitHub Actions secret only for the legacy PlanetScale fallback | Optional legacy database deployment; unset for the locked target | +| `AXIOM_TOKEN` | GitHub Actions secret only when paired with `AXIOM_ORG_ID` | Optional hosted tracing; unset for the locked target | +| `APNS_PRIVATE_KEY` | GitHub `production` environment secret only with the complete APNs set | Optional Apple push; unset for the locked target | + +`CLERK_PUBLISHABLE_KEY`, `CLERK_JWT_TEMPLATE`, `CLERK_JWT_AUDIENCE`, +`CLERK_CLI_OAUTH_CLIENT_ID`, Cloudflare account IDs, zone names, hostnames, and Hyperdrive IDs are +configuration identifiers, not secrets. diff --git a/infra/cloudflare/CHECKLIST.md b/infra/cloudflare/CHECKLIST.md new file mode 100644 index 0000000000000..99975e4942fb2 --- /dev/null +++ b/infra/cloudflare/CHECKLIST.md @@ -0,0 +1,194 @@ +# Cloudflare and GitHub Actions checklist + +This file is generated from the names referenced by `.github/workflows/deploy-relay.yml` and +`.github/workflows/release.yml` at this branch point. The lists contain names only so the file can +be shared safely. Values belong in GitHub repository settings or the `production` environment, +never in Git. + +## Locked-stack selection + +The current relay supports external PostgreSQL and makes Axiom/APNs optional. For T3 Turbo, set the +complete `DATABASE_*` group below and leave every PlanetScale, Axiom, and APNs setting unset. The +legacy and optional names remain in the exact workflow inventory because the workflow references +them; they are not required for this deployment. + +## `.github/workflows/deploy-relay.yml` + +Variables: + +- `APNS_BUNDLE_ID` +- `APNS_ENVIRONMENT` +- `APNS_KEY_ID` +- `APNS_TEAM_ID` +- `AXIOM_ORG_ID` +- `CLERK_JWT_AUDIENCE` +- `CLERK_PUBLISHABLE_KEY` +- `CLOUDFLARE_ACCOUNT_ID` +- `DATABASE_HOST` +- `DATABASE_NAME` +- `DATABASE_PORT` +- `DATABASE_USER` +- `PLANETSCALE_ORGANIZATION` +- `RELAY_API_ZONE_NAME` +- `RELAY_DOMAIN` +- `RELAY_TUNNEL_ZONE_NAME` + +Secrets: + +- `APNS_PRIVATE_KEY` +- `AXIOM_TOKEN` +- `CLERK_SECRET_KEY` +- `CLOUDFLARE_API_TOKEN` +- `DATABASE_PASSWORD` +- `PLANETSCALE_API_TOKEN` +- `PLANETSCALE_API_TOKEN_ID` + +## `.github/workflows/release.yml` + +Variables: + +- `APPLE_TEAM_ID` +- `CLERK_CLI_OAUTH_CLIENT_ID` +- `CLERK_JWT_TEMPLATE` +- `CLERK_PASSKEY_RP_DOMAINS` +- `CLERK_PUBLISHABLE_KEY` +- `CLOUDFLARE_ACCOUNT_ID` +- `RELAY_API_ZONE_NAME` +- `RELAY_DOMAIN` +- `T3CODE_WEB_LATEST_DOMAIN` +- `T3CODE_WEB_NIGHTLY_DOMAIN` +- `T3CODE_WEB_ROUTER_URL` +- `VERCEL_TEAM_SLUG` + +Secrets: + +- `APPLE_API_ISSUER` +- `APPLE_API_KEY` +- `APPLE_API_KEY_ID` +- `AZURE_CLIENT_ID` +- `AZURE_CLIENT_SECRET` +- `AZURE_TENANT_ID` +- `AZURE_TRUSTED_SIGNING_ACCOUNT_NAME` +- `AZURE_TRUSTED_SIGNING_CERTIFICATE_PROFILE_NAME` +- `AZURE_TRUSTED_SIGNING_ENDPOINT` +- `AZURE_TRUSTED_SIGNING_PUBLISHER_NAME` +- `CLOUDFLARE_API_TOKEN` +- `CSC_KEY_PASSWORD` +- `CSC_LINK` +- `DISCORD_RELEASE_LATEST_ROLE_ID` +- `DISCORD_RELEASE_NIGHTLY_ROLE_ID` +- `DISCORD_RELEASE_WEBHOOK_URL` +- `MACOS_PROVISIONING_PROFILE` +- `RELEASE_APP_ID` +- `RELEASE_APP_PRIVATE_KEY` +- `VERCEL_ORG_ID` +- `VERCEL_PROJECT_ID` +- `VERCEL_TOKEN` + +## Locked relay/release configuration + +Required variables: + +- `CLERK_CLI_OAUTH_CLIENT_ID` +- `CLERK_JWT_AUDIENCE` +- `CLERK_JWT_TEMPLATE` +- `CLERK_PUBLISHABLE_KEY` +- `CLOUDFLARE_ACCOUNT_ID` +- `DATABASE_HOST` +- `DATABASE_NAME` +- `DATABASE_PORT` +- `DATABASE_USER` +- `RELAY_API_ZONE_NAME` +- `RELAY_DOMAIN` +- `RELAY_TUNNEL_ZONE_NAME` + +Required secrets: + +- `CLERK_SECRET_KEY` +- `CLOUDFLARE_API_TOKEN` +- `DATABASE_PASSWORD` + +Keep these optional/legacy groups entirely unset: + +- `AXIOM_ORG_ID` and `AXIOM_TOKEN` +- `APNS_ENVIRONMENT`, `APNS_TEAM_ID`, `APNS_KEY_ID`, `APNS_BUNDLE_ID`, and `APNS_PRIVATE_KEY` +- `PLANETSCALE_ORGANIZATION`, `PLANETSCALE_API_TOKEN_ID`, and `PLANETSCALE_API_TOKEN` + +An incomplete `DATABASE_*` group causes the relay to select its PlanetScale fallback. Treat all +five database settings as one atomic configuration. + +## Cloudflare API token permissions + +Restrict the token to the one Cloudflare account and the `t3turbo.pro` API/managed-endpoint zones. +The resources currently created by `infra/relay` require these permission groups. + +Account permissions: + +- `Workers Scripts Write` +- `Hyperdrive Write` +- `Secrets Store Write` +- `Account API Tokens Write` +- `Cloudflare Tunnel Read` +- `Cloudflare Tunnel Write` + +Zone permissions: + +- `Zone Read` +- `DNS Read` +- `DNS Write` + +Cloudflare's dashboard may display `Edit` where the API permission group is named `Write`. +`Account API Tokens Write` is needed because the current Alchemy deployment mints narrower runtime +tokens for managed tunnel and DNS bindings. Revalidate these permissions against the actual +deployment plan after every upstream rebase. Add `Queues Write` only when APNs is enabled; the +locked Android-only deployment does not create APNs queues. + +Do not use a Global API Key. Set an expiration date, rotate the token, and avoid granting access to +unrelated accounts or zones. + +## Zero Trust Access + +Certificate Transparency and DNS make the production hostnames discoverable. Before production +traffic, create self-hosted Access applications for both `relay.t3turbo.pro` and +`app.t3turbo.pro` and attach deny-by-default policies. + +- Restrict the app to intended operator identities and require enrolled-device posture where + available. +- Restrict the relay to an Access mode every shipped web, CLI, and Android client can satisfy. + Prefer enrolled Cloudflare One/Gateway devices for this operator deployment. Do not embed a + service-token secret in a public client. +- Test an authorized request and a denied request from an unmanaged client for each hostname. +- Configure Access with an operator identity or a separate narrowly scoped token. The relay + deployment token above intentionally does not need `Access: Apps and Policies Write` because the + Alchemy stack does not reconcile Access policy. + +## Zone notes for `t3turbo.pro` + +- The zone must be active before production deploy. Alchemy adopts it as retained state; it does not + configure registrar nameservers. +- `RELAY_API_ZONE_NAME` is `t3turbo.pro`. +- `RELAY_DOMAIN` is `relay.t3turbo.pro`. +- `RELAY_TUNNEL_ZONE_NAME` may be `t3turbo.pro` or a separately activated/delegated child zone. +- `relay.t3turbo.pro` must not have a conflicting CNAME when the Worker custom domain is created. +- `app.t3turbo.pro` is a proxied CNAME created by `cloudflared tunnel route dns` or the Zero Trust + dashboard. +- `db.t3turbo.pro` is DNS-only for the locked public PostgreSQL TLS endpoint. Cloudflare's normal + HTTP proxy does not proxy PostgreSQL. +- Do not create both a Worker custom domain and a tunnel route for `relay.t3turbo.pro`. +- Keep SSL/TLS mode `Full (strict)` for HTTP origins. The PostgreSQL endpoint has separate TLS and + Hyperdrive certificate verification. +- If a separate managed-endpoint zone is used, restrict the deployment token to both zones and + document delegation/renewal ownership. + +## Preflight + +- [ ] `t3turbo.pro` is active in the intended Cloudflare account. +- [ ] `relay.t3turbo.pro`, `app.t3turbo.pro`, and `db.t3turbo.pro` have non-conflicting ownership. +- [ ] Deny-by-default Access applications protect both relay and app hostnames. +- [ ] Intended clients pass the relay Access policy; unmanaged clients are denied. +- [ ] The deployment token is scoped and stored as `CLOUDFLARE_API_TOKEN`. +- [ ] The GitHub `production` environment has required reviewers and branch protection. +- [ ] Public Clerk values are variables; `CLERK_SECRET_KEY` is a secret. +- [ ] All four database variables are set and `DATABASE_PASSWORD` is a secret. +- [ ] No PlanetScale, Axiom, or APNs value has been added for the locked target. +- [ ] Relay schema, seed contract, and RLS have been applied before workflow dispatch. diff --git a/infra/cloudflared/README.md b/infra/cloudflared/README.md new file mode 100644 index 0000000000000..a01beb9ac5e44 --- /dev/null +++ b/infra/cloudflared/README.md @@ -0,0 +1,198 @@ +# Cloudflare Tunnel on Ubuntu + +This guide installs `cloudflared`, exposes the Ubuntu-hosted T3 surface behind Cloudflare Access at +`app.t3turbo.pro`, and documents optional protected SSH/PostgreSQL routes. + +## Hostname ownership + +The locked topology assigns: + +- `relay.t3turbo.pro` to the Cloudflare Worker custom domain; +- `app.t3turbo.pro` to the Ubuntu `cloudflared` tunnel; +- `db.t3turbo.pro` to the DNS-only public TLS endpoint used by Hyperdrive. + +A hostname cannot simultaneously be a Worker custom domain and a tunnel CNAME. Do not run +`cloudflared tunnel route dns` for `relay.t3turbo.pro` while the Worker owns it. The example config +contains a commented relay-origin rule only to document the mutually exclusive fallback if the +architecture later stops using a Worker. + +Authoritative references: + +- [Download `cloudflared`](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/downloads/) +- [Create a locally managed tunnel](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/local-management/create-local-tunnel/) +- [Tunnel configuration files](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/local-management/configuration-file/) +- [Run a tunnel as a Linux service](https://developers.cloudflare.com/tunnel/advanced/local-management/as-a-service/linux/) + +## Install `cloudflared` + +Use Cloudflare's stable APT repository: + +```sh +sudo install -d -m 0755 /usr/share/keyrings +curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg \ + | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null +echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main' \ + | sudo tee /etc/apt/sources.list.d/cloudflared.list >/dev/null +sudo apt-get update +sudo apt-get install cloudflared +cloudflared --version +``` + +Cloudflare supports `cloudflared` releases within one year of the current release. Apply package +updates deliberately; an upgrade restarts the connector and can drop long-lived WebSockets unless +another replica is serving the same tunnel. + +## Choose one authentication model + +### Remotely managed tunnel: token + +Create the tunnel in **Cloudflare Dashboard > Zero Trust > Networks > Tunnels**, define public +hostnames there, and copy the connector token. The token is a runtime secret scoped to that tunnel. +Install the vendor service without placing the token in shell history by storing it in a +root-readable environment file and using a reviewed systemd unit, or use Cloudflare's generated +`service install` command from a private terminal: + +```sh +sudo cloudflared service install +``` + +For the token model, ingress is managed in the dashboard/API. Do not also maintain a local ingress +file as a competing source of truth. + +### Locally managed tunnel: `cert.pem` plus scoped JSON + +`cloudflared tunnel login` opens an authorization flow and writes `cert.pem`. That certificate is +account-wide management authority: it creates/deletes tunnels and routes DNS. It is **not** the +credential a named tunnel uses at runtime. + +```sh +cloudflared tunnel login +cloudflared tunnel create t3turbo-origin +cloudflared tunnel list +``` + +Creation writes `.json`, the scoped runtime credential. Copy only that JSON and the +rendered config to `/etc/cloudflared`, make them readable only by root and the dedicated service +group, and remove `cert.pem` from the production host when tunnel management occurs elsewhere. + +```sh +id -u cloudflared >/dev/null 2>&1 \ + || sudo useradd --system --home /nonexistent --shell /usr/sbin/nologin cloudflared +sudo install -d -o root -g cloudflared -m 0750 /etc/cloudflared +sudo install -o root -g cloudflared -m 0640 \ + "$HOME/.cloudflared/.json" \ + /etc/cloudflared/.json +sudo install -o root -g cloudflared -m 0640 \ + /path/to/t3code/infra/cloudflared/config.yml.example \ + /etc/cloudflared/config.yml +``` + +Replace the example markers in `/etc/cloudflared/config.yml`; never edit the checked-in example +with production values. + +## DNS routing + +For a locally managed tunnel, first create a self-hosted Cloudflare Access application for +`app.t3turbo.pro` with a deny-by-default operator policy. Then route the hosted application: + +```sh +cloudflared tunnel route dns t3turbo-origin app.t3turbo.pro +``` + +This creates a proxied CNAME to the tunnel. Confirm it does not overwrite an existing record. + +Do not route `relay.t3turbo.pro`; the relay Worker deployment creates its custom domain. Do not +route the locked public database endpoint through the normal HTTP tunnel. `db.t3turbo.pro` is +DNS-only and firewalled as described in `infra/supabase/SETUP-RULES.md`. + +## Ingress and Access + +The first matching ingress rule wins, and the final rule must be a catch-all. Validate and test the +rendered file before starting the service: + +```sh +sudo cloudflared tunnel --config /etc/cloudflared/config.yml ingress validate +sudo cloudflared tunnel --config /etc/cloudflared/config.yml ingress rule \ + https://app.t3turbo.pro +``` + +Requirements: + +- Bind local web services to loopback when only the tunnel should reach them. +- Protect `app.t3turbo.pro` with a deny-by-default Access application and test both authorized and + denied clients. The hostname is discoverable after certificate issuance. +- Put Cloudflare Access in front of optional SSH or TCP public hostnames before adding their DNS + routes. +- Raw TCP ingress requires a compatible Cloudflare client path. It is not an unauthenticated public + TCP proxy. +- For Hyperdrive's private-database Tunnel integration, follow Cloudflare's dedicated Hyperdrive + setup rather than treating the optional example TCP rule as sufficient. +- Keep the catch-all `http_status:404` rule. + +## Install as a systemd service + +Cloudflare's package can install its own unit: + +```sh +sudo cloudflared --config /etc/cloudflared/config.yml service install +sudo systemctl enable --now cloudflared +``` + +Alternatively, install the reviewed template: + +```sh +sudo install -o root -g root -m 0644 \ + /path/to/t3code/infra/cloudflared/cloudflared.service.example \ + /etc/systemd/system/cloudflared.service +sudo systemctl daemon-reload +sudo systemctl enable --now cloudflared +``` + +The template is for locally managed JSON credentials. Token-managed installations should use the +vendor-generated unit or a separate root-readable environment-file design. + +## Verify + +```sh +systemctl is-enabled cloudflared +systemctl is-active cloudflared +systemctl status cloudflared --no-pager +journalctl -u cloudflared --since '15 minutes ago' --no-pager +cloudflared access curl https://app.t3turbo.pro/ +``` + +Also confirm: + +- at least four edge connections are reported for one connector; +- the app response comes from the intended local service; +- an unmanaged client is denied by Access before it reaches the app; +- a nonexistent path/hostname does not expose another local service; +- WebSocket traffic survives normal use; +- `relay.t3turbo.pro` still resolves to the Worker custom domain, not the tunnel. + +Run `cloudflared tunnel info t3turbo-origin` from the administrative workstation that retains +`cert.pem`; it is expected to fail on a hardened runtime host where that account credential was +removed. + +## Update and rollback + +For a single connector, package updates briefly interrupt traffic: + +```sh +sudo apt-get update +sudo apt-get install --only-upgrade cloudflared +sudo systemctl restart cloudflared +``` + +For lower disruption, run a second connector replica with the same tunnel credential, update and +verify it, then update the first. + +Rollback order: + +1. Restore the last known-good `/etc/cloudflared/config.yml`. +2. Validate it. +3. Restart one connector and verify application/WebSocket traffic. +4. Roll back the package only when the configuration is known good. + +To withdraw exposure without deleting state, remove the `app.t3turbo.pro` tunnel route or stop the +service. Deleting the tunnel invalidates all replicas and should be reserved for decommissioning. diff --git a/infra/cloudflared/cloudflared.service.example b/infra/cloudflared/cloudflared.service.example new file mode 100644 index 0000000000000..860cc3ab8d02a --- /dev/null +++ b/infra/cloudflared/cloudflared.service.example @@ -0,0 +1,23 @@ +[Unit] +Description=Cloudflare Tunnel for T3 Turbo +Documentation=https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/ +After=network-online.target +Wants=network-online.target + +[Service] +Type=notify +User=cloudflared +Group=cloudflared +ExecStart=/usr/bin/cloudflared --no-autoupdate --config /etc/cloudflared/config.yml tunnel run +Restart=on-failure +RestartSec=5s +TimeoutStartSec=0 +LimitNOFILE=1048576 +NoNewPrivileges=true +PrivateTmp=true +ProtectHome=true +ProtectSystem=strict +ReadOnlyPaths=/etc/cloudflared + +[Install] +WantedBy=multi-user.target diff --git a/infra/cloudflared/config.yml.example b/infra/cloudflared/config.yml.example new file mode 100644 index 0000000000000..53c10e5a560d6 --- /dev/null +++ b/infra/cloudflared/config.yml.example @@ -0,0 +1,34 @@ +# Locally managed tunnel example. Copy to /etc/cloudflared/config.yml, replace all markers there, +# and keep the rendered file and credential JSON out of Git. +tunnel: +credentials-file: /etc/cloudflared/.json + +# QUIC is preferred; cloudflared falls back according to its supported transport behavior. +protocol: quic + +ingress: + # Active locked-stack application route. Protect the hostname with Cloudflare Access before + # publishing its DNS route, and keep the T3 server on loopback. + - hostname: app.t3turbo.pro + service: http://127.0.0.1: + originRequest: + connectTimeout: 10s + noHappyEyeballs: false + + # MUTUALLY EXCLUSIVE FALLBACK ONLY. Do not enable while relay.t3turbo.pro is a Worker custom + # domain. This rule is only for a future architecture with a local relay HTTP origin. + # - hostname: relay.t3turbo.pro + # service: http://127.0.0.1: + + # Optional SSH route. Create a Cloudflare Access application/policy before adding its DNS route. + # Clients must use a supported Access/cloudflared SSH flow. + # - hostname: ssh.t3turbo.pro + # service: ssh://127.0.0.1:22 + + # Optional private PostgreSQL route. This is not the locked public-TLS path. For Hyperdrive, + # configure the documented private-database Tunnel integration and Access restrictions first. + # - hostname: db-private.t3turbo.pro + # service: tcp://127.0.0.1:5432 + + # Required catch-all: never fall through to an unintended local service. + - service: http_status:404 diff --git a/infra/relay/DEPLOY.md b/infra/relay/DEPLOY.md new file mode 100644 index 0000000000000..1854161e8f6fb --- /dev/null +++ b/infra/relay/DEPLOY.md @@ -0,0 +1,275 @@ +# Deploy the T3 Turbo relay + +This runbook targets `https://relay.t3turbo.pro`, Clerk authentication, and a Hyperdrive connection +to self-hosted Supabase PostgreSQL 17. + +## Locked deployment mode + +The current Alchemy stack supports this deployment without source changes: + +- A complete `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, and + `DATABASE_PASSWORD` set selects external PostgreSQL and provisions Hyperdrive for it. +- An absent/incomplete `AXIOM_ORG_ID` and `AXIOM_TOKEN` pair disables Axiom provisioning and trace + export. +- An absent/incomplete APNs set disables APNs resources and uses the runtime's disabled delivery + layer. + +Set the database group completely. Leave all PlanetScale, Axiom, and APNs inputs unset. If the +database group is incomplete, configuration falls back to the legacy PlanetScale path and deploy +fails without PlanetScale credentials. + +## Prerequisites + +- Active Cloudflare zone `t3turbo.pro`. +- `relay.t3turbo.pro` free of conflicting DNS/tunnel records. +- Deny-by-default Cloudflare Access application and policy for `relay.t3turbo.pro`, with an + authentication path already proven on every intended client surface. +- Cloudflare account features used by the locked relay: Workers, Hyperdrive, Secrets Store/Alchemy + state, DNS, and Cloudflare Tunnel. Queues are not created when APNs is disabled. +- A least-privilege `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID`. +- Healthy self-hosted Supabase PostgreSQL 17 with `schema.sql`, `seed.sql`, and `rls.sql` applied. +- Dedicated `relay_runtime` database role with `service_role` membership. +- A TLS database endpoint reachable by Hyperdrive and limited to Cloudflare IP ranges, or an + approved private-database Tunnel/VPC path. +- Clerk production publishable/secret keys, JWT template `t3-relay`, audience + `t3-code-relay`, and a public CLI OAuth application. +- Vite+ installed through the repository's normal setup and relay dependencies installed. + +## Hyperdrive against self-hosted Supabase + +### Locked option: public TLS endpoint + +1. Enable native PostgreSQL SSL and expose the reviewed direct listener at + `db.t3turbo.pro:15432` with a WebPKI certificate valid for that hostname. +2. Do not stack Hyperdrive on Supavisor transaction mode. Hyperdrive already pools connections, and + a direct PostgreSQL origin preserves relay transaction semantics without double pooling. +3. Firewall the TLS port to the current Cloudflare IPv4/IPv6 ranges and operator addresses only. +4. Use database `postgres` and username `relay_runtime`. +5. Put the host, port, database, and user in GitHub variables and the password in the + `DATABASE_PASSWORD` GitHub secret. Do not construct or print a connection URL. +6. Allow Alchemy to create `RelayHyperdrive`. The current source sets `sslmode=require`, disables + query caching, and limits the origin to 20 connections. +7. Tune only after measuring PostgreSQL and Hyperdrive; do not change pooling limits during first + bring-up. + +Cloudflare documentation: + +- [Connect Hyperdrive to PostgreSQL](https://developers.cloudflare.com/hyperdrive/examples/connect-to-postgres/) +- [Hyperdrive firewall and networking](https://developers.cloudflare.com/hyperdrive/configuration/firewall-and-networking-configuration/) +- [Hyperdrive TLS certificates](https://developers.cloudflare.com/hyperdrive/configuration/tls-ssl-certificates-for-hyperdrive/) + +Security caveat: Cloudflare's published source ranges are shared. The firewall is only one layer; +retain password authentication, a dedicated role, TLS verification, RLS, and rotation. + +### Private alternatives + +- [Hyperdrive private-database integration over Cloudflare Tunnel and Access](https://developers.cloudflare.com/hyperdrive/configuration/connect-to-private-database/). +- Workers VPC where available. + +These remove the public database listener and are preferable if the locked public-endpoint decision +changes. A normal `tcp://` cloudflared ingress entry alone is not the full Hyperdrive private +database integration. + +### Verify Hyperdrive + +After deployment, inspect the created Hyperdrive configuration, issue the Worker health request, +and query: + +```sql +SELECT DISTINCT usename, application_name +FROM pg_stat_activity +WHERE application_name = 'Cloudflare Hyperdrive'; +``` + +The role must be the dedicated relay identity. A `postgres` Hyperdrive session is a configuration +failure. + +## Clerk wiring + +Use one Clerk production instance for released clients and the relay. + +1. In **JWT templates**, create: + + | Setting | Value | + | ------- | ---------------------------- | + | Name | `t3-relay` | + | Claims | `{ "aud": "t3-code-relay" }` | + +2. Set the relay's `CLERK_JWT_AUDIENCE` to `t3-code-relay`. +3. Set the clients' `CLERK_JWT_TEMPLATE`/`T3CODE_CLERK_JWT_TEMPLATE` to `t3-relay`. +4. Store `CLERK_PUBLISHABLE_KEY` as public configuration and `CLERK_SECRET_KEY` only as a Worker or + GitHub environment secret. +5. Create a public Clerk OAuth application for the CLI with PKCE, `openid`, `profile`, and `email`. +6. Allow: + - `http://127.0.0.1:34338/callback` + - `https://app.t3turbo.pro/connect/callback` +7. Put its public client ID in `CLERK_CLI_OAUTH_CLIENT_ID`. +8. Configure the production Android/native application and restrict origins/sign-ups to intended + users. + +The relay first verifies Clerk template/session tokens and then accepts Clerk OAuth tokens for the +headless CLI. The CLI stores no client secret. Cloudflare Access is the outer network gate and +Clerk remains the relay's application identity layer. Do not enable Access Managed OAuth on this +hostname because the relay publishes its own OAuth discovery and authorization flow. + +## Workflow usage + +`.github/workflows/deploy-relay.yml` has two triggers: + +- push to `main`; +- manual `workflow_dispatch`. + +This infrastructure branch does not auto-deploy because it is not `main`. After the target +variables/secrets are configured and the SQL is applied, a maintainer can dispatch it explicitly: + +```sh +gh workflow run deploy-relay.yml --ref infra/t3turbo-relay +gh run list --workflow deploy-relay.yml --limit 5 +``` + +Watch the selected run without printing environment values: + +```sh +gh run watch --exit-status +``` + +The workflow deploys Alchemy stage `prod` with `--yes`. Before the first apply, review a local +deployment plan with the same source and no secret output. The production plan must create the +Worker and external-database Hyperdrive, must not create PlanetScale or Axiom resources, and must +not create APNs queues. + +Required public configuration for the target: + +- `CLOUDFLARE_ACCOUNT_ID` +- `RELAY_API_ZONE_NAME=t3turbo.pro` +- `RELAY_DOMAIN=relay.t3turbo.pro` +- `RELAY_TUNNEL_ZONE_NAME` +- `CLERK_PUBLISHABLE_KEY` +- `CLERK_JWT_AUDIENCE=t3-code-relay` +- `CLERK_JWT_TEMPLATE=t3-relay` +- `CLERK_CLI_OAUTH_CLIENT_ID` +- `DATABASE_HOST=db.t3turbo.pro` +- `DATABASE_PORT=15432` +- `DATABASE_NAME=postgres` +- `DATABASE_USER=relay_runtime` + +Required target secrets: + +- `CLOUDFLARE_API_TOKEN` +- `CLERK_SECRET_KEY` +- `DATABASE_PASSWORD` + +Leave `PLANETSCALE_*`, `AXIOM_*`, and `APNS_*` unset. The workflow's report step must emit notices +that hosted tracing and mobile push are disabled. `DATABASE_PASSWORD` is consumed during Hyperdrive +reconciliation; Actions and Alchemy output must never print it. + +## Post-deploy verification + +### Public readiness and documentation + +```sh +cloudflared access curl https://relay.t3turbo.pro/health +cloudflared access curl https://relay.t3turbo.pro/openapi.json >/dev/null +cloudflared access curl https://relay.t3turbo.pro/docs >/dev/null +``` + +Expected health response: + +```json +{ "ok": true, "service": "relay" } +``` + +`/health` executes `SELECT 1`; it proves both Worker readiness and database connectivity. + +### OAuth/DPoP discovery + +```sh +cloudflared access curl \ + https://relay.t3turbo.pro/.well-known/oauth-authorization-server +cloudflared access curl \ + https://relay.t3turbo.pro/.well-known/oauth-protected-resource +``` + +Confirm the issuer/resource is `https://relay.t3turbo.pro`, the token endpoint is under that origin, +and DPoP ES256 support is advertised. + +### End-to-end + +1. Build a client with the four public T3 Connect values. +2. From an enrolled Android device, pass the Cloudflare Access policy and sign in through Clerk. +3. Link an environment with `t3 connect link`. +4. Run `t3 connect status --json` and confirm the environment is linked and its endpoint is ready. +5. Connect from Android and exercise API and WebSocket traffic. +6. Confirm normal environment traffic reaches the managed environment hostname directly; it must + not transit the relay Worker. +7. Confirm the database contains the expected link/allocation rows without exposing tokens or + credential hashes in logs. +8. Confirm an unmanaged device is denied by Access before any relay response is returned. + +## Troubleshooting + +### Workflow says deployment was skipped + +Both `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` must be non-empty. Check repository versus +`production` environment scope and environment protection. Do not print the secret. + +### Deployment requests PlanetScale + +At least one required `DATABASE_*` value is absent or empty, so the stack selected its legacy +fallback. Stop and correct the complete external database group; do not supply PlanetScale +credentials for the locked deployment. + +### Axiom or APNs resources appear in the plan + +One or more excluded optional values are still defined at repository or `production` environment +scope. Remove the complete optional group, rerun the plan, and confirm the report says the service +is disabled. + +### Worker custom domain fails + +Remove conflicting `relay.t3turbo.pro` A/AAAA/CNAME or tunnel routes, confirm the zone is active in +the selected account, and verify Zone/DNS/Workers token permissions. Do not delete unrelated zone +records. + +### `/health` returns `database_unavailable` + +Check, in order: + +1. Supabase container health and `pg_isready`. +2. database TLS certificate/SAN and expiry. +3. Cloudflare IP allowlist completeness for IPv4 and IPv6. +4. Hyperdrive hostname, port, database, TLS mode, and credential generation. +5. `relay_runtime` login, `service_role` membership, grants, and connection limits. +6. whether the direct listener accepts `relay_runtime` and requires SSL. + +### Clerk requests return 401 + +Confirm the client used template `t3-relay`, `aud` is exactly `t3-code-relay`, the Worker uses the +matching production publishable/secret pair, the issuer is the production instance, clocks are +synchronized, and the token is unexpired. Never paste a bearer token into an issue or workflow log. +An Access redirect or `403` occurs before the Worker and must be diagnosed in Zero Trust policy; +it is not a Clerk `401`. + +### Managed endpoint provisioning fails + +Confirm `RELAY_TUNNEL_ZONE_NAME`, Cloudflare Tunnel/DNS runtime permissions, zone ownership, tunnel +quota, and the per-user default limit of three. `seed.sql` correctly contains no limit row; +`relay_managed_tunnel_limits` stores overrides only. + +### App works but release artifacts do not show Connect + +`release.yml` statically injects public configuration at build time. Confirm +`CLERK_PUBLISHABLE_KEY`, `CLERK_JWT_TEMPLATE`, `CLERK_CLI_OAUTH_CLIENT_ID`, relay domain variables, +and the Cloudflare enablement pair were present in the selected GitHub scope before the build. + +## Rollback + +1. Redeploy the last known-good Worker version or detach `relay.t3turbo.pro` to stop new requests. +2. Keep Hyperdrive and PostgreSQL intact while in-flight sessions drain. +3. Revert only backward-compatible code first. +4. Restore the database only when a tested backup exists and the last-good Worker cannot read the + current schema. +5. Revoke newly introduced credentials after the rollback is verified. + +Never use `alchemy destroy`, delete the Cloudflare zone, reset Supabase, or drop relay tables as an +ordinary application rollback. diff --git a/infra/relay/scripts/deploy.ts b/infra/relay/scripts/deploy.ts index 0d711a775f2cf..648b1b4d5c680 100644 --- a/infra/relay/scripts/deploy.ts +++ b/infra/relay/scripts/deploy.ts @@ -410,7 +410,13 @@ const runRelayDeploy = Effect.fn("relay.deploy.run")( Layer.mergeAll( Layer.effect( AlchemyContext, - AlchemyContext.pipe(Effect.map((context) => ({ ...context, adopt: options.adopt }))), + AlchemyContext.pipe( + Effect.map((context) => ({ + ...context, + adopt: options.adopt, + updateStateStore: options.yes, + })), + ), ), Layer.succeed(AdoptPolicy, options.adopt), Layer.succeed(AuthProviders, {}), diff --git a/infra/supabase/README.md b/infra/supabase/README.md new file mode 100644 index 0000000000000..f6e7250cb52a0 --- /dev/null +++ b/infra/supabase/README.md @@ -0,0 +1,436 @@ +# Self-hosted Supabase Docker for the relay + +This guide installs the official Supabase Docker stack on Ubuntu with PostgreSQL 17, persistent +host storage, and the relay schema. It deliberately keeps secrets and mutable runtime state outside +this repository. + +Authoritative upstream references: + +- [Self-hosting with Docker](https://supabase.com/docs/guides/self-hosting/docker) +- [PostgreSQL 17 self-hosting](https://supabase.com/docs/guides/self-hosting/postgres-upgrade-17) +- [Self-hosted authentication keys](https://supabase.com/docs/guides/self-hosting/self-hosted-auth-keys) + +## Host layout and prerequisites + +Use a dedicated persistent filesystem: + +```text +/srv/t3turbo/supabase/ + .env # mode 0600; never Git-tracked + docker-compose.yml + run.sh + utils/ + volumes/ + db/data/ # PostgreSQL data bind mount + storage/ # Storage file backend + functions/ # Edge Functions, if retained +``` + +The official compose file also creates a Docker named volume named `db-config`. It contains the +pgsodium root key and must be included in recovery planning; backing up `volumes/db/data` alone is +not sufficient for deployments that store Vault secrets. + +Minimum production starting point: 4 CPU cores, 8 GB RAM, and 80 GB of SSD storage. Install Docker +Engine, Docker Compose v2, Git, OpenSSL, `curl`, and PostgreSQL 17 client tools. + +## Clone and pin the Docker configuration + +Never run production from an ephemeral application checkout. Clone upstream into a staging +directory, review an immutable commit, and copy only the Docker deployment into `/srv`: + +```sh +git clone --filter=blob:none --no-checkout https://github.com/supabase/supabase.git \ + /var/tmp/supabase-source +cd /var/tmp/supabase-source +git sparse-checkout init --cone +git sparse-checkout set docker +git checkout +reviewed_supabase_commit=$(git rev-parse HEAD) +sudo install -d -m 0750 /srv/t3turbo/supabase +sudo cp -a docker/. /srv/t3turbo/supabase/ +cd /srv/t3turbo/supabase +sudo cp .env.example .env +sudo chmod 0600 .env +printf '%s\n' "$reviewed_supabase_commit" | sudo tee SUPABASE_UPSTREAM_COMMIT >/dev/null +unset reviewed_supabase_commit +``` + +`SUPABASE_UPSTREAM_COMMIT` is operational metadata, not a secret. Review image changes and the +upstream self-hosting changelog before changing it. Fresh current Supabase Docker installs use +PostgreSQL 17 by default; do not add the legacy PostgreSQL 15 compose override. + +## Generate `.env` safely + +Set a restrictive umask and run the official generator from a private terminal. It generates the +legacy HS256 keys and the other required random values and updates `.env`: + +```sh +cd /srv/t3turbo/supabase +umask 077 +sudo sh utils/generate-keys.sh --update-env +sudo sh utils/add-new-auth-keys.sh --update-env +sudo chmod 0600 .env +sudo rm -f .env.old docker-compose.yml.old +``` + +The generator prints generated values. Do not run it in CI, a recorded terminal, a shared shell, +or with `tee`. Move the values into the password manager without copying them into notes or Git. + +Before first start, replace every insecure default in `.env`. At minimum review: + +- `POSTGRES_PASSWORD` +- `JWT_SECRET` +- `ANON_KEY` +- `SERVICE_ROLE_KEY` +- `SUPABASE_PUBLISHABLE_KEY` +- `SUPABASE_SECRET_KEY` +- `JWT_KEYS` and `JWT_JWKS` +- `DASHBOARD_USERNAME` and `DASHBOARD_PASSWORD` +- `SECRET_KEY_BASE` +- `REALTIME_DB_ENC_KEY` +- `VAULT_ENC_KEY` +- `PG_META_CRYPTO_KEY` +- `LOGFLARE_PUBLIC_ACCESS_TOKEN` and `LOGFLARE_PRIVATE_ACCESS_TOKEN` +- `S3_PROTOCOL_ACCESS_KEY_ID` and `S3_PROTOCOL_ACCESS_KEY_SECRET` +- `POOLER_TENANT_ID` + +The locked stack does not enable the optional `docker-compose.logs.yml` override. Its Logflare +tokens are still generated so an insecure default is never left in `.env`; they are unrelated to +Axiom. + +### `VAULT_ENC_KEY` is exactly 32 bytes + +Supavisor requires `VAULT_ENC_KEY` to be exactly 32 characters. Generate 16 random bytes encoded as +32 lowercase hexadecimal ASCII bytes: + +```sh +openssl rand -hex 16 +``` + +After placing it in `.env`, validate without printing it: + +```sh +set -a +. /srv/t3turbo/supabase/.env +set +a +test "$(LC_ALL=C printf %s "$VAULT_ENC_KEY" | wc -c)" -eq 32 +unset VAULT_ENC_KEY +``` + +A zero exit code is required. Do not use a 32-byte random input encoded as hex; that produces 64 +characters and is invalid for this setting. + +### HS256 `ANON_KEY` and `SERVICE_ROLE_KEY` + +`ANON_KEY` and `SERVICE_ROLE_KEY` are JWTs, not random strings. Both must: + +- use header `{"alg":"HS256","typ":"JWT"}`; +- be signed with the same `JWT_SECRET` configured for Supabase services; +- contain `role: "anon"` or `role: "service_role"` respectively; +- contain valid `iss`, `iat`, and `exp` claims. + +`utils/generate-keys.sh` performs this generation with OpenSSL and writes all three matching values. +Do not generate the API JWTs independently and do not paste sample JWTs from documentation. To +rotate `JWT_SECRET`, generate a new matching trio, update them together during a maintenance +window, recreate affected containers, and invalidate the old tokens. + +The newer asymmetric keys created by `utils/add-new-auth-keys.sh` supplement the legacy HS256 keys; +they do not make mismatched `ANON_KEY` or `SERVICE_ROLE_KEY` values safe. + +## URLs and ports + +The relay uses PostgreSQL directly and Clerk for authentication; it does not need the Supabase HTTP +APIs on a public hostname. Keep Kong/Studio private and use coherent loopback URLs: + +```dotenv +SUPABASE_PUBLIC_URL=http://127.0.0.1:8000 +API_EXTERNAL_URL=http://127.0.0.1:8000/auth/v1 +SITE_URL=https://app.t3turbo.pro +``` + +Access Studio through an SSH port forward to loopback. If Supabase HTTP APIs are later required, +give them a separate protected hostname behind Cloudflare Access; do not reuse +`app.t3turbo.pro`, which belongs to the T3 server, and never publish Studio unauthenticated. + +Port `8443` conflicts with an existing host service in this deployment. Remap only the host side by +setting: + +```dotenv +KONG_HTTPS_PORT=18443 +``` + +The compose mapping remains `${KONG_HTTPS_PORT}:8443`, so containers still use Kong's internal +port `8443`. Leave `POSTGRES_PORT=5432` and `POOLER_PROXY_PORT_TRANSACTION=6543` unless a reviewed +network design changes both the compose mapping and every connection string. + +Restrict host firewall access before starting. Kong, Studio, Supavisor, and PostgreSQL must not be +world-accessible by default. See [SETUP-RULES.md](./SETUP-RULES.md) for the database TLS boundary. + +### Pinned gateway compatibility + +This runbook's compose override assumes the pinned Supabase revision uses the `kong` service and +`KONG_HTTP_PORT`/`KONG_HTTPS_PORT`. Supabase has announced that Envoy becomes the default API +gateway during the week of 2026-08-09. Keep the deployment on a reviewed Kong-based commit until an +Envoy override has been tested. Before every repin, inspect `docker compose config --services` and +the rendered port bindings; if the gateway service or variables changed, stop and adapt this +override instead of applying the `kong` block blindly. + +## Native PostgreSQL TLS for Hyperdrive + +Hyperdrive already pools database connections, so the locked path connects it directly to +PostgreSQL rather than stacking it on Supavisor. PostgreSQL must perform its own protocol-aware TLS +negotiation; a generic HTTPS/TCP TLS terminator is not a substitute for PostgreSQL SSL. + +Use the DNS-only hostname `db.t3turbo.pro` and public port `15432`. Obtain a WebPKI certificate by +DNS-01 so no database HTTP service must be exposed. With a dedicated Cloudflare token restricted to +Zone Read and DNS Write for `t3turbo.pro`: + +```sh +sudo apt-get install certbot python3-certbot-dns-cloudflare +sudo install -d -m 0700 /root/.secrets/certbot +sudo install -m 0600 /dev/null /root/.secrets/certbot/cloudflare.ini +sudoedit /root/.secrets/certbot/cloudflare.ini +sudo certbot certonly \ + --dns-cloudflare \ + --dns-cloudflare-credentials /root/.secrets/certbot/cloudflare.ini \ + --dns-cloudflare-propagation-seconds 30 \ + --non-interactive --agree-tos \ + --email \ + -d db.t3turbo.pro +``` + +The credential file contains only: + +```ini +dns_cloudflare_api_token = +``` + +Copy the issued material into the persistent `db-config` volume, with the private key readable only +by PostgreSQL: + +```sh +cd /srv/t3turbo/supabase +sudo docker compose exec db install -d -o postgres -g postgres -m 0700 \ + /etc/postgresql-custom/tls +sudo docker compose cp -L /etc/letsencrypt/live/db.t3turbo.pro/fullchain.pem \ + db:/etc/postgresql-custom/tls/server.crt.next +sudo docker compose cp -L /etc/letsencrypt/live/db.t3turbo.pro/privkey.pem \ + db:/etc/postgresql-custom/tls/server.key.next +sudo docker compose exec db install -o postgres -g postgres -m 0644 \ + /etc/postgresql-custom/tls/server.crt.next \ + /etc/postgresql-custom/tls/server.crt +sudo docker compose exec db install -o postgres -g postgres -m 0600 \ + /etc/postgresql-custom/tls/server.key.next \ + /etc/postgresql-custom/tls/server.key +sudo docker compose exec db rm -f \ + /etc/postgresql-custom/tls/server.crt.next \ + /etc/postgresql-custom/tls/server.key.next +``` + +Create `/srv/t3turbo/supabase/docker-compose.t3turbo.yml` as an operator-owned, reviewed override: + +```yaml +services: + kong: + ports: !override + - "127.0.0.1:8000:8000" + - "127.0.0.1:18443:8443" + + supavisor: + ports: !override + - "127.0.0.1:5432:5432" + - "127.0.0.1:6543:6543" + + db: + ports: + - "15432:5432" + command: + - postgres + - -c + - config_file=/etc/postgresql/postgresql.conf + - -c + - log_min_messages=fatal + - -c + - ssl=on + - -c + - ssl_cert_file=/etc/postgresql-custom/tls/server.crt + - -c + - ssl_key_file=/etc/postgresql-custom/tls/server.key +``` + +Set `COMPOSE_FILE=docker-compose.yml:docker-compose.t3turbo.yml` in the server's `.env`, then +reconcile and verify: + +```sh +sudo docker compose up -d +sudo docker compose exec db psql -U postgres -d postgres -Atc 'show ssl' +openssl s_client -starttls postgres -connect 127.0.0.1:15432 \ + -servername db.t3turbo.pro -verify_hostname db.t3turbo.pro ` and open +`http://127.0.0.1:8000` locally. + +Map the direct endpoint to the relay deployment as one atomic group: + +| GitHub setting | Locked value source | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| `DATABASE_HOST` | `db.t3turbo.pro` | +| `DATABASE_PORT` | `15432` | +| `DATABASE_NAME` | `postgres` | +| `DATABASE_USER` | `relay_runtime` | +| `DATABASE_PASSWORD` | The interactive `relay_runtime` password created in [Install relay tables and policies](#install-relay-tables-and-policies); store as a GitHub secret | + +The first four are GitHub variables. `DATABASE_PASSWORD` is a GitHub `production` environment +secret. Never store a rendered PostgreSQL URL. The relay uses the complete group to create +Hyperdrive; if any member is empty it selects the legacy PlanetScale fallback instead. + +Automate certificate renewal with a deploy hook that repeats the two copies/installs above and sends +PostgreSQL a configuration reload. Test the hook with `certbot renew --dry-run`. Keep the hook and +its logs free of token and private-key content. + +## Persistence + +The official compose deployment uses host bind mounts below `./volumes`; placing the compose +directory at `/srv/t3turbo/supabase` makes those paths durable. Confirm mount sources before first +write: + +```sh +cd /srv/t3turbo/supabase +docker compose config +docker compose config --volumes +``` + +Required durable material includes: + +- `volumes/db/data` +- `volumes/storage` +- `volumes/functions` if functions are used +- `.env` +- local compose overrides and TLS configuration +- the Docker `db-config` named volume +- the ACME account/configuration and a recovery copy of the database TLS private key + +Never place `volumes/` on `/tmp`, an ephemeral cloud disk, or the application Git worktree. Never +symlink production data into a developer checkout. + +## Start and verify PostgreSQL 17 + +```sh +cd /srv/t3turbo/supabase +sudo docker compose pull +sudo docker compose up -d +sudo docker compose ps +sudo docker compose exec db pg_isready -U postgres +sudo docker compose exec db psql -U postgres -d postgres -Atc 'show server_version' +``` + +Every required service must be healthy and the server version must begin with `17`. If a service is +unhealthy, inspect only its logs first: + +```sh +sudo docker compose logs --tail=200 +``` + +Do not repeatedly recreate the whole stack; that obscures the first failure. + +## Install relay tables and policies + +Apply files in this order from a trusted checkout: + +```sh +cd /srv/t3turbo/supabase +sudo docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/schema.sql +sudo docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/seed.sql +sudo docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \ + < /path/to/t3code/infra/supabase/rls.sql +``` + +`schema.sql` is the fresh-database projection of the current Drizzle schema and migrations. +`seed.sql` is intentionally empty because the relay has no required configuration rows. `rls.sql` +enables RLS on every relay table, denies `anon` and `authenticated`, and grants `service_role` full +table access. + +Create the Worker login interactively: + +```sh +sudo docker compose exec db psql -U postgres -d postgres +``` + +```sql +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'relay_runtime') THEN + CREATE ROLE relay_runtime LOGIN; + END IF; +END +$$; +\password relay_runtime +GRANT service_role TO relay_runtime; +``` + +Use the `relay_runtime.` username when connecting through Supavisor session mode. +Never give Hyperdrive the `postgres` password. + +## Backup and restore + +Self-hosting makes the operator responsible for recovery. Use both logical and encrypted +configuration backups: + +1. Run a daily custom-format logical backup with PostgreSQL 17 `pg_dump`. +2. Encrypt it before sending it to storage on a different host/account. +3. Back up `.env`, reviewed compose overrides, TLS certificates, and the `db-config` named volume + separately; encrypt them with a key not stored beside the backup. +4. Back up Storage objects separately. A PostgreSQL dump contains Storage metadata, not the files. +5. Retain daily, weekly, and monthly generations according to the recovery objective. +6. Restore into an isolated PostgreSQL 17 environment on a schedule and run the relay table/RLS and + `/health` checks. A backup that has not been restored is unverified. + +Example logical backup, with the destination already on encrypted storage: + +```sh +sudo install -d -m 0700 /srv/t3turbo/backups/staging +sudo docker compose exec -T db pg_dump -U postgres -d postgres \ + --format=custom --no-owner --no-privileges \ + > /srv/t3turbo/backups/staging/relay-$(date -u +%Y%m%dT%H%M%SZ).dump +``` + +Do not leave the staging dump unencrypted. Do not commit dumps. Coordinate `pg_dump`/`pg_restore` +client major versions with PostgreSQL 17. + +For a physical snapshot, stop writes or use storage-level snapshot semantics proven consistent for +PostgreSQL. Copying a live `volumes/db/data` directory is not a valid backup. Preserve the +`db-config` volume because loss of its pgsodium root key can make Vault data unrecoverable. + +## Safe lifecycle operations + +Safe routine stop/start: + +```sh +sudo docker compose stop +sudo docker compose start +``` + +Safe container reconciliation after an environment change: + +```sh +sudo docker compose up -d --remove-orphans +``` + +Destructive and prohibited during routine operations: + +- `reset.sh` +- `docker compose down -v` +- deletion of `volumes/db/data`, `volumes/storage`, or the `db-config` volume +- replacing `.env` with `.env.example` diff --git a/infra/supabase/SETUP-RULES.md b/infra/supabase/SETUP-RULES.md new file mode 100644 index 0000000000000..2faaf4b940359 --- /dev/null +++ b/infra/supabase/SETUP-RULES.md @@ -0,0 +1,180 @@ +# Supabase operational rules + +These are invariants for the relay database. They apply during first install, upgrades, incident +response, and credential rotation. + +## Secret handling + +- Never commit `.env`, connection strings, database dumps, TLS keys, tunnel credentials, generated + JWTs, or command output containing them. +- Keep `/srv/t3turbo/supabase/.env` owned by the operator account or root with mode `0600`. +- Use an interactive `\password` prompt for PostgreSQL roles. Do not put passwords in shell history, + process arguments, GitHub variables, or SQL files. +- Store `DATABASE_PASSWORD` in the GitHub `production` environment and the password manager. + Alchemy uses it to reconcile Cloudflare's encrypted Hyperdrive configuration; the Worker receives + a Hyperdrive binding, not the origin password. +- `ANON_KEY` is designed for client use, but it remains a rotatable credential and must not be used + to authenticate the relay. `SERVICE_ROLE_KEY`, `JWT_SECRET`, `JWT_KEYS`, and database passwords + are secrets. +- Keep backup-encryption keys in separate custody from backup objects. +- Treat `/etc/cloudflared/cert.pem` as an account-wide management credential. A running named tunnel + uses its scoped JSON credential instead. + +## Database identities + +- Administrative work uses `postgres` only from the Ubuntu host or a protected operator network. +- Hyperdrive uses `relay_runtime`, a dedicated LOGIN role with a unique password and membership in + `service_role`. +- Do not use the Supabase `postgres` password in Hyperdrive. +- `anon` and `authenticated` have no privileges on relay tables and have restrictive deny policies. +- RLS must remain enabled on all nine `relay_*` tables after schema changes. + +## Hyperdrive network boundary + +The locked network path is a **public TLS PostgreSQL endpoint restricted to Cloudflare IP ranges**. +Cloudflare's current networking guidance states that Hyperdrive connects from the published +Cloudflare IP ranges: + +- [Hyperdrive firewall and networking](https://developers.cloudflare.com/hyperdrive/configuration/firewall-and-networking-configuration/) +- [Cloudflare IP ranges](https://www.cloudflare.com/ips/) +- [Hyperdrive TLS certificates](https://developers.cloudflare.com/hyperdrive/configuration/tls-ssl-certificates-for-hyperdrive/) + +Apply all of the following: + +1. Publish a DNS-only record such as `db.t3turbo.pro` for the Ubuntu server. The normal Cloudflare + HTTP proxy does not proxy PostgreSQL. +2. Enable native PostgreSQL SSL and publish the reviewed direct listener on port `15432` with a + WebPKI certificate whose SAN includes `db.t3turbo.pro`. Hyperdrive already pools connections, so + do not stack it on Supavisor transaction mode. A generic TLS terminator that does not understand + PostgreSQL's SSL negotiation is not sufficient. +3. The current relay creates Hyperdrive with `sslmode=require`. Cloudflare documents this as + encrypted connectivity with WebPKI certificate validation. A future move to an uploaded CA and + `verify-full` requires a reviewed source/configuration change. +4. Permit the database TLS port only from the complete, current Cloudflare IPv4 and IPv6 ranges and + explicitly approved operator addresses. Deny all other sources. +5. Automate a daily comparison with Cloudflare's published range lists. Add new ranges before + removing old ones, test Hyperdrive, and alert on drift. +6. Rate-limit and log rejected connections without logging credentials or complete connection + strings. +7. Keep Supavisor ports `5432` and `6543` blocked from the public Internet. Docker-published ports + can bypass simple UFW rules, so enforce the allowlist at the provider firewall and in the + `DOCKER-USER`/equivalent nftables path. + +Cloudflare IP ranges are shared with other Cloudflare products and customers. An IP allowlist is +not client identity. Retain PostgreSQL password authentication, least-privilege roles, TLS hostname +verification, and credential rotation. + +Cloudflare also supports Hyperdrive-to-private-database connectivity through Cloudflare Tunnel and +Workers VPC. Those options remove the public listener and are preferable if the locked network +decision changes: + +- [Connect Hyperdrive to a private database](https://developers.cloudflare.com/hyperdrive/configuration/connect-to-private-database/) + +## Health checks + +Run locally after every change: + +```sh +cd /srv/t3turbo/supabase +sudo docker compose ps +sudo docker compose exec db pg_isready -U postgres +sudo docker compose exec db psql -U postgres -d postgres -Atc 'select 1' +sudo docker compose exec db psql -U postgres -d postgres -Atc \ + "select count(*) from pg_tables where schemaname = 'public' and tablename like 'relay_%'" +sudo docker compose exec db psql -U postgres -d postgres -Atc \ + "select count(*) from pg_tables where schemaname = 'public' and tablename like 'relay_%' and rowsecurity" +``` + +Both table counts must be `9`. + +After a relay request, confirm Hyperdrive reached PostgreSQL: + +```sql +SELECT DISTINCT usename, application_name +FROM pg_stat_activity +WHERE application_name = 'Cloudflare Hyperdrive'; +``` + +External readiness, once the compatible Worker exists: + +```sh +curl --fail --show-error --silent https://relay.t3turbo.pro/health +``` + +Expected response: + +```json +{ "ok": true, "service": "relay" } +``` + +A Worker response with `database_unavailable` is a failed database/Hyperdrive health check even if +the HTTP edge itself is reachable. + +## Rotation procedures + +### Relay database password + +Use an overlap rotation rather than changing the only active login in place: + +1. Create `relay_runtime_next`, set its password interactively, and grant `service_role`. +2. Replace the four `DATABASE_*` variables as needed and the `DATABASE_PASSWORD` secret, then + redeploy so Alchemy updates Hyperdrive. +3. Call `/health` and exercise an authenticated read/write path. +4. Terminate old `relay_runtime` sessions only after the new path is proven. +5. Revoke login from the old role, observe one full rollback window, then drop it. +6. Rename roles only if tooling requires it; stable role generations are easier to audit. + +### Supabase `JWT_SECRET` + +1. Schedule a maintenance window; the legacy HS256 services share one active secret. +2. Run the reviewed `utils/generate-keys.sh` to produce a matching `JWT_SECRET`, `ANON_KEY`, and + `SERVICE_ROLE_KEY` trio. +3. Update `.env` atomically, recreate affected containers, and verify Auth/PostgREST/Storage. +4. Revoke and remove the old trio from every consumer and the password manager history according to + policy. + +Never change only one member of the trio. + +### Cloudflare deployment token + +1. Create a second token with the exact scoped permissions in `infra/cloudflare/CHECKLIST.md`. +2. replace `CLOUDFLARE_API_TOKEN` in GitHub, run a read-only or no-op reconciliation, then a normal + deployment. +3. Revoke the old token after success and record the rotation date. + +### Clerk secret key + +1. Issue the next secret in Clerk. +2. replace `CLERK_SECRET_KEY` in the GitHub production environment and Worker secret binding. +3. Verify token validation and an authenticated relay endpoint. +4. Revoke the old key after the overlap window. + +Client builds use the publishable key; a Clerk domain change can also require a new publishable key +and a new client build. + +### Tunnel credentials + +For a remotely managed tunnel, rotate `CLOUDFLARE_TUNNEL_TOKEN`, update the root-readable systemd +environment file, restart one connector replica, verify it, then replace the other replica. For a +locally managed tunnel, create a new tunnel credential JSON and replace replicas one at a time. +`cert.pem` is not a runtime credential and should not remain on the server when management is done +elsewhere. + +### TLS certificates and backups + +- Renew the database certificate before one-third of its lifetime remains, reload PostgreSQL, and + test Hyperdrive TLS connectivity. +- Rotate `BACKUP_ENCRYPTION_KEY` by writing new backups with the new key while retaining the old key + until all backups encrypted by it expire. +- Test recovery after every rotation. Do not discover missing keys during an incident. + +## Change and rollback rules + +- Take and verify a logical backup before schema or major-version changes. +- Apply `schema.sql`, `seed.sql`, and `rls.sql` only to a fresh database. Existing deployments move + forward through reviewed migrations. +- Never use Supabase `reset.sh` or `docker compose down -v` as rollback. +- Keep the previous Worker version deployable until the database change is proven compatible. +- Do not downgrade PostgreSQL data directories. Restore into the matching PostgreSQL major version + and migrate forward. +- Record the pinned Supabase commit and image digests used for every production change. diff --git a/infra/supabase/rls.sql b/infra/supabase/rls.sql new file mode 100644 index 0000000000000..a955e38ade5f0 --- /dev/null +++ b/infra/supabase/rls.sql @@ -0,0 +1,69 @@ +-- Row Level Security for every current T3 Connect relay table. +-- +-- Supabase creates anon, authenticated, and service_role. The Worker should connect through a +-- dedicated LOGIN role named relay_runtime that is a member of service_role. Supabase commonly +-- gives service_role BYPASSRLS; the explicit policies and grants below still document the intended +-- authorization boundary and support deployments where that attribute is removed. + +BEGIN; + +GRANT USAGE ON SCHEMA public TO service_role; + +DO $rls$ +DECLARE + relay_table text; + relay_tables constant text[] := ARRAY[ + 'relay_mobile_devices', + 'relay_live_activities', + 'relay_environment_links', + 'relay_managed_endpoint_allocations', + 'relay_managed_tunnel_limits', + 'relay_environment_credentials', + 'relay_agent_activity_rows', + 'relay_delivery_attempts', + 'relay_dpop_proofs' + ]; +BEGIN + FOREACH relay_table IN ARRAY relay_tables LOOP + EXECUTE format('ALTER TABLE public.%I ENABLE ROW LEVEL SECURITY', relay_table); + + EXECUTE format( + 'REVOKE ALL PRIVILEGES ON TABLE public.%I FROM PUBLIC, anon, authenticated', + relay_table + ); + EXECUTE format( + 'GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE public.%I TO service_role', + relay_table + ); + + EXECUTE format( + 'DROP POLICY IF EXISTS relay_service_role_full_access ON public.%I', + relay_table + ); + EXECUTE format( + 'CREATE POLICY relay_service_role_full_access ON public.%I FOR ALL TO service_role USING (true) WITH CHECK (true)', + relay_table + ); + + EXECUTE format( + 'DROP POLICY IF EXISTS relay_anon_deny_all ON public.%I', + relay_table + ); + EXECUTE format( + 'CREATE POLICY relay_anon_deny_all ON public.%I AS RESTRICTIVE FOR ALL TO anon USING (false) WITH CHECK (false)', + relay_table + ); + + EXECUTE format( + 'DROP POLICY IF EXISTS relay_authenticated_deny_all ON public.%I', + relay_table + ); + EXECUTE format( + 'CREATE POLICY relay_authenticated_deny_all ON public.%I AS RESTRICTIVE FOR ALL TO authenticated USING (false) WITH CHECK (false)', + relay_table + ); + END LOOP; +END +$rls$; + +COMMIT; diff --git a/infra/supabase/schema.sql b/infra/supabase/schema.sql new file mode 100644 index 0000000000000..2c6628f5b296b --- /dev/null +++ b/infra/supabase/schema.sql @@ -0,0 +1,174 @@ +-- T3 Connect relay schema for a fresh PostgreSQL database. +-- +-- Authoritative definitions: +-- infra/relay/src/persistence/schema.ts +-- infra/relay/migrations/postgres/*/migration.sql +-- +-- This file represents the current schema after all checked-in migrations. In particular, +-- idx_relay_live_activities_user and idx_relay_mobile_devices_user are intentionally absent because +-- infra/relay/migrations/postgres/20260727144838_migration/migration.sql drops them. + +BEGIN; + +CREATE TABLE public.relay_mobile_devices ( + user_id varchar(255) NOT NULL, + device_id varchar(255) NOT NULL, + label text DEFAULT 'iOS device' NOT NULL, + platform varchar(16) NOT NULL, + ios_major_version integer NOT NULL, + app_version varchar(64), + bundle_id varchar(255), + aps_environment varchar(16), + push_token text, + push_to_start_token text, + preferences_json jsonb NOT NULL, + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL, + CONSTRAINT relay_mobile_devices_pkey PRIMARY KEY (user_id, device_id) +); + +CREATE UNIQUE INDEX idx_relay_mobile_devices_push_token + ON public.relay_mobile_devices (push_token); + +CREATE UNIQUE INDEX idx_relay_mobile_devices_push_to_start_token + ON public.relay_mobile_devices (push_to_start_token); + +CREATE TABLE public.relay_live_activities ( + user_id varchar(255) NOT NULL, + device_id varchar(255) NOT NULL, + activity_push_token text, + remote_start_queued_at varchar(64), + remote_started_at varchar(64), + ended_at varchar(64), + last_aggregate_json jsonb, + last_live_activity_delivery_at varchar(64), + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL, + CONSTRAINT relay_live_activities_pkey PRIMARY KEY (user_id, device_id) +); + +CREATE UNIQUE INDEX idx_relay_live_activities_activity_push_token + ON public.relay_live_activities (activity_push_token); + +CREATE TABLE public.relay_environment_links ( + user_id varchar(191) NOT NULL, + environment_id varchar(191) NOT NULL, + environment_label text DEFAULT 'T3 Environment' NOT NULL, + environment_public_key text NOT NULL, + endpoint_http_base_url text NOT NULL, + endpoint_ws_base_url text NOT NULL, + endpoint_provider_kind varchar(32) NOT NULL, + notifications_enabled boolean DEFAULT true NOT NULL, + live_activities_enabled boolean DEFAULT true NOT NULL, + managed_tunnels_enabled boolean DEFAULT false NOT NULL, + created_by_device_id varchar(191), + revoked_at varchar(64), + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL, + CONSTRAINT relay_environment_links_pkey PRIMARY KEY (user_id, environment_id) +); + +CREATE INDEX idx_relay_environment_links_environment + ON public.relay_environment_links (environment_id, revoked_at); + +CREATE TABLE public.relay_managed_endpoint_allocations ( + user_id varchar(191) NOT NULL, + environment_id varchar(191) NOT NULL, + hostname text NOT NULL, + tunnel_id varchar(191), + tunnel_name text NOT NULL, + dns_record_id varchar(191), + ready_at varchar(64), + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL, + CONSTRAINT relay_managed_endpoint_allocations_pkey PRIMARY KEY (user_id, environment_id) +); + +CREATE UNIQUE INDEX idx_relay_managed_endpoint_allocations_hostname + ON public.relay_managed_endpoint_allocations (hostname); + +CREATE UNIQUE INDEX idx_relay_managed_endpoint_allocations_tunnel_name + ON public.relay_managed_endpoint_allocations (tunnel_name); + +CREATE TABLE public.relay_managed_tunnel_limits ( + user_id varchar(191) PRIMARY KEY, + max_tunnels integer NOT NULL, + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL +); + +CREATE TABLE public.relay_environment_credentials ( + credential_id varchar(64) PRIMARY KEY, + environment_id varchar(191) NOT NULL, + environment_public_key text NOT NULL, + credential_hash varchar(191) NOT NULL, + revoked_at varchar(64), + created_at varchar(64) NOT NULL, + updated_at varchar(64) NOT NULL +); + +CREATE UNIQUE INDEX idx_relay_environment_credentials_hash + ON public.relay_environment_credentials (credential_hash); + +CREATE INDEX idx_relay_environment_credentials_environment + ON public.relay_environment_credentials (environment_id, revoked_at); + +CREATE INDEX idx_relay_environment_credentials_environment_key + ON public.relay_environment_credentials ( + environment_id, + environment_public_key, + revoked_at + ); + +CREATE TABLE public.relay_agent_activity_rows ( + environment_id varchar(191) NOT NULL, + environment_public_key text NOT NULL, + thread_id varchar(191) NOT NULL, + state_json jsonb NOT NULL, + updated_at varchar(64) NOT NULL, + created_at varchar(64) NOT NULL, + CONSTRAINT relay_agent_activity_rows_pkey PRIMARY KEY ( + environment_id, + environment_public_key, + thread_id + ) +); + +CREATE INDEX idx_relay_agent_activity_rows_updated + ON public.relay_agent_activity_rows (updated_at); + +CREATE TABLE public.relay_delivery_attempts ( + id varchar(36) PRIMARY KEY, + created_at varchar(64) NOT NULL, + user_id varchar(255), + environment_id varchar(191), + thread_id varchar(191), + device_id varchar(255), + kind varchar(64) NOT NULL, + source_job_id varchar(64), + token_suffix varchar(16), + apns_status integer, + apns_reason text, + apns_id varchar(128), + transport_error text +); + +CREATE INDEX idx_relay_delivery_attempts_environment + ON public.relay_delivery_attempts (environment_id, thread_id, created_at); + +CREATE UNIQUE INDEX idx_relay_delivery_attempts_source_job + ON public.relay_delivery_attempts (source_job_id); + +CREATE TABLE public.relay_dpop_proofs ( + thumbprint varchar(128) NOT NULL, + jti varchar(255) NOT NULL, + iat integer NOT NULL, + expires_at varchar(64) NOT NULL, + created_at varchar(64) NOT NULL, + CONSTRAINT relay_dpop_proofs_pkey PRIMARY KEY (thumbprint, jti) +); + +CREATE INDEX idx_relay_dpop_proofs_expires_at + ON public.relay_dpop_proofs (expires_at); + +COMMIT; diff --git a/infra/supabase/seed.sql b/infra/supabase/seed.sql new file mode 100644 index 0000000000000..94080ba1f3eb9 --- /dev/null +++ b/infra/supabase/seed.sql @@ -0,0 +1,13 @@ +-- T3 Connect relay seed contract. +-- +-- The current relay has no required seed or configuration rows. All persistent rows are created by +-- authenticated runtime operations. The default managed tunnel limit is the code constant +-- DEFAULT_MANAGED_TUNNEL_LIMIT = 3 in +-- infra/relay/src/environments/ManagedTunnelLimits.ts; a row in +-- relay_managed_tunnel_limits is only a per-user override. +-- +-- Keep this file as an explicit, executable no-op so fresh-environment automation can always apply +-- schema.sql, seed.sql, and rls.sql in the same order. + +BEGIN; +COMMIT;