Scalable and flexible JWT based authentication

[alex-berger]

Scalable and flexible JWT based authentication

Motivation / Goals

We intend to leverage CockroachDB's
Cluster Single Sign-on (SSO) using a JSON web token (JWT) capability to enable secure database access for Kubernetes workloads (Pods) without the need to explicitly have to maintain (create, rotate, delete) authentication credentials. We can achieve this by using ServiceAccount token volume projection to provision such workloads with JWT tokens suitable for authentication with CockroachDB.

• Support up to several dozens or hundreds (but not thousands) JWT issuers per database cluster.
• Reduce blast radius of misconfigured issuers by scoping configuration per issuer instead of exposing global configuration to end-user, such that misconfigured issuer will not negatively impact valid issuers.

Problem Statement

Starting with v22.2 CockroachDB added support for OpenID Connect (OIDC) compatible JWT based authentication as documented in Cluster Single Sign-on (SSO) using a JSON web token (JWT). However, the current implementation based on the below outlined cluster settings has several drawbacks and limitations.

Cluster Setting	Remarks
server.jwt_authentication.enabled	Serves as general feature flag and does not pose any problems.
server.jwt_authentication.jwks	Problematic to maintain if there are many issuers, which in turn can have many signing keys. If there is/were no size limit on this settings value it might scale in terms of the number or signing keys, but still it is very brittle and error prone to manage those keys. It's not obvioius which JWK belongs to which issuer. It could easily happen that during update of this setting some keys get lost either due to (a) user errror or (b) potentially parallel updates (read/modify/write) that would be subject to race conditions (without proper locking).
server.jwt_authentication.issuers	Problematic to maintain if there are many issuers. If there is/were no size limit on this settings value it might scale in terms of the number or issuers, but still it is very brittle and error prone to manage those. It could easily happen that during update of this setting some issuers get lost either due to (a) user errror or (b) potentially parallel updates (read/modify/write) that would be subject to race conditions (without proper locking)
server.jwt_authentication.audience	Problematic to maintain if there are many audiences. If there is/were no size limit on this settings value it might scale in terms of the number or audiences, but still it is very brittle and error prone to manage those. It's not obvioius which audience belongs to which issuer. It could easily happen that during update of this setting some audience gets lost either due to (a) user errror or (b) potentially parallel updates (read/modify/write) that would be subject to race conditions (without proper locking).
server.identity_map.configuration	Problematic to maintain if there are many issuers. It could easily happen that during update of this setting some mappings gets lost either due to (a) user errror or (b) potentially parallel updates (read/modify/write) that would be subject to race conditions (without proper locking).

---

Proposal

We propose to extend CockroachDB with more powerfull support to configure and manage Identity Providers (IdP), which support JWT resp. which are OpenID Connect compatible.

Managing Issuers

Creating Issuers

We propose to introduce an explicit JWT_ISSUER resource type, which supports CREATE OR REPLACE ... semantics to enable atomic creation and update of a JWT issuer configuration. The issuer name must be unique and must support at least 1024 characters.

CREATE [OR REPLACE] JWT_ISSUER "https://example.com/my/issuer"
    WITH
        oidc_discovery = false,
        identity_map = ARRAY['/^system:serviceaccount:([^:]+):([^:]+)$ \1:\2'],
        audience = ARRAY['a', 'b'],
        jwks = '{"keys": [{"kty":"...","e":"...","use":"sig","kid":"...","alg":"...","n":"..."}, {...}]}'
;

Option	Type	Format	Description
oidc_discovery	BOOL		Whether to use OpenID Connect Discovery or not to automatically discover (and cache) the signing keys (JWKs) for that issuer. Enabling JWK auto discovery only makes sense for public issuers respectively for issuers with discovery endpoints that are network-reachable by the CockroachDB nodes.
identity_map	STRING[]	Similar to server.identity_map.configuration but without the issuer prefix.	Identity mapping
audience	STRING[]		Valid audience values.
jwks	JSONB	JSON Web Key Set (JWKS)	Issuer's signing keys.

Listing Issuers

SHOW JWT_ISSUERS;

ISSUER                          | OIDC_DISCOVERY | AUDIENCE | JWKS  | IDENTITY_MAP |
--------------------------------+----------------+----------+-------+--------------+
https://example.com/my/issuer   | false          | {a,b}    | {...} | {...}        | 
...

Column	Type	Format
ISSUER	STRING
OIDC_DISCOVERY	BOOL
IDENTITY_MAP	STRING[]	Similar to server.identity_map.configuration but without the issuer prefix.
AUDIENCE	STRING[]
JWKS	JSONB	JSON Web Key Set (JWKS)

Droping Issuers

Dropping an issuer, will also cascadingly drop all related JWKs, audience and identity mapping configuration.

DROP JWT_ISSUER [IF EXISTS] "https://example.com/my/issuer";

Related Issues and Pull Request

• Scalable and flexible JWT based authentication #103110 (CRDB-28040 : JWKS fetch from jwks_uri #117054)
• jwt: support authorization #104671
• oidc: support for matching on group claims from auth provider #97301

[abhinavg6]

Thanks for filing this @alex-berger. We'll discuss it internally and then get back to you.

cc @kpatron-cockroachlabs @rafiss @dikshant

[Yash-xoxo]

This is a great proposal for scaling authentication! A few thoughts:

On the issuer management design:

The table-based approach for managing issuers is solid. One suggestion - add a valid_from timestamp alongside JWKS URL validation to support key rotation without downtime:

CREATE TABLE system.jwt_issuers (
  issuer STRING PRIMARY KEY,
  jwks_url STRING,
  valid_from TIMESTAMP,
  audience STRING[]
)

On identity mapping:

The identity_map JSON structure is flexible, but consider adding a priority field for when multiple rules could match:

{
  "map": [
    {"claim": "sub", "prefix": "user:", "priority": 1},
    {"claim": "email", "extract": "^(.+)@company.com$", "priority": 2}
  ]
}

Security consideration:

For jwks_only_pubkey, ensure the JWKS endpoint itself is validated (HTTPS, cert pinning?) to prevent MITM attacks during key fetch.

Question: How does this interact with existing server.jwt_authentication.issuers cluster setting? Migration path for existing JWT configs?

Nice additions:

• Automatic JWKS refresh on key rotation
• Per-issuer audience validation
• Clear separation of issuer config from identity mapping

Excited to see this move forward - JWT at scale is a common pain point!