Skip to content

Commit aa36be2

Browse files
YujohnNattrassMastra CodeYJ
authored
feat(storage): add tool_provider_connections domain (core + libsql) — PR 1/3 of #17224 split (#17247)
# PR 1 of 3 — split of #17224 (v1 ToolProvider backend) Adds the `tool_provider_connections` storage domain that the v1 ToolProvider runtime (PR 2) and the label PATCH endpoint (PR 3) build on. No runtime, no server routes, no SDK surface — pure storage. Also adds a `toolProviders` jsonb column to `AGENT_VERSIONS_SCHEMA` so stored agents can persist a per-agent ToolProvider config (consumed in PR 2). Nullable, additive — no migration story for existing rows. ## Scope (18 files, +1135 / -3) **Core storage domain (new)** - `packages/core/src/storage/domains/tool-provider-connections/base.ts` — `ToolProviderConnectionsStorage` abstract class (`getConnectionById`, `upsertConnection`, `listConnectionsByAuthor`, `deleteConnection`, `dangerouslyClearAll`) - `packages/core/src/storage/domains/tool-provider-connections/index.ts` — barrel - `packages/core/src/storage/domains/tool-provider-connections/inmemory.ts` — in-memory reference impl - `packages/core/src/storage/domains/tool-provider-connections/inmemory.test.ts` — 15 tests covering all CRUD paths **Core storage wiring** - `packages/core/src/storage/base.ts` — abstract `toolProviders` getter - `packages/core/src/storage/constants.ts` — `TABLE_TOOL_PROVIDER_CONNECTIONS = 'mastra_tool_provider_connections'`; **`AGENT_VERSIONS_SCHEMA` gains `toolProviders: { type: 'jsonb', nullable: true }`** so stored agent rows can carry per-agent ToolProvider config - `packages/core/src/storage/types.ts` — `StorageToolProviderConnection`, `StorageUpsertToolProviderConnectionInput`, `StorageToolProviderConnectionScope` (union of `'shared' | 'per-author' | 'caller-supplied'`), `StorageToolProviderConnectionKey` - `packages/core/src/storage/mock.ts` — mock storage exposes new domain - `packages/core/src/storage/domains/index.ts` — re-export - `packages/core/src/storage/domains/inmemory-db.ts` — in-memory table - `packages/core/src/storage/domains/operations/inmemory.ts` — operations bind - `packages/core/src/storage/domains/agents/filesystem.ts` — `'toolProviders'` added to `PERSISTED_SNAPSHOT_FIELDS` so the filesystem agent storage round-trips the new column **libsql adapter** - `stores/libsql/src/storage/domains/tool-provider-connections/index.ts` — libsql impl of `ToolProviderConnectionsStorage` - `stores/libsql/src/storage/domains/tool-provider-connections/index.test.ts` — 16 tests - `stores/libsql/src/storage/domains/agents/index.ts` — schema migration via `alterTable(... ifNotExists: ['toolProviders'])` for `TABLE_AGENT_VERSIONS`; read/write/create paths round-trip `toolProviders` (`parseJson` on read, `?? null` on insert) - `stores/libsql/src/storage/index.ts` — wire domain into `LibSQLStore` **Cross-store table-constant recognition (no impl)** - `stores/clickhouse/src/storage/db/utils.ts` — recognize new table constant - `stores/cloudflare/src/kv/storage/types.ts` — recognize new table constant - These stores do **not** implement `ToolProviderConnectionsStorage` yet. Same applies to pg / mssql / mongodb / etc. That is acceptable here because `StorageDomains.toolProviderConnections` is optional, but it means PR 2's runtime resolver will silently no-op on those stores until adapter impls land. PR 2 will flag this — see the "Forward-looking notes" section below. **Changeset** - `.changeset/v1-tp-storage.md` ## Intentionally excluded - Tool-provider runtime resolver, server routes, `@mastra/client-js` resource, Composio rewrite — see **PR 2** (stacked on this one). - `SHARED_BUCKET_ID` constant (used to bucket `scope: 'shared'` connections under a known authorId) lives in the runtime layer, not storage — it arrives in PR 2. - PATCH endpoint to rename connection labels — see **PR 3** (stacked on PR 2). ## Naming choices in this PR CRUD methods on the storage class use `getConnectionById` / `upsertConnection` / `listConnectionsByAuthor` / `deleteConnection` to match the `verb-Entity[-By-key]` pattern used by sibling storage domains (channels, favorites, etc.). Bare `get`/`upsert`/`list`/`delete` were ruled out because they collide with mental models from other domains. ## Forward-looking notes (not blocking this PR) - **Adapter coverage** — only `libsql` implements `ToolProviderConnectionsStorage` in this PR. `clickhouse` / `cloudflare` / `pg` / `mssql` / `mongodb` only get the table constant added to their `TABLE_NAMES` union. PR 2 should call this out so reviewers don't expect Composio/Arcade tool resolution to work on those stores until follow-ups land, and may want to add an explicit "not implemented" path that throws rather than silently no-op. - **Admin cross-author listing** — `listConnectionsByAuthor` with `authorId` omitted returns rows across all authors (documented admin path). PR 2 must ensure no unauthenticated server route reaches this branch. ## Verification - `pnpm build:core` — clean - `pnpm --filter ./packages/core test` (focused on `tool-provider-connections/inmemory.test.ts`) — 15/15 pass - `pnpm --filter ./stores/libsql build` — clean - `pnpm --filter ./stores/libsql test` — 721/721 pass (16 new for tool-provider-connections) ## Stack 1. **PR 1 / 3** (this PR) — storage domain 2. **PR 2 / 3** — runtime + server + client SDK + editor wiring 3. **PR 3 / 3** — PATCH endpoint to rename connection labels Backup of the original branch lives at `backup/v1-tp-backend-original` locally for cross-checking; range-diff against the full stack is functionally identical (whitespace-only delta in one test file). Replaces #17224. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## ELI5 This PR adds a storage system for remembering how users have connected to external tools and services. Instead of asking users to re-authenticate every time they use a tool, the system can now save and reuse these connections—like remembering a username and password so you don't have to log in again. ## Overview Introduces a new `tool_provider_connections` storage domain to enable stored agents to persist per-agent ToolProvider OAuth-style connection configurations. This storage-only implementation (no runtime, server routes, or SDK surface) allows agent runs to resolve connections via multiple scopes (per-author, shared, or caller-supplied) without re-prompting for authentication on each run. ## Core Changes **Storage Domain Contract** (`packages/core/src/storage/domains/tool-provider-connections/`) - New abstract `ToolProviderConnectionsStorage` class defining six key methods: `init()`, `getConnectionById()`, `upsertConnection()`, `listConnectionsByAuthor()`, `deleteConnection()`, and `dangerouslyClearAll()` - In-memory reference implementation (`InMemoryToolProviderConnectionsStorage`) with 15 passing tests covering upsert, retrieval, listing with optional filters, deletion, and scope handling - Type definitions for `StorageToolProviderConnection`, `StorageUpsertToolProviderConnectionInput`, `StorageToolProviderConnectionScope` ('shared' | 'per-author' | 'caller-supplied'), and related input types **Schema & Type Updates** - New `TOOL_PROVIDER_CONNECTIONS_SCHEMA` with composite primary key on `(authorId, providerId, connectionId)` - `AGENT_VERSIONS_SCHEMA` extended with optional `toolProviders` JSONB column to persist per-agent ToolProvider configurations (additive, nullable; no migration required for existing rows) - New type exports in `StorageAgentSnapshotType` to support agent-level tool provider configs - Storage type wiring into `StorageDomains` interface and `EDITOR_DOMAINS` routing list **Storage Layer Integration** - Wired into core `InMemoryStore` and mock storage via shared `InMemoryDB` - Updated `InMemoryDB` to track tool-provider connections with composite key formatting - Filesystem agent storage updated to persist `toolProviders` field in snapshots (`PERSISTED_SNAPSHOT_FIELDS` includes `toolProviders`) - In-memory operations table initialized with new storage table ## LibSQL Adapter **Domain Implementation** (`stores/libsql/src/storage/domains/tool-provider-connections/`) - `ToolProviderConnectionsLibSQL` class extending base storage domain - `init()` creates table with composite primary key and index for author-scoped queries - CRUD operations implemented with parameterized queries, transactional upsert logic, scope normalization, date conversion, and MastraError-wrapped errors - Upsert checks for existing row then UPDATEs or INSERTs with proper createdAt/updatedAt handling - 16 passing tests validating insert, update, filtering, deletion, and cross-author isolation **Agent Storage Updates** (`stores/libsql/src/storage/domains/agents/`) - `AgentsLibSQL.init()` altered to add `toolProviders` column if missing (non-breaking migration) - Legacy migration copies `toolProviders` from legacy rows - `createVersion()` now persists `input.toolProviders` - `parseVersionRow()` parses `toolProviders` from result rows **Storage Index Wiring** (`stores/libsql/src/storage/index.ts`) - `ToolProviderConnectionsLibSQL` instantiated and included in `LibSQLStore.stores` map ## Cross-Store Table Constant Updates - **ClickHouse** (`stores/clickhouse/src/storage/db/utils.ts`): Added `TABLE_TOOL_PROVIDER_CONNECTIONS` entry with `ReplacingMergeTree()` engine config - **Cloudflare** (`stores/cloudflare/src/kv/storage/types.ts`): Extended `RecordTypes` mapping to recognize the new table constant and associate it with `StorageToolProviderConnection` - Other adapters (Postgres, MSSQL, MongoDB, etc.): table constant recognized but domain implementation deferred (storage domain optional) ## Changeset & Release Notes Added changeset documenting minor/patch version bumps for `@mastra/core`, `@mastra/libsql`, `@mastra/clickhouse`, and `@mastra/cloudflare`. Notes include usage example for upsert/list operations and clarify that existing stored agents remain compatible; runtime integration will follow in subsequent PRs. ## Test Coverage - Core in-memory: 15 tests (upsert, retrieval, filtering, deletion, scope behavior, clear-all) - LibSQL: 16 tests (same coverage plus LibSQL-specific transaction and date handling) - Relevant test suites pass for core and libsql ## Intentionally Excluded (Planned for Follow-Up PRs) - Runtime resolver and server routes (PR 2) - Client SDK and Composio changes (PR 2) - PATCH endpoint for label renames and other runtime-facing features (PR 3) <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/17247?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Mastra Code <noreply@mastra.ai> Co-authored-by: YJ <yj@example.com>
1 parent 3fce5e7 commit aa36be2

19 files changed

Lines changed: 1188 additions & 3 deletions

File tree

.changeset/frank-zebras-learn.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
'@mastra/core': minor
3+
'@mastra/libsql': minor
4+
'@mastra/clickhouse': patch
5+
'@mastra/cloudflare': patch
6+
---
7+
8+
Added the `tool_provider_connections` storage domain. Stored agents can now persist per-agent ToolProvider config that round-trips on read/write/create. Runtime connection resolution (per-author, shared, caller-supplied) ships in a follow-up PR.
9+
10+
**What you can do**
11+
12+
- Pin a connection on a stored agent's config and have it round-trip on read/write/create.
13+
- Persist multiple connections per toolkit so a follow-up runtime PR can fan-out to the right one at execution time.
14+
15+
**Example**
16+
17+
```ts
18+
import { LibSQLStore } from '@mastra/libsql';
19+
20+
const storage = new LibSQLStore({ url: process.env.DATABASE_URL });
21+
22+
// Persist an OAuth connection that an agent can pin later
23+
await storage.toolProviders.upsertConnection({
24+
authorId: 'user-123',
25+
providerId: 'composio',
26+
connectionId: 'auth_abc',
27+
toolkit: 'gmail',
28+
label: 'Work inbox',
29+
scope: 'per-author',
30+
});
31+
32+
// List a user's own connections (admin can omit authorId to list across users)
33+
const { items } = await storage.toolProviders.listConnectionsByAuthor({
34+
authorId: 'user-123',
35+
providerId: 'composio',
36+
});
37+
```
38+
39+
Additive — existing stored agents continue to work unchanged. The runtime that consumes this domain ships in a follow-up PR.
40+
41+
PR 1 of 3 split from #17224.

packages/core/src/storage/base.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import type {
1919
BackgroundTasksStorage,
2020
SchedulesStorage,
2121
ChannelsStorage,
22+
ToolProviderConnectionsStorage,
2223
} from './domains';
2324

2425
/** Map of all storage domain interfaces available in a composite store. */
@@ -41,6 +42,7 @@ export type StorageDomains = {
4142
blobs?: BlobStore;
4243
backgroundTasks?: BackgroundTasksStorage;
4344
schedules?: SchedulesStorage;
45+
toolProviderConnections?: ToolProviderConnectionsStorage;
4446
};
4547

4648
/**
@@ -57,6 +59,7 @@ export const EDITOR_DOMAINS = [
5759
'workspaces',
5860
'skills',
5961
'favorites',
62+
'toolProviderConnections',
6063
] as const satisfies ReadonlyArray<keyof StorageDomains>;
6164

6265
/**
@@ -324,6 +327,7 @@ export class MastraCompositeStore extends MastraBase {
324327
backgroundTasks: resolve('backgroundTasks'),
325328
schedules: resolve('schedules'),
326329
channels: resolve('channels'),
330+
toolProviderConnections: resolve('toolProviderConnections'),
327331
} as StorageDomains;
328332
}
329333
// Otherwise, subclasses set stores themselves
@@ -450,6 +454,7 @@ export class MastraCompositeStore extends MastraBase {
450454
maybeInit(this.stores.backgroundTasks);
451455
maybeInit(this.stores.schedules);
452456
maybeInit(this.stores.channels);
457+
maybeInit(this.stores.toolProviderConnections);
453458
}
454459

455460
await Promise.all(initTasks);

packages/core/src/storage/constants.ts

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,9 @@ export const TABLE_SCHEDULE_TRIGGERS = 'mastra_schedule_triggers';
4545
export const TABLE_CHANNEL_INSTALLATIONS = 'mastra_channel_installations';
4646
export const TABLE_CHANNEL_CONFIG = 'mastra_channel_config';
4747

48+
// Tool provider connections
49+
export const TABLE_TOOL_PROVIDER_CONNECTIONS = 'mastra_tool_provider_connections';
50+
4851
/** Union of all core table name constants. */
4952
export type TABLE_NAMES =
5053
| typeof TABLE_WORKFLOW_SNAPSHOT
@@ -79,7 +82,8 @@ export type TABLE_NAMES =
7982
| typeof TABLE_SCHEDULES
8083
| typeof TABLE_SCHEDULE_TRIGGERS
8184
| typeof TABLE_CHANNEL_INSTALLATIONS
82-
| typeof TABLE_CHANNEL_CONFIG;
85+
| typeof TABLE_CHANNEL_CONFIG
86+
| typeof TABLE_TOOL_PROVIDER_CONNECTIONS;
8387

8488
export const SCORERS_SCHEMA: Record<string, StorageColumn> = {
8589
id: { type: 'text', nullable: false, primaryKey: true },
@@ -172,6 +176,7 @@ export const AGENT_VERSIONS_SCHEMA: Record<string, StorageColumn> = {
172176
workflows: { type: 'jsonb', nullable: true },
173177
agents: { type: 'jsonb', nullable: true },
174178
integrationTools: { type: 'jsonb', nullable: true },
179+
toolProviders: { type: 'jsonb', nullable: true },
175180
inputProcessors: { type: 'jsonb', nullable: true },
176181
outputProcessors: { type: 'jsonb', nullable: true },
177182
memory: { type: 'jsonb', nullable: true },
@@ -338,6 +343,25 @@ export const FAVORITES_SCHEMA: Record<string, StorageColumn> = {
338343
createdAt: { type: 'timestamp', nullable: false },
339344
};
340345

346+
/**
347+
* Per-author registry of authorized tool provider connections. Stores a stable
348+
* user-supplied label across agents. Composite primary key on
349+
* (authorId, providerId, connectionId). `scope` buckets identity:
350+
* 'per-author' (default), 'shared' (visible to all callers), or
351+
* 'caller-supplied' (authorId is a host-app end-user id forwarded via request
352+
* context).
353+
*/
354+
export const TOOL_PROVIDER_CONNECTIONS_SCHEMA: Record<string, StorageColumn> = {
355+
authorId: { type: 'text', nullable: false },
356+
providerId: { type: 'text', nullable: false },
357+
connectionId: { type: 'text', nullable: false },
358+
toolkit: { type: 'text', nullable: false },
359+
label: { type: 'text', nullable: true },
360+
scope: { type: 'text', nullable: false },
361+
createdAt: { type: 'timestamp', nullable: false },
362+
updatedAt: { type: 'timestamp', nullable: false },
363+
};
364+
341365
export const SKILL_VERSIONS_SCHEMA: Record<string, StorageColumn> = {
342366
id: { type: 'text', nullable: false, primaryKey: true },
343367
skillId: { type: 'text', nullable: false },
@@ -636,6 +660,7 @@ export const TABLE_SCHEMAS: Record<TABLE_NAMES, Record<string, StorageColumn>> =
636660
data: { type: 'jsonb', nullable: false },
637661
updatedAt: { type: 'timestamp', nullable: false },
638662
},
663+
[TABLE_TOOL_PROVIDER_CONNECTIONS]: TOOL_PROVIDER_CONNECTIONS_SCHEMA,
639664
};
640665

641666
/**
@@ -645,6 +670,10 @@ export const TABLE_SCHEMAS: Record<TABLE_NAMES, Record<string, StorageColumn>> =
645670
export const TABLE_CONFIGS: Partial<Record<TABLE_NAMES, StorageTableConfig>> = {
646671
[TABLE_DATASET_ITEMS]: { columns: DATASET_ITEMS_SCHEMA, compositePrimaryKey: ['id', 'datasetVersion'] },
647672
[TABLE_FAVORITES]: { columns: FAVORITES_SCHEMA, compositePrimaryKey: ['userId', 'entityType', 'entityId'] },
673+
[TABLE_TOOL_PROVIDER_CONNECTIONS]: {
674+
columns: TOOL_PROVIDER_CONNECTIONS_SCHEMA,
675+
compositePrimaryKey: ['authorId', 'providerId', 'connectionId'],
676+
},
648677
};
649678

650679
/**

packages/core/src/storage/domains/agents/filesystem.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ const PERSISTED_SNAPSHOT_FIELDS = new Set([
2121
'model',
2222
'tools',
2323
'integrationTools',
24+
'toolProviders',
2425
'mcpClients',
2526
'requestContextSchema',
2627
]);

packages/core/src/storage/domains/index.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,4 @@ export * from './datasets';
2121
export * from './experiments';
2222
export * from './background-tasks';
2323
export * from './schedules';
24-
export * from './schedules';
24+
export * from './tool-provider-connections';

packages/core/src/storage/domains/inmemory-db.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import type {
1212
StorageFavoriteType,
1313
StorageWorkspaceType,
1414
StorageSkillType,
15+
StorageToolProviderConnection,
1516
StorageWorkflowRun,
1617
ObservationalMemoryRecord,
1718
DatasetRecord,
@@ -99,6 +100,11 @@ export class InMemoryDB {
99100
readonly schedules = new Map<string, Schedule>();
100101
readonly scheduleTriggers: ScheduleTrigger[] = [];
101102

103+
/**
104+
* Tool provider connections keyed by `${authorId}\u0000${providerId}\u0000${connectionId}`.
105+
*/
106+
readonly toolProviderConnections = new Map<string, StorageToolProviderConnection>();
107+
102108
/**
103109
* Clears all data from all collections.
104110
* Useful for testing.
@@ -145,5 +151,6 @@ export class InMemoryDB {
145151
this.backgroundTasks.clear();
146152
this.schedules.clear();
147153
this.scheduleTriggers.length = 0;
154+
this.toolProviderConnections.clear();
148155
}
149156
}

packages/core/src/storage/domains/operations/inmemory.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ export class StoreOperationsInMemory extends StoreOperations {
4646
mastra_schedule_triggers: new Map(),
4747
mastra_channel_installations: new Map(),
4848
mastra_channel_config: new Map(),
49+
mastra_tool_provider_connections: new Map(),
4950
};
5051
}
5152

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
import { MastraBase } from '../../../base';
2+
import type {
3+
StorageDeleteToolProviderConnectionInput,
4+
StorageListToolProviderConnectionsInput,
5+
StorageToolProviderConnection,
6+
StorageToolProviderConnectionKey,
7+
StorageUpsertToolProviderConnectionInput,
8+
} from '../../types';
9+
10+
/**
11+
* Abstract base class for the tool-provider-connections storage domain.
12+
*
13+
* Persists a per-author, provider-agnostic registry of authorized tool
14+
* provider connections so the UI can surface a stable, user-supplied label
15+
* (e.g. "Work Gmail") across agents. Rows are keyed by
16+
* `(authorId, providerId, connectionId)`. The label is the only mutable field.
17+
*
18+
* Adapter-native connection state (status, scopes, expiry) still lives with the
19+
* provider — this domain is purely a name lookup.
20+
*/
21+
export abstract class ToolProviderConnectionsStorage extends MastraBase {
22+
constructor() {
23+
super({
24+
component: 'STORAGE',
25+
name: 'TOOL_PROVIDER_CONNECTIONS',
26+
});
27+
}
28+
29+
/** Initialize the store (create tables, indexes, etc). */
30+
abstract init(): Promise<void>;
31+
32+
/**
33+
* Fetch a single tool provider connection row. Returns `null` when no row
34+
* exists for the given `(authorId, providerId, connectionId)`.
35+
*/
36+
abstract getConnectionById(key: StorageToolProviderConnectionKey): Promise<StorageToolProviderConnection | null>;
37+
38+
/**
39+
* Insert or update a tool provider connection row. Idempotent on
40+
* `(authorId, providerId, connectionId)` — the existing label/toolkit are
41+
* overwritten. `createdAt` is preserved on update.
42+
*/
43+
abstract upsertConnection(input: StorageUpsertToolProviderConnectionInput): Promise<StorageToolProviderConnection>;
44+
45+
/**
46+
* List tool provider connection rows for the given author. Optionally
47+
* narrow by `providerId` and/or `toolkit`. Order is not guaranteed.
48+
*/
49+
abstract listConnectionsByAuthor(
50+
input: StorageListToolProviderConnectionsInput,
51+
): Promise<StorageToolProviderConnection[]>;
52+
53+
/**
54+
* Remove a single tool provider connection row. Idempotent — returns
55+
* silently when the row does not exist.
56+
*/
57+
abstract deleteConnection(input: StorageDeleteToolProviderConnectionInput): Promise<void>;
58+
59+
/**
60+
* Delete every tool provider connection row. Used by tests.
61+
*/
62+
abstract dangerouslyClearAll(): Promise<void>;
63+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
export { ToolProviderConnectionsStorage } from './base';
2+
export { InMemoryToolProviderConnectionsStorage } from './inmemory';

0 commit comments

Comments
 (0)