Skip to content

Commit 0a0aa94

Browse files
NikAiyerMastra Code (anthropic/claude-opus-4-5)
andauthored
feat(browser): add @mastra/browser-viewer package with Playwright-based CLI provider (#15415)
## Summary Adds package for Playwright-based browser management with CLI providers (agent-browser, browser-use, browse-cli), enabling screencast visualization in Studio. ## What's New ### @mastra/browser-viewer (new package) - **`BrowserViewer`** class that launches Chrome via Playwright with `--remote-debugging-port` - Thread-isolated browser sessions with automatic lifecycle management - CDP URL exposure for CLI tools to connect as secondary clients - Screencast support via direct page-level CDP sessions ### @mastra/core changes - **`Workspace.browser`** configuration for CLI provider integration - **`BrowserCliHandler`** for automatic CDP URL injection into browser CLI commands - **`MastraBrowser.providerType`** property (`'sdk'` | `'cli'`) and `connectToExternalCdp()` method - Support for external CDP connections when agents provide their own browser endpoints - Fixed CWD path duplication in LocalProcessManager ## How It Works ```typescript import { BrowserViewer } from '@mastra/browser-viewer'; const workspace = new Workspace({ sandbox: new LocalSandbox({ cwd: './workspace' }), browser: new BrowserViewer({ headless: false }), }); ``` When an agent uses a browser CLI command: 1. `BrowserViewer` lazily launches Chrome with remote debugging enabled 2. `BrowserCliHandler` automatically injects the CDP URL into CLI commands 3. CLI tools connect to the existing browser as secondary CDP clients 4. Screencast streams from the browser to Studio **Supported CLIs:** - `agent-browser` — via `--cdp <port>` and warmup `connect` command - `browser-use` — via `--cdp-url <url>` - `browse` — via `--ws <url>` **External CDP support:** If an agent provides their own CDP URL (e.g., browser-use cloud), the system detects it, skips injection, and connects `BrowserViewer` to the external browser for screencast. ## Testing Notes - **Local sandbox only** — Remote sandbox support not yet implemented (would need different CDP discovery) - Browser CLIs must be installed separately (`npm i -g agent-browser`, `pipx install browser-use`, etc.) ## Related - Builds on SDK providers from #14938 (merged) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ELI5: This PR adds a small tool that can launch or connect to Chrome and share a WebSocket CDP URL so CLI browser tools can talk to it—letting Studio show live browser screencasts and interact with those CLIs. It also wires core workspace/agent logic so CLI browser commands get auto-configured, warmed up, and lifecycle-managed. - New package: @mastra/browser-viewer - Playwright-based BrowserViewer that can launch Chrome (with --remote-debugging-port) or connect to an existing CDP endpoint. - Thread-isolated or shared browser sessions via BrowserViewerThreadManager; exposes getCdpUrl(threadId?) for external CLI clients. - Screencast support using per-page CDP sessions with reconnection handling, URL update events, and input injection (mouse/keyboard) through CDP. - Public API exports: BrowserViewer, BrowserViewerThreadManager, BrowserViewerConfig, CLIProvider, and thread-manager config type. - README, build/test/lint configs, package.json, ESLint config and changeset for initial release. - Core integrations (packages/core) - Workspace.browser: optional MastraBrowser provider (must be providerType: 'cli'); browser closed on workspace.destroy; browser not auto-launched on init. - MastraBrowser additions: - providerType: 'sdk' | 'cli' (default 'sdk'). - getCdpUrl(threadId?): string | null. - launch(threadId?) and isBrowserRunning(threadId?) support thread-aware lifecycles. - connectToExternalCdp(cdpUrl, threadId?) hook (base throws by default; BrowserViewer implements). - Screencast gating now checks thread-aware running state and thread-session presence. - Browser CLI handling: - New BrowserCliHandler (singleton browserCliHandler) with CLI detection, external-CDP extraction, CDP injection into chained shell commands, warmup command generation, and warmup lifecycle cleanup tied to browser close. - Shell utilities: splitShellCommand and reassembleShellCommand for quote-aware command splitting/rewriting, plus tests. - execute-command flow: - Commands are analyzed for browser CLIs before sandbox execution. - If workspace.browser exists and the CLI is internal (no external CDP), launch browser per-thread as needed, obtain cdpUrl, run warmup commands (best-effort), inject CDP URL and session flags into the command, and register warmup cleanup. - If an external CDP URL is detected, attempt browser.connectToExternalCdp(externalCdpUrl, threadId) (errors ignored), and skip injection. - Agent/browser interactions: - Agent construction now validates explicit config.browser must be providerType === 'sdk' (otherwise AGENT_INVALID_BROWSER_PROVIDER). - Agent resolution favors explicitly configured browser; workspace-derived browser is applied unless explicit. - Per-request browser context injection now includes providerType and, for CLI providers, cdpUrl when the thread is running. - Agents without browsers can use a workspace-provided browser when appropriate. - ThreadManager visibility tweak: onSessionCreated/onSessionDestroyed changed to protected for subclass use. - Browser-viewer internals - Thread manager supports owned browserServer launches (chromium.launchServer), discovers DevToolsActivePort to build CDP URL, connects via Playwright (connectOverCDP) for external endpoints, creates contexts/pages, caches per-page CDP sessions, wires disconnect handlers, and supports clean closeAll. - BrowserViewer implements startScreencast to provide a stream backed by per-page CDP sessions and handles reconnects on navigation/new page/close. - Input injection routes Input.dispatchMouseEvent/Input.dispatchKeyEvent via per-thread CDP sessions. - Commits / runtime fixes and refinements - Fixes to active-tab logic and caching of CDP sessions for input injection to reduce overhead. - Dedicated screencast CDP session creation and improved external CDP detection (including --cdp <port> patterns). - Explicit browser unset/ disconnect detection (Target.targetDestroyed) and propagation of browser close events from ThreadManager to ViewerRegistry. - ViewerRegistry enhanced to explicitly stop streams and broadcast browser_closed on screencast errors. - Bug fixes - LocalProcessManager cwd duplication fixed: relative cwd resolution avoids duplicating workspace path when already workspace-relative. - ViewerRegistry now broadcasts browser_closed on screencast stream errors so UI won't remain 'Live' after errors. - Tests - New and extended tests: - BrowserCliHandler end-to-end suite (detection, extraction, injection, warmup lifecycle). - Utilities tests for shell splitting and reassembly. - execute-command tests validating browser launch, CDP injection, warmup behavior, and external CDP connect handling. - Adjusted mocks/serialization to include providerType where needed. - Docs & release - README for @mastra/browser-viewer and changesets added for @mastra/browser-viewer and @mastra/core; release notes document initial package and core CLI-browser integration. - Operational notes: local sandbox only; browser CLIs must be installed separately (e.g., agent-browser, browser-use, browse). Remote sandbox/browser-hosting not included in this PR. - Scope & reviewer notes - Large new feature surface: major review focus on browser-viewer thread manager/browser-viewer files and core/browser CLI-handling integration. Backwards compatibility preserved: MastraBrowser defaults remain SDK-type and workspace.browser validation prevents misconfiguration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Mastra Code (anthropic/claude-opus-4-5) <noreply@mastra.ai>
1 parent 2a87046 commit 0a0aa94

31 files changed

Lines changed: 2999 additions & 24 deletions

.changeset/bright-windows-glow.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
'@mastra/browser-viewer': minor
3+
---
4+
5+
Initial release of @mastra/browser-viewer
6+
7+
Playwright-based browser viewer for CLI providers that enables screencast visualization in Studio. Supports thread-isolated browser sessions and automatic CDP connection management.
8+
9+
```typescript
10+
import { BrowserViewer } from '@mastra/browser-viewer';
11+
12+
const workspace = new Workspace({
13+
sandbox: new LocalSandbox({ cwd: './workspace' }),
14+
browser: new BrowserViewer({
15+
cli: 'agent-browser',
16+
headless: false,
17+
}),
18+
});
19+
```

.changeset/old-guests-hammer.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
'@mastra/core': minor
3+
---
4+
5+
Added support for CLI-driven browser automation with screencast support in `@mastra/core`, including automatic CDP injection for browser CLIs.
6+
7+
Fixed local process spawning so workspace-relative `cwd` values no longer get duplicated.

.changeset/warm-dots-flow.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@mastra/server': patch
3+
---
4+
5+
Fixed screencast panel staying "Live" after browser closes due to an error. The `ViewerRegistry` now broadcasts `browser_closed` status when a screencast stream emits an error, not just when it stops cleanly.

browser/agent-browser/tsconfig.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
"extends": "../../tsconfig.node.json",
33
"compilerOptions": {
44
"outDir": "./dist",
5-
"rootDir": "./src",
65
"moduleResolution": "bundler",
76
"types": ["node"]
87
},

browser/browser-viewer/README.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# @mastra/browser-viewer
2+
3+
Playwright-based browser viewer for Mastra workspaces with CLI provider support.
4+
5+
## Overview
6+
7+
`@mastra/browser-viewer` provides `BrowserViewer`, which launches Chrome via Playwright and exposes the CDP URL for CLI tools (agent-browser, browser-use, browse-cli) to connect. This gives you:
8+
9+
- **Full screencast support** — Direct page-level CDP sessions
10+
- **Input injection** — Mouse and keyboard events work correctly
11+
- **Browser lifecycle control** — Browser starts/stops with the server
12+
- **CLI flexibility** — Agent uses skills + workspace commands to drive any CLI
13+
14+
## Installation
15+
16+
```bash
17+
npm install @mastra/browser-viewer
18+
# or
19+
pnpm add @mastra/browser-viewer
20+
```
21+
22+
## Usage
23+
24+
### Basic Setup
25+
26+
```typescript
27+
import { BrowserViewer } from '@mastra/browser-viewer';
28+
29+
const viewer = new BrowserViewer({
30+
cli: 'agent-browser', // Which CLI the agent will use
31+
headless: false, // Show browser window
32+
});
33+
34+
// Launch browser
35+
await viewer.launch();
36+
37+
// Get CDP URL for CLIs to connect
38+
const cdpUrl = await viewer.getCdpUrl();
39+
console.log(cdpUrl); // ws://127.0.0.1:9222/devtools/browser/...
40+
```
41+
42+
### Connect to Existing Browser
43+
44+
```typescript
45+
import { BrowserViewer } from '@mastra/browser-viewer';
46+
47+
const viewer = new BrowserViewer({
48+
cli: 'agent-browser',
49+
cdpUrl: 'ws://127.0.0.1:9222/devtools/browser/abc123',
50+
});
51+
```
52+
53+
### With Workspace
54+
55+
The CDP URL is automatically injected into CLI commands when used with workspace tools.
56+
57+
```typescript
58+
import { Workspace, LocalSandbox } from '@mastra/core';
59+
import { BrowserViewer } from '@mastra/browser-viewer';
60+
61+
const workspace = new Workspace({
62+
sandbox: new LocalSandbox({ cwd: './workspace' }),
63+
browser: new BrowserViewer({
64+
cli: 'agent-browser',
65+
headless: false,
66+
}),
67+
});
68+
69+
// When agent runs: agent-browser open https://google.com
70+
// Mastra auto-injects the CDP connection so CLI uses Mastra's browser
71+
```
72+
73+
## Configuration
74+
75+
| Option | Type | Default | Description |
76+
| ---------------- | -------------------------------------------------- | ---------- | ------------------------------------------------ |
77+
| `cli` | `'agent-browser' \| 'browser-use' \| 'browse-cli'` | Required | Which CLI the agent uses |
78+
| `cdpUrl` | `string` | - | Connect to existing browser instead of launching |
79+
| `headless` | `boolean` | `false` | Run browser in headless mode |
80+
| `cdpPort` | `number` | `0` (auto) | Port for Chrome remote debugging |
81+
| `viewport` | `{ width, height }` | `1280x720` | Browser viewport size |
82+
| `userDataDir` | `string` | - | Chrome profile directory |
83+
| `executablePath` | `string` | - | Path to Chrome executable |
84+
85+
## How It Works
86+
87+
1. **BrowserViewer launches Chrome** via Playwright with `--remote-debugging-port`
88+
2. **Agent calls CLI commands** via `workspace_execute_command`
89+
3. **CDP URL is auto-injected** so CLI connects to Mastra-managed Chrome
90+
4. **Screencast streams** directly from page-level CDP sessions
91+
5. **Browser closes** when server exits
92+
93+
## Supported CLIs
94+
95+
- **agent-browser** — Vercel's browser automation CLI (`--cdp <port>`)
96+
- **browser-use** — Python-based browser automation (`--cdp-url <url>`)
97+
- **browse-cli** — Browserbase's Stagehand CLI (`--ws <url>`)
98+
99+
## License
100+
101+
Apache-2.0
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
import { createConfig } from '@internal/lint/eslint';
2+
3+
const config = await createConfig();
4+
5+
/** @type {import("eslint").Linter.Config[]} */
6+
export default [...config];
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
{
2+
"name": "@mastra/browser-viewer",
3+
"version": "0.0.0",
4+
"description": "Playwright-based browser viewer for Mastra CLI providers",
5+
"type": "module",
6+
"files": [
7+
"dist",
8+
"CHANGELOG.md"
9+
],
10+
"main": "dist/index.js",
11+
"types": "dist/index.d.ts",
12+
"exports": {
13+
".": {
14+
"import": {
15+
"types": "./dist/index.d.ts",
16+
"default": "./dist/index.js"
17+
},
18+
"require": {
19+
"types": "./dist/index.d.ts",
20+
"default": "./dist/index.cjs"
21+
}
22+
},
23+
"./package.json": "./package.json"
24+
},
25+
"scripts": {
26+
"build": "tsup --silent --config tsup.config.ts",
27+
"build:watch": "tsup --watch --silent --config tsup.config.ts",
28+
"test": "vitest run",
29+
"lint": "eslint ."
30+
},
31+
"license": "Apache-2.0",
32+
"dependencies": {
33+
"playwright-core": "^1.52.0",
34+
"typed-emitter": "^2.1.0"
35+
},
36+
"devDependencies": {
37+
"@internal/lint": "workspace:*",
38+
"@internal/types-builder": "workspace:*",
39+
"@mastra/core": "workspace:*",
40+
"@types/node": "22.15.21",
41+
"@vitest/coverage-v8": "catalog:",
42+
"@vitest/ui": "catalog:",
43+
"eslint": "^9.27.0",
44+
"tsup": "^8.5.0",
45+
"typescript": "catalog:",
46+
"vitest": "catalog:",
47+
"zod": "catalog:"
48+
},
49+
"peerDependencies": {
50+
"@mastra/core": ">=1.26.0-0 <2.0.0-0",
51+
"zod": "^3.25.0 || ^4.0.0"
52+
},
53+
"homepage": "https://mastra.ai",
54+
"repository": {
55+
"type": "git",
56+
"url": "git+https://github.com/mastra-ai/mastra.git",
57+
"directory": "browser/browser-viewer"
58+
},
59+
"bugs": {
60+
"url": "https://github.com/mastra-ai/mastra/issues"
61+
},
62+
"engines": {
63+
"node": ">=22.13.0"
64+
}
65+
}

0 commit comments

Comments
 (0)