Commit 0a0aa94
feat(browser): add @mastra/browser-viewer package with Playwright-based CLI provider (#15415)
## Summary
Adds package for Playwright-based browser management with CLI providers
(agent-browser, browser-use, browse-cli), enabling screencast
visualization in Studio.
## What's New
### @mastra/browser-viewer (new package)
- **`BrowserViewer`** class that launches Chrome via Playwright with
`--remote-debugging-port`
- Thread-isolated browser sessions with automatic lifecycle management
- CDP URL exposure for CLI tools to connect as secondary clients
- Screencast support via direct page-level CDP sessions
### @mastra/core changes
- **`Workspace.browser`** configuration for CLI provider integration
- **`BrowserCliHandler`** for automatic CDP URL injection into browser
CLI commands
- **`MastraBrowser.providerType`** property (`'sdk'` | `'cli'`) and
`connectToExternalCdp()` method
- Support for external CDP connections when agents provide their own
browser endpoints
- Fixed CWD path duplication in LocalProcessManager
## How It Works
```typescript
import { BrowserViewer } from '@mastra/browser-viewer';
const workspace = new Workspace({
sandbox: new LocalSandbox({ cwd: './workspace' }),
browser: new BrowserViewer({ headless: false }),
});
```
When an agent uses a browser CLI command:
1. `BrowserViewer` lazily launches Chrome with remote debugging enabled
2. `BrowserCliHandler` automatically injects the CDP URL into CLI
commands
3. CLI tools connect to the existing browser as secondary CDP clients
4. Screencast streams from the browser to Studio
**Supported CLIs:**
- `agent-browser` — via `--cdp <port>` and warmup `connect` command
- `browser-use` — via `--cdp-url <url>`
- `browse` — via `--ws <url>`
**External CDP support:** If an agent provides their own CDP URL (e.g.,
browser-use cloud), the system detects it, skips injection, and connects
`BrowserViewer` to the external browser for screencast.
## Testing Notes
- **Local sandbox only** — Remote sandbox support not yet implemented
(would need different CDP discovery)
- Browser CLIs must be installed separately (`npm i -g agent-browser`,
`pipx install browser-use`, etc.)
## Related
- Builds on SDK providers from #14938 (merged)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
ELI5: This PR adds a small tool that can launch or connect to Chrome and
share a WebSocket CDP URL so CLI browser tools can talk to it—letting
Studio show live browser screencasts and interact with those CLIs. It
also wires core workspace/agent logic so CLI browser commands get
auto-configured, warmed up, and lifecycle-managed.
- New package: @mastra/browser-viewer
- Playwright-based BrowserViewer that can launch Chrome (with
--remote-debugging-port) or connect to an existing CDP endpoint.
- Thread-isolated or shared browser sessions via
BrowserViewerThreadManager; exposes getCdpUrl(threadId?) for external
CLI clients.
- Screencast support using per-page CDP sessions with reconnection
handling, URL update events, and input injection (mouse/keyboard)
through CDP.
- Public API exports: BrowserViewer, BrowserViewerThreadManager,
BrowserViewerConfig, CLIProvider, and thread-manager config type.
- README, build/test/lint configs, package.json, ESLint config and
changeset for initial release.
- Core integrations (packages/core)
- Workspace.browser: optional MastraBrowser provider (must be
providerType: 'cli'); browser closed on workspace.destroy; browser not
auto-launched on init.
- MastraBrowser additions:
- providerType: 'sdk' | 'cli' (default 'sdk').
- getCdpUrl(threadId?): string | null.
- launch(threadId?) and isBrowserRunning(threadId?) support thread-aware
lifecycles.
- connectToExternalCdp(cdpUrl, threadId?) hook (base throws by default;
BrowserViewer implements).
- Screencast gating now checks thread-aware running state and
thread-session presence.
- Browser CLI handling:
- New BrowserCliHandler (singleton browserCliHandler) with CLI
detection, external-CDP extraction, CDP injection into chained shell
commands, warmup command generation, and warmup lifecycle cleanup tied
to browser close.
- Shell utilities: splitShellCommand and reassembleShellCommand for
quote-aware command splitting/rewriting, plus tests.
- execute-command flow:
- Commands are analyzed for browser CLIs before sandbox execution.
- If workspace.browser exists and the CLI is internal (no external CDP),
launch browser per-thread as needed, obtain cdpUrl, run warmup commands
(best-effort), inject CDP URL and session flags into the command, and
register warmup cleanup.
- If an external CDP URL is detected, attempt
browser.connectToExternalCdp(externalCdpUrl, threadId) (errors ignored),
and skip injection.
- Agent/browser interactions:
- Agent construction now validates explicit config.browser must be
providerType === 'sdk' (otherwise AGENT_INVALID_BROWSER_PROVIDER).
- Agent resolution favors explicitly configured browser;
workspace-derived browser is applied unless explicit.
- Per-request browser context injection now includes providerType and,
for CLI providers, cdpUrl when the thread is running.
- Agents without browsers can use a workspace-provided browser when
appropriate.
- ThreadManager visibility tweak: onSessionCreated/onSessionDestroyed
changed to protected for subclass use.
- Browser-viewer internals
- Thread manager supports owned browserServer launches
(chromium.launchServer), discovers DevToolsActivePort to build CDP URL,
connects via Playwright (connectOverCDP) for external endpoints, creates
contexts/pages, caches per-page CDP sessions, wires disconnect handlers,
and supports clean closeAll.
- BrowserViewer implements startScreencast to provide a stream backed by
per-page CDP sessions and handles reconnects on navigation/new
page/close.
- Input injection routes Input.dispatchMouseEvent/Input.dispatchKeyEvent
via per-thread CDP sessions.
- Commits / runtime fixes and refinements
- Fixes to active-tab logic and caching of CDP sessions for input
injection to reduce overhead.
- Dedicated screencast CDP session creation and improved external CDP
detection (including --cdp <port> patterns).
- Explicit browser unset/ disconnect detection (Target.targetDestroyed)
and propagation of browser close events from ThreadManager to
ViewerRegistry.
- ViewerRegistry enhanced to explicitly stop streams and broadcast
browser_closed on screencast errors.
- Bug fixes
- LocalProcessManager cwd duplication fixed: relative cwd resolution
avoids duplicating workspace path when already workspace-relative.
- ViewerRegistry now broadcasts browser_closed on screencast stream
errors so UI won't remain 'Live' after errors.
- Tests
- New and extended tests:
- BrowserCliHandler end-to-end suite (detection, extraction, injection,
warmup lifecycle).
- Utilities tests for shell splitting and reassembly.
- execute-command tests validating browser launch, CDP injection, warmup
behavior, and external CDP connect handling.
- Adjusted mocks/serialization to include providerType where needed.
- Docs & release
- README for @mastra/browser-viewer and changesets added for
@mastra/browser-viewer and @mastra/core; release notes document initial
package and core CLI-browser integration.
- Operational notes: local sandbox only; browser CLIs must be installed
separately (e.g., agent-browser, browser-use, browse). Remote
sandbox/browser-hosting not included in this PR.
- Scope & reviewer notes
- Large new feature surface: major review focus on browser-viewer thread
manager/browser-viewer files and core/browser CLI-handling integration.
Backwards compatibility preserved: MastraBrowser defaults remain
SDK-type and workspace.browser validation prevents misconfiguration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mastra Code (anthropic/claude-opus-4-5) <noreply@mastra.ai>1 parent 2a87046 commit 0a0aa94
31 files changed
Lines changed: 2999 additions & 24 deletions
File tree
- .changeset
- browser
- agent-browser
- browser-viewer
- src
- stagehand
- packages
- core/src
- agent
- __tests__
- browser
- __tests__
- workspace
- sandbox
- tools
- __tests__
- server/src/server
- browser-stream
- handlers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
6 | 5 | | |
7 | 6 | | |
8 | 7 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
0 commit comments