Add client-side tool execution tracing (#16425)

## Description

Implements end-to-end tracing for client-side tools executed via
`@mastra/client-js`.

When an agent calls a tool that runs in the browser, the server records
a `CLIENT_TOOL_CALL` span under the current agent run and propagates W3C
trace context to the client. The client SDK buffers child spans, logs,
and execution duration while the tool runs, then sends that telemetry
back on the next agent request. The server validates and ingests the
payload through `@mastra/observability`, so configured exporters see the
client-side work in the same trace as the agent run.

### Key changes

**`@mastra/core`**
- Adds `SpanType.CLIENT_TOOL_CALL` and client observability wire types.
- Adds the `ClientObservabilityProxy` interface used by observability
implementations.
- Injects trace carriers into client tool-call chunks and preserves
those carriers through stream output assembly and AI SDK transforms.
- Creates the `CLIENT_TOOL_CALL` span early enough for streaming clients
to receive the carrier, then ends it once tool args are available so the
span shows which args the client tool received.
- Extracts `__mastraObservability` from returned tool-result messages
before model input, forwards it to the proxy, and strips the metadata.
- Adds the `observe` helper to tool execution context with a no-op
default.

**`@mastra/client-js`**
- Adds a browser-safe in-memory observability collector.
- Automatically wraps client tool execution when a server trace carrier
is present.
- Provides `observe.span()` and `observe.log()` inside client tool
`execute(input, context)`.
- Sends buffered OTLP/JSON telemetry back with the tool result.
- Keeps client tool continuation routing correct for `stream`,
`resumeStream`, and `streamUntilIdle` flows.

**`@mastra/observability`**
- Implements `ClientObservabilityProxy.inject()` and `receive()`.
- Validates W3C trace context, trace IDs, parent relationships, payload
size, span count, and log count.
- Maps client OTLP/JSON spans and logs into the existing observability
bus.
- Emits `mastra_tool_duration_ms` with `toolType: "client"` for the
measured browser-side execution duration.

### Architecture notes

Client-side tool execution spans two agent requests:

1. **Server to client:** the agent loop emits a client tool call,
creates a `CLIENT_TOOL_CALL` span, injects a W3C carrier into the
outgoing tool-call chunk, and later ends the span with tool args.
2. **Client execution:** `@mastra/client-js` runs the tool with an
`observe` helper, buffering child spans, logs, and duration.
3. **Client to server:** the SDK attaches `__mastraObservability` to the
tool-result message. The server extracts and strips that metadata before
model input, then ingests it through the observability proxy.

If `@mastra/observability` is not installed or no carrier is present,
client-side tool tracing degrades to a no-op.

## Related issue(s)

Fixes #10889

## Type of change

- [x] New feature (non-breaking change that adds functionality)

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have addressed all Coderabbit comments on this PR

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Anurag Ojha <aojharaj2004@gmail.com>
Co-authored-by: Anurag Ojha <160232626+intojhanurag@users.noreply.github.com>

Author: 3 people

.changeset/client-tool-tracing-client-js.md

@@ -0,0 +1,14 @@
+---
+'@mastra/client-js': minor
+---
+
+Client-side tool tracing is now built in. When server-side observability is configured, the SDK automatically measures execution duration and ships it back to the server. To add child spans and structured logs from inside your tool's `execute(input, context)` function, use the `observe` helper on the execution context:
+
+```ts
+execute: async ({ userId }, { observe }) => {
+  observe.log('info', 'fetching user', { userId })
+  return observe.span('fetch user', () => fetch(`/api/users/${userId}`))
+}
+```
+
+The `createTool()` helper now calls `execute(input, context)` so client tools receive the same execution context shape as core tools.

.changeset/client-tool-tracing-core.md

@@ -0,0 +1,14 @@
+---
+'@mastra/core': minor
+---
+
+Client-side tools now appear in your traces when observability is configured. When an agent calls a tool that executes in the browser via `@mastra/client-js`, a `CLIENT_TOOL_CALL` span is recorded on the server trace so you can see which client tools were invoked, what arguments they received, and how they relate to the rest of the agent run.
+
+Tools also gain an `observe` helper on their execution context for recording child spans and logs from inside `execute`:
+
+```ts
+execute: async ({ userId }, { observe }) => {
+  observe.log('info', 'fetching user', { userId })
+  return observe.span('fetch user', () => fetch(`/api/users/${userId}`))
+}
+```

.changeset/client-tool-tracing-observability.md

@@ -0,0 +1,5 @@
+---
+'@mastra/observability': minor
+---
+
+Support ingesting client-side tool telemetry. Spans, logs, and duration metrics captured by the client SDK during tool execution are forwarded through the observability bus to your existing exporters. Client tool durations are reported via the existing `mastra_tool_duration_ms` metric with a `toolType: 'client'` label to distinguish them from server-side tool durations.

client-sdks/client-js/src/index.ts

@@ -16,6 +16,9 @@ export type {
   VerifyAgentCardSignatureOptions,
 } from './resources/a2a';
 export { RequestContext } from '@mastra/core/request-context';
+// ObservabilityCollector type is available for power users but most
+// users interact via `observe` on the tool execution context.
+export type { ObservabilityCollector } from './observability/types';
 export type { UIMessageWithMetadata } from '@mastra/core/agent';
 export type {
   Body,

client-sdks/client-js/src/observability/collector.test.ts

@@ -0,0 +1,256 @@
+import { describe, expect, it } from 'vitest';
+
+import { createObservabilityCollector, getCurrentObservabilityCollector } from './collector';
+
+const TRACE_ID = '11111111111111111111111111111111';
+const PARENT_SPAN_ID = 'aaaaaaaaaaaaaaaa';
+
+function makeCarrier() {
+  return { traceparent: `00-${TRACE_ID}-${PARENT_SPAN_ID}-01` };
+}
+
+interface OtlpSpan {
+  traceId: string;
+  spanId: string;
+  parentSpanId: string;
+  name: string;
+  startTimeUnixNano: string;
+  endTimeUnixNano: string;
+  status: { code: number; message?: string };
+  attributes: unknown[];
+}
+
+interface OtlpLogRecord {
+  traceId: string;
+  spanId: string;
+  severityText: string;
+  body: { stringValue: string };
+  attributes: unknown[];
+}
+
+function flushSpans(payload: ReturnType<ReturnType<typeof createObservabilityCollector>['flush']>) {
+  const spans = (payload.spans as { resourceSpans: { scopeSpans: { spans: OtlpSpan[] }[] }[] } | undefined)
+    ?.resourceSpans?.[0]?.scopeSpans?.[0]?.spans;
+  return spans ?? [];
+}
+
+function flushLogs(payload: ReturnType<ReturnType<typeof createObservabilityCollector>['flush']>) {
+  const logs = (payload.logs as { resourceLogs: { scopeLogs: { logRecords: OtlpLogRecord[] }[] }[] } | undefined)
+    ?.resourceLogs?.[0]?.scopeLogs?.[0]?.logRecords;
+  return logs ?? [];
+}
+
+describe('ObservabilityCollector', () => {
+  it('exposes the original parentContext on the collector', () => {
+    const carrier = makeCarrier();
+    const collector = createObservabilityCollector(carrier);
+    expect(collector.parentContext).toBe(carrier);
+  });
+
+  it('captures a single span parented under the carrier spanId', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await collector.span('inner work', async () => 42);
+    });
+    const spans = flushSpans(collector.flush());
+    expect(spans).toHaveLength(1);
+    expect(spans[0]!.traceId).toBe(TRACE_ID);
+    expect(spans[0]!.parentSpanId).toBe(PARENT_SPAN_ID);
+    expect(spans[0]!.name).toBe('inner work');
+    expect(spans[0]!.status.code).toBe(1);
+  });
+
+  it('nests spans correctly when called recursively', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await collector.span('outer', async () => {
+        await collector.span('inner', async () => 'ok');
+      });
+    });
+    const spans = flushSpans(collector.flush());
+    expect(spans).toHaveLength(2);
+    const outer = spans.find(s => s.name === 'outer')!;
+    const inner = spans.find(s => s.name === 'inner')!;
+    expect(outer.parentSpanId).toBe(PARENT_SPAN_ID);
+    expect(inner.parentSpanId).toBe(outer.spanId);
+  });
+
+  it('records error status when the wrapped function throws', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await expect(
+        collector.span('failing', async () => {
+          throw new Error('boom');
+        }),
+      ).rejects.toThrow('boom');
+    });
+    const spans = flushSpans(collector.flush());
+    expect(spans).toHaveLength(1);
+    expect(spans[0]!.status.code).toBe(2);
+    expect(spans[0]!.status.message).toBe('boom');
+  });
+
+  it('serializes attributes as OTLP tagged values', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await collector.span('with attrs', async () => null, {
+        'http.method': 'GET',
+        'http.status_code': 200,
+        'cache.hit': true,
+        'request.duration_ms': 12.5,
+      });
+    });
+    const spans = flushSpans(collector.flush());
+    const attrs = spans[0]!.attributes as Array<{ key: string; value: Record<string, unknown> }>;
+    const byKey: Record<string, unknown> = {};
+    for (const a of attrs) byKey[a.key] = a.value;
+    expect(byKey['http.method']).toEqual({ stringValue: 'GET' });
+    expect(byKey['http.status_code']).toEqual({ intValue: 200 });
+    expect(byKey['cache.hit']).toEqual({ boolValue: true });
+    expect(byKey['request.duration_ms']).toEqual({ doubleValue: 12.5 });
+  });
+
+  it('captures logs against the active span', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      collector.log('info', 'before span');
+      await collector.span('work', async () => {
+        collector.log('warn', 'inside span', { extra: 'context' });
+      });
+    });
+    const logs = flushLogs(collector.flush());
+    expect(logs).toHaveLength(2);
+    expect(logs[0]!.spanId).toBe(PARENT_SPAN_ID);
+    expect(logs[0]!.severityText).toBe('INFO');
+    expect(logs[0]!.body.stringValue).toBe('before span');
+    expect(logs[1]!.severityText).toBe('WARN');
+    // The inner log should be parented under the work span, not the carrier.
+    expect(logs[1]!.spanId).not.toBe(PARENT_SPAN_ID);
+  });
+
+  it('measures wall-clock execution duration in withContext', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await new Promise(resolve => setTimeout(resolve, 25));
+    });
+    const payload = collector.flush();
+    expect(payload.executionDurationMs).toBeDefined();
+    // Allow some slack for slow CI; the floor is what matters.
+    expect(payload.executionDurationMs!).toBeGreaterThanOrEqual(20);
+  });
+
+  it('does not include duration when withContext was never called', () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    const payload = collector.flush();
+    expect(payload.executionDurationMs).toBeUndefined();
+  });
+
+  it('flush() returns empty payload after first call', async () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    await collector.withContext(async () => {
+      await collector.span('once', async () => null);
+    });
+    const first = collector.flush();
+    expect(flushSpans(first)).toHaveLength(1);
+    const second = collector.flush();
+    expect(second).toEqual({});
+  });
+
+  it('returns empty payload when no spans or logs were captured', () => {
+    const collector = createObservabilityCollector(makeCarrier());
+    expect(collector.flush()).toEqual({});
+  });
+
+  describe('getCurrentObservabilityCollector', () => {
+    it('returns undefined outside withContext', () => {
+      expect(getCurrentObservabilityCollector()).toBeUndefined();
+    });
+
+    it('returns the active collector inside withContext', async () => {
+      const collector = createObservabilityCollector(makeCarrier());
+      let observed: ReturnType<typeof getCurrentObservabilityCollector> = undefined;
+      await collector.withContext(async () => {
+        observed = getCurrentObservabilityCollector();
+      });
+      expect(observed).toBe(collector);
+      // Cleared after withContext returns.
+      expect(getCurrentObservabilityCollector()).toBeUndefined();
+    });
+
+    it('restores the previous collector after a nested withContext', async () => {
+      const outer = createObservabilityCollector(makeCarrier());
+      const inner = createObservabilityCollector(makeCarrier());
+      await outer.withContext(async () => {
+        expect(getCurrentObservabilityCollector()).toBe(outer);
+        await inner.withContext(async () => {
+          expect(getCurrentObservabilityCollector()).toBe(inner);
+        });
+        expect(getCurrentObservabilityCollector()).toBe(outer);
+      });
+      expect(getCurrentObservabilityCollector()).toBeUndefined();
+    });
+
+    it('isolates overlapping async spans through explicit collector instances', async () => {
+      const first = createObservabilityCollector(makeCarrier());
+      const second = createObservabilityCollector(makeCarrier());
+      let releaseFirst!: () => void;
+      let releaseSecond!: () => void;
+
+      const firstRun = first.withContext(async () => {
+        await first.span('first span', async () => {
+          await new Promise<void>(resolve => {
+            releaseFirst = resolve;
+          });
+        });
+      });
+
+      const secondRun = second.withContext(async () => {
+        await second.span('second span', async () => {
+          await new Promise<void>(resolve => {
+            releaseSecond = resolve;
+          });
+        });
+      });
+
+      releaseFirst();
+      await firstRun;
+
+      releaseSecond();
+      await secondRun;
+      expect(getCurrentObservabilityCollector()).toBeUndefined();
+
+      expect(flushSpans(first.flush()).map(span => span.name)).toEqual(['first span']);
+      expect(flushSpans(second.flush()).map(span => span.name)).toEqual(['second span']);
+    });
+  });
+
+  it('degrades to a synthetic root when traceparent is malformed', async () => {
+    const collector = createObservabilityCollector({ traceparent: 'not-a-traceparent' });
+    await collector.withContext(async () => {
+      await collector.span('orphan', async () => null);
+    });
+    const spans = flushSpans(collector.flush());
+    expect(spans).toHaveLength(1);
+    // Synthetic IDs are all zeros so the server-side ingest can detect
+    // and reject these (it validates traceId match against the actual
+    // parentContext).
+    expect(spans[0]!.traceId).toBe('00000000000000000000000000000000');
+  });
+
+  it('degrades to a synthetic root when traceparent components are W3C-invalid', async () => {
+    for (const traceparent of [
+      `ff-${TRACE_ID}-${PARENT_SPAN_ID}-01`,
+      `00-00000000000000000000000000000000-${PARENT_SPAN_ID}-01`,
+      `00-${TRACE_ID}-0000000000000000-01`,
+    ]) {
+      const collector = createObservabilityCollector({ traceparent });
+      await collector.withContext(async () => {
+        await collector.span('orphan', async () => null);
+      });
+      const spans = flushSpans(collector.flush());
+      expect(spans).toHaveLength(1);
+      expect(spans[0]!.traceId).toBe('00000000000000000000000000000000');
+      expect(spans[0]!.parentSpanId).toBe('0000000000000000');
+    }
+  });
+});