Add scorer tracing and export scores through the observability bus#14920
Merged
Conversation
🦋 Changeset detectedLatest commit: 86bd798 The changes in this PR will be included in the next version bump. This PR includes changesets to release 22 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds scorer observability: introduces SCORER_RUN and SCORER_STEP spans, creates/uses these spans during scorer execution, conditionally exports scores via observability.addScore() when anchored to a trace, extends score types/schemas with scorerName and targetEntityType, and marks legacy scores-store helper as deprecated while preserving legacy writes. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
epinzur
force-pushed
the
esp/new_score_storage
branch
from
April 1, 2026 11:07
cde5dd8 to
19cf6cd
Compare
greglobinski
approved these changes
Apr 1, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/core/src/mastra/hooks.ts (1)
99-99:⚠️ Potential issue | 🟡 MinorUse scorer display name for
scorerNamein exporter payload.
scorerNameis currently populated withscorer.id, which can produce misleading metadata when ID and name diverge.Proposed fix
- scorerName: scorerToUse.scorer.id, + scorerName: scorerToUse.scorer.name ?? scorerToUse.scorer.id,🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/mastra/hooks.ts` at line 99, The exporter payload is setting scorerName to scorerToUse.scorer.id which can be misleading; change the assignment for scorerName to use the scorer's display name (scorerToUse.scorer.displayName) and fall back to scorerToUse.scorer.id if the display name is missing or empty so metadata remains accurate; update the code where scorerName is assigned (currently referencing scorerToUse.scorer.id) to use this displayName-with-fallback logic.
🧹 Nitpick comments (1)
packages/core/src/observability/utils.ts (1)
150-164: Consider mapping scorer span types in the fallback path.Given this PR introduces scorer tracing spans, mapping
SCORER_RUN/SCORER_STEPtoEntityType.SCORERwould make fallback entity resolution more complete whenentityTypeis absent.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/observability/utils.ts` around lines 150 - 164, The fallback entity mapping switch over span.spanType currently omits scorer spans; add cases for SpanType.SCORER_RUN and SpanType.SCORER_STEP that return EntityType.SCORER so scorer tracing spans resolve correctly when entityType is absent—update the switch in the function handling span-based entity resolution (the switch on span.spanType in packages/core/src/observability/utils.ts) to include these two new cases mapping to EntityType.SCORER.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.changeset/cozy-waves-teach.md:
- Around line 5-11: The changeset mentions an internal plumbing call
(mastra.observability.addScore()) — rephrase that line to focus on the
observable outcome: state that scorer results are now exported to the
observability bus and available for downstream tooling/monitoring, and keep
references to the new tracing spans (SCORER_RUN, SCORER_STEP) and deprecated
scores-store helper only as implementation notes or in parentheses; replace the
specific method name with outcome-focused wording like "exported scorer results
to the observability bus for consumption by monitoring and analytics" while
preserving mention of added score metadata (scorer name, target entity type,
target scope, and scorer trace links).
In `@packages/core/src/evals/run/index.ts`:
- Line 651: The JSDoc `@deprecated` line currently opens an inline code span for
mastra.observability.addScore() but is missing the closing backtick; update the
JSDoc near the deprecated annotation in the file (the comment containing
"@deprecated Legacy scores-store path. New score emission should use
`mastra.observability.addScore().") to properly close the inline code span by
adding the trailing backtick so it reads `mastra.observability.addScore()`.
In `@packages/core/src/mastra/hooks.ts`:
- Around line 59-68: The current call to scorerToUse.scorer.run uses a spread of
rest and an unsafe "as any" cast; replace that by explicitly building a
ScorerRun-typed input object (use fields like runId: hookData.runId, input,
output, scoreSource: 'live', targetScope: 'span', targetEntityType:
toScorerTargetEntityType(entityType), targetTraceId: traceId, targetSpanId:
spanId) and conditionally include requestContext/additionalContext from hookData
instead of spreading rest, invoke scorerToUse.scorer.run with that object (no
"as any"), assign the result to runResult, then merge runResult into the payload
that previously used rest so types remain strict and correct.
In `@packages/core/src/storage/domains/observability/record-builders.ts`:
- Around line 291-293: The current traceId fallback uses an empty string which
creates synthetic trace buckets; update the traceId assignment in
record-builders.ts (where traceId: s.traceId ?? s.correlationContext?.traceId ??
'') to return null instead of '' (i.e., traceId: s.traceId ??
s.correlationContext?.traceId ?? null), and update the CreateScoreRecord
type/schema to allow traceId: string | null so the persistence and contract
correctly accept nullable trace IDs.
---
Outside diff comments:
In `@packages/core/src/mastra/hooks.ts`:
- Line 99: The exporter payload is setting scorerName to scorerToUse.scorer.id
which can be misleading; change the assignment for scorerName to use the
scorer's display name (scorerToUse.scorer.displayName) and fall back to
scorerToUse.scorer.id if the display name is missing or empty so metadata
remains accurate; update the code where scorerName is assigned (currently
referencing scorerToUse.scorer.id) to use this displayName-with-fallback logic.
---
Nitpick comments:
In `@packages/core/src/observability/utils.ts`:
- Around line 150-164: The fallback entity mapping switch over span.spanType
currently omits scorer spans; add cases for SpanType.SCORER_RUN and
SpanType.SCORER_STEP that return EntityType.SCORER so scorer tracing spans
resolve correctly when entityType is absent—update the switch in the function
handling span-based entity resolution (the switch on span.spanType in
packages/core/src/observability/utils.ts) to include these two new cases mapping
to EntityType.SCORER.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5701a917-3a03-4bcc-b508-848e640fa538
📒 Files selected for processing (24)
.changeset/cozy-waves-teach.md.changeset/lemon-webs-tickle.mdobservability/mastra/src/exporters/test.tsobservability/sentry/src/tracing.tspackages/_internal-core/src/storage/domains/observability/scores.tspackages/_internal-core/src/storage/domains/shared.tspackages/core/src/agent/agent.tspackages/core/src/datasets/experiment/scorer.tspackages/core/src/evals/base.test.tspackages/core/src/evals/base.tspackages/core/src/evals/run/index.tspackages/core/src/evals/scoreTraces/scoreTracesWorkflow.tspackages/core/src/evals/types.tspackages/core/src/mastra/hooks.tspackages/core/src/mastra/index.tspackages/core/src/observability/no-op.tspackages/core/src/observability/types/core.tspackages/core/src/observability/types/feedback.tspackages/core/src/observability/types/scores.tspackages/core/src/observability/types/tracing.tspackages/core/src/observability/utils.tspackages/core/src/storage/domains/observability/base.tspackages/core/src/storage/domains/observability/record-builders.test.tspackages/core/src/storage/domains/observability/record-builders.ts
Comment on lines
+291
to
293
| //TODO: update this and CreateScoreRecord to be able to handle a null traceId | ||
| traceId: s.traceId ?? s.correlationContext?.traceId ?? '', | ||
| spanId: s.spanId ?? s.correlationContext?.spanId ?? null, |
Contributor
There was a problem hiding this comment.
Avoid persisting missing score trace IDs as empty strings.
Using '' as the fallback creates a synthetic trace bucket and can incorrectly correlate unrelated score records. Persist this as null and allow nullable traceId in the score record contract instead.
Proposed fix
- //TODO: update this and CreateScoreRecord to be able to handle a null traceId
- traceId: s.traceId ?? s.correlationContext?.traceId ?? '',
+ traceId: s.traceId ?? s.correlationContext?.traceId ?? null,// Also update the corresponding CreateScoreRecord type/schema:
// traceId: string | null🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/core/src/storage/domains/observability/record-builders.ts` around
lines 291 - 293, The current traceId fallback uses an empty string which creates
synthetic trace buckets; update the traceId assignment in record-builders.ts
(where traceId: s.traceId ?? s.correlationContext?.traceId ?? '') to return null
instead of '' (i.e., traceId: s.traceId ?? s.correlationContext?.traceId ??
null), and update the CreateScoreRecord type/schema to allow traceId: string |
null so the persistence and contract correctly accept nullable trace IDs.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
packages/core/src/evals/types.ts (1)
733-759:⚠️ Potential issue | 🟠 MajorDon’t flatten scorer internals back into the extracted trajectory.
spanToTrajectorySteps()promotes children of every skipped span. ForSCORER_RUN/SCORER_STEP, that means judge-modelMODEL_GENERATIONspans and tool calls get reintroduced as if they were part of the target execution. Those scorer subtrees should be dropped entirely, not flattened.Suggested fix
const SKIPPED_SPAN_TYPES = new Set([ - SpanType.SCORER_RUN, - SpanType.SCORER_STEP, SpanType.GENERIC, SpanType.MODEL_STEP, SpanType.MODEL_CHUNK, SpanType.WORKFLOW_CONDITIONAL_EVAL, ]); @@ function spanToTrajectorySteps(node: SpanTreeNode): TrajectoryStep[] { const { span, children: childNodes } = node; + if (span.spanType === SpanType.SCORER_RUN || span.spanType === SpanType.SCORER_STEP) { + return []; + } + if (SKIPPED_SPAN_TYPES.has(span.spanType)) { // Promote children of skipped spans so their subtree is preserved return childNodes.flatMap(spanToTrajectorySteps); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/evals/types.ts` around lines 733 - 759, spanToTrajectorySteps currently promotes the children of all skipped spans (SKIPPED_SPAN_TYPES), which incorrectly reinserts scorer internals; change the logic in spanToTrajectorySteps so that when span.spanType is SpanType.SCORER_RUN or SpanType.SCORER_STEP the function returns an empty array (drop the entire subtree) instead of flat-mapping children, while keeping the existing promotion behavior for the other SKIPPED_SPAN_TYPES.packages/core/src/mastra/index.ts (1)
966-1050:⚠️ Potential issue | 🟠 MajorPropagate the agent/scorer definition source when auto-registering agent scorers.
addAgent()now accepts anyDefinitionSource, but Line 1044 still registers discovered scorers with{ source: 'code' }. That means a stored or remotely-defined agent can emit scorer spans/scores with the wrongscorerDefinition, which makes the new score provenance inaccurate.Suggested fix
- this.addScorer(entry.scorer, undefined, { source: 'code' }); + this.addScorer(entry.scorer, undefined, { + source: entry.scorer.source ?? mastraAgent.source ?? options?.source ?? 'code', + });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/mastra/index.ts` around lines 966 - 1050, The auto-registered scorer entries in addAgent are being registered with a hardcoded source '{ source: "code" }' which loses the original DefinitionSource; update the addScorer call inside addAgent (the block iterating Object.entries(scorers)) to pass the correct source by propagating the agent-level source: use options?.source ?? mastraAgent.source ?? 'code' (or equivalent) when constructing the third argument to this.addScorer so the scorerDefinition provenance reflects the agent's provided source.packages/core/src/evals/scoreTraces/scoreTracesWorkflow.ts (1)
119-123:⚠️ Potential issue | 🟠 MajorPreserve trace-level targets when
spanIdis omitted.This branch uses the root span as scorer input/output, but the emitted target metadata is still forced to
targetScope: 'span'with atargetSpanId. That reclassifies whole-trace scoring as root-span scoring, so downstream observability records lose the caller’s original scope.Suggested change
+ const isSpanTarget = Boolean(target.spanId); const result = await scorer.run({ ...scorerRun, scoreSource: 'trace', - targetScope: 'span', + targetScope: isSpanTarget ? 'span' : 'trace', targetEntityType: getEntityTypeForSpan(span), targetTraceId: target.traceId, - targetSpanId: span.spanId, + targetSpanId: isSpanTarget ? span.spanId : undefined, });Also applies to: 136-143
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/evals/scoreTraces/scoreTracesWorkflow.ts` around lines 119 - 123, The current logic in scoreTracesWorkflow.ts picks the root span when target.spanId is absent but still forces emitted metadata to targetScope: 'span' and sets targetSpanId, which mislabels trace-level targets; update the code so that when target.spanId is provided you select the span and set emitted metadata to targetScope: 'span' with targetSpanId, but when target.spanId is omitted you may use the root span for scoring (variable span = trace.spans.find(...)) while leaving the emitted metadata as a trace-level target (e.g., targetScope: 'trace' and no targetSpanId). Apply the same change to the other similar branch that builds output metadata (the second block around the other span-selection code) so only explicit span targets become span-scoped in the emitted target object.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/evals/base.ts`:
- Around line 537-554: The SCORER_RUN span (evalSpan) can remain open if
this.toMastraWorkflow() or workflow.createRun() throws; update the code so any
exception during workflow setup (calls to this.toMastraWorkflow() and
workflow.createRun()) catches the error, calls evalSpan?.error({ error: err,
endSpan: true }) or otherwise ends the span, and then rethrows; ensure the
existing catch around executeWithContext remains for start(), but add a
preliminary try/catch around the workflow creation sequence (referencing
toMastraWorkflow, workflow.createRun, and evalSpan) to guarantee the span is
closed on setup failures.
In `@packages/core/src/mastra/hooks.ts`:
- Around line 59-68: The code currently takes the result of
scorerToUse.scorer.run(...) and then publishes the same score directly to
exporters, duplicating live scores; remove the extra exporter fan-out after the
scorer.run() call and any direct calls that push the score to exporters (the
second-publish logic that follows the run) so that scores are only emitted via
the internal scorer hook mechanism (i.e., rely on span.addScore() being invoked
by the scorer implementation). Locate the post-run export logic around
scorerToUse.scorer.run and the similar block referenced in the review (lines
noted as also applying to 88-112) and delete the direct exporter/publish code
while leaving the scorer.run() invocation and normal span handling intact.
---
Outside diff comments:
In `@packages/core/src/evals/scoreTraces/scoreTracesWorkflow.ts`:
- Around line 119-123: The current logic in scoreTracesWorkflow.ts picks the
root span when target.spanId is absent but still forces emitted metadata to
targetScope: 'span' and sets targetSpanId, which mislabels trace-level targets;
update the code so that when target.spanId is provided you select the span and
set emitted metadata to targetScope: 'span' with targetSpanId, but when
target.spanId is omitted you may use the root span for scoring (variable span =
trace.spans.find(...)) while leaving the emitted metadata as a trace-level
target (e.g., targetScope: 'trace' and no targetSpanId). Apply the same change
to the other similar branch that builds output metadata (the second block around
the other span-selection code) so only explicit span targets become span-scoped
in the emitted target object.
In `@packages/core/src/evals/types.ts`:
- Around line 733-759: spanToTrajectorySteps currently promotes the children of
all skipped spans (SKIPPED_SPAN_TYPES), which incorrectly reinserts scorer
internals; change the logic in spanToTrajectorySteps so that when span.spanType
is SpanType.SCORER_RUN or SpanType.SCORER_STEP the function returns an empty
array (drop the entire subtree) instead of flat-mapping children, while keeping
the existing promotion behavior for the other SKIPPED_SPAN_TYPES.
In `@packages/core/src/mastra/index.ts`:
- Around line 966-1050: The auto-registered scorer entries in addAgent are being
registered with a hardcoded source '{ source: "code" }' which loses the original
DefinitionSource; update the addScorer call inside addAgent (the block iterating
Object.entries(scorers)) to pass the correct source by propagating the
agent-level source: use options?.source ?? mastraAgent.source ?? 'code' (or
equivalent) when constructing the third argument to this.addScorer so the
scorerDefinition provenance reflects the agent's provided source.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3e7ba2f4-1180-4509-b29a-76a73bda42a7
📒 Files selected for processing (21)
.changeset/cozy-waves-teach.md.changeset/lemon-webs-tickle.mdobservability/mastra/src/exporters/test.tsobservability/sentry/src/tracing.tspackages/_internal-core/src/storage/domains/observability/scores.tspackages/_internal-core/src/storage/domains/shared.tspackages/core/src/agent/agent.tspackages/core/src/datasets/experiment/scorer.tspackages/core/src/evals/base.test.tspackages/core/src/evals/base.tspackages/core/src/evals/run/index.tspackages/core/src/evals/scoreTraces/scoreTracesWorkflow.tspackages/core/src/evals/types.tspackages/core/src/mastra/hooks.tspackages/core/src/mastra/index.tspackages/core/src/observability/no-op.tspackages/core/src/observability/types/core.tspackages/core/src/observability/types/feedback.tspackages/core/src/observability/types/scores.tspackages/core/src/observability/types/tracing.tspackages/core/src/observability/utils.ts
✅ Files skipped from review due to trivial changes (5)
- .changeset/lemon-webs-tickle.md
- .changeset/cozy-waves-teach.md
- observability/mastra/src/exporters/test.ts
- packages/core/src/observability/types/feedback.ts
- packages/core/src/evals/run/index.ts
🚧 Files skipped from review as they are similar to previous changes (8)
- packages/core/src/observability/no-op.ts
- packages/_internal-core/src/storage/domains/shared.ts
- packages/core/src/observability/utils.ts
- packages/core/src/agent/agent.ts
- packages/_internal-core/src/storage/domains/observability/scores.ts
- packages/core/src/datasets/experiment/scorer.ts
- packages/core/src/evals/base.test.ts
- packages/core/src/observability/types/core.ts
Comment on lines
+59
to
+68
| const runResult = (await scorerToUse.scorer.run({ | ||
| ...rest, | ||
| input, | ||
| output, | ||
| }); | ||
|
|
||
| let spanId; | ||
| let traceId; | ||
| const currentSpan = hookData.tracingContext?.currentSpan; | ||
| if (currentSpan && currentSpan.isValid) { | ||
| spanId = currentSpan.id; | ||
| traceId = currentSpan.traceId; | ||
| } | ||
| scoreSource: 'live', | ||
| targetScope: 'span', | ||
| targetEntityType: toScorerTargetEntityType(entityType), | ||
| targetTraceId: traceId, | ||
| targetSpanId: spanId, | ||
| } as any)) as Record<string, unknown>; |
Contributor
There was a problem hiding this comment.
Remove the extra exporter fan-out after scorer.run().
Now that scorer.run() receives the target trace/span IDs, this block publishes the same score a second time directly to exporters. That can duplicate live scores and bypass the unified observability payload shape.
Suggested change
- if (currentSpan && spanId && traceId) {
- await pMap(
- currentSpan.observabilityInstance.getExporters(),
- async exporter => {
- if (exporter.addScoreToTrace) {
- try {
- await exporter.addScoreToTrace({
- traceId: traceId,
- spanId: spanId,
- score: runResult.score as number,
- reason: runResult.reason as string,
- scorerName: scorerToUse.scorer.id,
- metadata: {
- ...(currentSpan.metadata ?? {}),
- },
- });
- } catch (error) {
- // Log error but don't fail the hook if exporter fails
- mastra.getLogger()?.error(`Failed to add score to trace via exporter: ${error}`);
- }
- }
- },
- { concurrency: 3 },
- );
- }Based on learnings: in the Mastra codebase, span.addScore() is an internal/automatic API called by scorer hooks, not a public API that developers call directly in their application code.
Also applies to: 88-112
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/core/src/mastra/hooks.ts` around lines 59 - 68, The code currently
takes the result of scorerToUse.scorer.run(...) and then publishes the same
score directly to exporters, duplicating live scores; remove the extra exporter
fan-out after the scorer.run() call and any direct calls that push the score to
exporters (the second-publish logic that follows the run) so that scores are
only emitted via the internal scorer hook mechanism (i.e., rely on
span.addScore() being invoked by the scorer implementation). Locate the post-run
export logic around scorerToUse.scorer.run and the similar block referenced in
the review (lines noted as also applying to 88-112) and delete the direct
exporter/publish code while leaving the scorer.run() invocation and normal span
handling intact.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/core/src/evals/base.ts (1)
649-653:⚠️ Potential issue | 🟠 MajorGuard prompt-step descriptions in
getSteps().Prompt-based
analyze(),generateScore(), andgenerateReason()steps setdefinitiontoundefined, sostep.definition.descriptionwill throw for exactly the scorer shapes this API needs to introspect. Read the description fromoriginalPromptObjectswhenstep.isPromptObject.🛠️ Suggested fix
getSteps(): Array<{ name: string; type: ScorerStepType; description?: string }> { - return this.steps.map(step => ({ - name: step.name, - type: step.isPromptObject ? 'prompt' : 'function', - description: step.definition.description, - })); + return this.steps.map(step => { + const description = step.isPromptObject + ? this.originalPromptObjects.get(step.name)?.description + : step.definition?.description; + + return { + name: step.name, + type: step.isPromptObject ? 'prompt' : 'function', + description, + }; + }); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/evals/base.ts` around lines 649 - 653, getSteps() currently reads step.definition.description which will throw when step.isPromptObject because analyze()/generateScore()/generateReason() set definition to undefined; change getSteps() to use the prompt description from originalPromptObjects for prompt steps: when step.isPromptObject, look up the corresponding originalPromptObjects entry (matching by step.name or the same index) and return its description instead of step.definition.description, otherwise keep using step.definition.description; update getSteps() logic to guard access to definition and prefer originalPromptObjects for prompt-type steps.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/evals/base.ts`:
- Around line 508-514: The span payloads currently include requestContext (see
the input object built around
input.input/output/groundTruth/expectedTrajectory/requestContext), which can
leak sensitive or non-serializable data into SCORER_RUN and SCORER_STEP spans;
update the span payload construction to omit requestContext (do not serialize
it) while continuing to use the normalized requestContext only when
creating/opening the span itself. Locate the input object construction in
base.ts (the block with input: { input: input.input, output: ..., groundTruth:
..., expectedTrajectory: ..., requestContext: ... }) and remove or skip the
requestContext property from that payload, and make the same change in the
corresponding input construction used for SCORER_STEP (the similar block around
lines 676-682).
---
Outside diff comments:
In `@packages/core/src/evals/base.ts`:
- Around line 649-653: getSteps() currently reads step.definition.description
which will throw when step.isPromptObject because
analyze()/generateScore()/generateReason() set definition to undefined; change
getSteps() to use the prompt description from originalPromptObjects for prompt
steps: when step.isPromptObject, look up the corresponding originalPromptObjects
entry (matching by step.name or the same index) and return its description
instead of step.definition.description, otherwise keep using
step.definition.description; update getSteps() logic to guard access to
definition and prefer originalPromptObjects for prompt-type steps.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a76f7c15-2f46-47fc-ad86-ae476938a816
📒 Files selected for processing (2)
packages/core/src/evals/base.tspackages/core/src/observability/utils.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/core/src/observability/utils.ts
Comment on lines
+508
to
514
| input: { | ||
| input: input.input, | ||
| output: input.output, | ||
| groundTruth: input.groundTruth, | ||
| expectedTrajectory: input.expectedTrajectory, | ||
| requestContext: input.requestContext, | ||
| }, |
Contributor
There was a problem hiding this comment.
Don't serialize requestContext into scorer spans.
requestContext is arbitrary propagation data, so recording it in the SCORER_RUN input and again in each SCORER_STEP input can leak auth/user metadata into traces and may break exporters on non-serializable values. Keep using the normalized requestContext for span creation, but strip it from the stored span payloads.
🔒 Suggested fix
+ const evalSpanInput = {
+ input: input.input,
+ output: input.output,
+ groundTruth: input.groundTruth,
+ expectedTrajectory: input.expectedTrajectory,
+ };
const evalSpan = getOrCreateSpan({
type: SpanType.SCORER_RUN,
name: `scorer run: '${this.id}'`,
entityType: EntityType.SCORER,
entityId: this.id,
- input: {
- input: input.input,
- output: input.output,
- groundTruth: input.groundTruth,
- expectedTrajectory: input.expectedTrajectory,
- requestContext: input.requestContext,
- },
+ input: evalSpanInput,
attributes: {
scorerId: this.id,
scorerName: this.name,
@@
+ const stepSpanInput = {
+ ...context,
+ run: { ...context.run },
+ };
+ delete stepSpanInput.run.requestContext;
const stepSpan = scorerRunSpan?.createChildSpan({
type: SpanType.SCORER_STEP,
name: `scorer step: '${scorerStep.name}'`,
entityType: EntityType.SCORER,
entityId: this.config.id ?? this.config.name,
- input: context,
+ input: stepSpanInput,
attributes: {
step: scorerStep.name,
stepType: scorerStep.isPromptObject ? 'prompt' : 'function',Also applies to: 676-682
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/core/src/evals/base.ts` around lines 508 - 514, The span payloads
currently include requestContext (see the input object built around
input.input/output/groundTruth/expectedTrajectory/requestContext), which can
leak sensitive or non-serializable data into SCORER_RUN and SCORER_STEP spans;
update the span payload construction to omit requestContext (do not serialize
it) while continuing to use the normalized requestContext only when
creating/opening the span itself. Locate the input object construction in
base.ts (the block with input: { input: input.input, output: ..., groundTruth:
..., expectedTrajectory: ..., requestContext: ... }) and remove or skip the
requestContext property from that payload, and make the same change in the
corresponding input construction used for SCORER_STEP (the similar block around
lines 676-682).
epinzur
force-pushed
the
esp/new_score_storage
branch
from
April 1, 2026 12:45
d3d90d7 to
621f020
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
packages/core/src/evals/base.ts (1)
502-528:⚠️ Potential issue | 🟠 MajorPass
requestContextthrough the tracing API instead of serializing it.
normalizedRequestContexton Line 502 never reaches the dedicatedrequestContextoption thatgetOrCreateSpan()uses for request-scoped instance selection. Instead it gets recorded in theSCORER_RUNpayload here and then copied into eachSCORER_STEP, which can leak arbitrary request data or non-serializableRequestContextinstances into traces.🔒 Suggested fix
const normalizedRequestContext = this.normalizeRunRequestContext(input.requestContext); const evalSpan = getOrCreateSpan({ type: SpanType.SCORER_RUN, name: `scorer run: '${this.id}'`, entityType: EntityType.SCORER, entityId: this.id, + requestContext: normalizedRequestContext, input: { input: input.input, output: input.output, groundTruth: input.groundTruth, expectedTrajectory: input.expectedTrajectory, - requestContext: normalizedRequestContext, }, attributes: { scorerId: this.id, scorerName: this.name, ...(input.scoreSource ? { scoreSource: input.scoreSource } : {}), @@ + const { requestContext: _requestContext, ...spanRun } = context.run; + const stepSpanInput = { ...context, run: spanRun }; const stepSpan = scorerRunSpan?.createChildSpan({ type: SpanType.SCORER_STEP, name: `scorer step: '${scorerStep.name}'`, entityType: EntityType.SCORER, entityId: this.config.id ?? this.config.name, - input: context, + input: stepSpanInput, attributes: { step: scorerStep.name, stepType: scorerStep.isPromptObject ? 'prompt' : 'function', }, });Also applies to: 675-680
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/evals/base.ts` around lines 502 - 528, The trace span currently embeds normalizedRequestContext inside the SCORER_RUN input payload instead of passing it via the dedicated requestContext parameter; update the getOrCreateSpan call (SpanType.SCORER_RUN in this file) to pass normalizedRequestContext as the requestContext option, remove it from the input.payload fields (the object under input: { ..., requestContext: ... }), and ensure subsequent SCORER_STEP spans also receive the requestContext via getOrCreateSpan rather than copying it from the payload (look for the SCORER_STEP creation code around lines ~675-680 and replicate the same change).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/evals/base.ts`:
- Around line 692-708: The scorer step body (createPrompt / executePromptStep /
executeFunctionStep) is executed outside the stepSpan async context so nested
work attaches to the wrong span; wrap the step execution inside
executeWithContext using stepObservabilityContext (the SCORER_STEP span) so that
calls in executePromptStep, executeFunctionStep and createPrompt run with the
ambient span; specifically, call executeWithContext(stepObservabilityContext,
async () => { ...invoke executePromptStep or executeFunctionStep here... })
around the existing try block that sets stepResult/prompt/judgeModel to ensure
all downstream async work inherits the SCORER_STEP context.
---
Duplicate comments:
In `@packages/core/src/evals/base.ts`:
- Around line 502-528: The trace span currently embeds normalizedRequestContext
inside the SCORER_RUN input payload instead of passing it via the dedicated
requestContext parameter; update the getOrCreateSpan call (SpanType.SCORER_RUN
in this file) to pass normalizedRequestContext as the requestContext option,
remove it from the input.payload fields (the object under input: { ...,
requestContext: ... }), and ensure subsequent SCORER_STEP spans also receive the
requestContext via getOrCreateSpan rather than copying it from the payload (look
for the SCORER_STEP creation code around lines ~675-680 and replicate the same
change).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 619ebac1-520d-4a2f-90de-9844ce179fca
📒 Files selected for processing (1)
packages/core/src/evals/base.ts
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (2)
packages/core/src/evals/base.ts (2)
702-714:⚠️ Potential issue | 🟠 MajorExecute scorer step bodies inside the
SCORER_STEPasync context.Line 702-714 runs the step body directly. Without wrapping it in
executeWithContext({ span: stepSpan, ... }), nested calls that rely on ambient current span can attach to the wrong parent.Suggested patch
try { - if (scorerStep.isPromptObject) { - const promptStepResult = await this.executePromptStep( - scorerStep, - stepObservabilityContext, - executionContext, - ); - stepResult = promptStepResult.result; - prompt = promptStepResult.prompt; - judgeModel = promptStepResult.judgeModel; - } else { - stepResult = await this.executeFunctionStep(scorerStep, executionContext); - } + await executeWithContext({ + span: stepSpan, + fn: async () => { + if (scorerStep.isPromptObject) { + const promptStepResult = await this.executePromptStep( + scorerStep, + stepObservabilityContext, + executionContext, + ); + stepResult = promptStepResult.result; + prompt = promptStepResult.prompt; + judgeModel = promptStepResult.judgeModel; + } else { + stepResult = await this.executeFunctionStep(scorerStep, executionContext); + } + }, + }); } catch (error) {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/evals/base.ts` around lines 702 - 714, The scorer step bodies (both branches that call executePromptStep and executeFunctionStep) must be executed inside the SCORER_STEP async context so nested operations inherit the correct ambient span; wrap the calls to this.executePromptStep(...) and this.executeFunctionStep(...) with executeWithContext({ span: stepSpan, kind: SCORER_STEP, observabilityContext: stepObservabilityContext }, async () => { ... }) and perform the existing assignments to stepResult, prompt and judgeModel inside that callback so behavior is unchanged but the current span is correct.
508-514:⚠️ Potential issue | 🟠 MajorAvoid serializing
requestContextinto scorer span payloads.Line 513 and Line 686 still allow
requestContextto be written intoSCORER_RUN/SCORER_STEPspan inputs. This can leak sensitive data and break exporters on non-serializable values. Keep it for execution context, but strip it from persisted spaninput.Suggested patch
- const evalSpan = getOrCreateSpan({ + const evalSpanInput = { + input: input.input, + output: input.output, + groundTruth: input.groundTruth, + expectedTrajectory: input.expectedTrajectory, + }; + + const evalSpan = getOrCreateSpan({ type: SpanType.SCORER_RUN, name: `scorer run: '${this.id}'`, entityType: EntityType.SCORER, entityId: this.id, - input: { - input: input.input, - output: input.output, - groundTruth: input.groundTruth, - expectedTrajectory: input.expectedTrajectory, - requestContext: normalizedRequestContext, - }, + input: evalSpanInput, @@ - const stepSpan = scorerRunSpan?.createChildSpan({ + const { requestContext: _requestContext, ...runWithoutRequestContext } = context.run; + const stepSpanInput = { ...context, run: runWithoutRequestContext }; + const stepSpan = scorerRunSpan?.createChildSpan({ type: SpanType.SCORER_STEP, name: `scorer step: '${scorerStep.name}'`, entityType: EntityType.SCORER, entityId: this.config.id ?? this.config.name, - input: context, + input: stepSpanInput, attributes: {Also applies to: 681-687
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@packages/core/src/evals/base.ts`:
- Around line 702-714: The scorer step bodies (both branches that call
executePromptStep and executeFunctionStep) must be executed inside the
SCORER_STEP async context so nested operations inherit the correct ambient span;
wrap the calls to this.executePromptStep(...) and this.executeFunctionStep(...)
with executeWithContext({ span: stepSpan, kind: SCORER_STEP,
observabilityContext: stepObservabilityContext }, async () => { ... }) and
perform the existing assignments to stepResult, prompt and judgeModel inside
that callback so behavior is unchanged but the current span is correct.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3bdce61d-def4-4512-abce-ba11edf7fd4f
📒 Files selected for processing (1)
packages/core/src/evals/base.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
epinzur
force-pushed
the
esp/new_score_storage
branch
from
April 1, 2026 13:17
2bbd7c7 to
86bd798
Compare
This was referenced Apr 1, 2026
roaminro
pushed a commit
that referenced
this pull request
Apr 3, 2026
…14920) ## Description This PR adds scorer observability and prepares scorer results for the new score-storage model. The main change is that scorer execution is now traced for the first time. `scorer.run()` creates `SCORER_RUN` spans, scorer pipeline steps create `SCORER_STEP` spans, and scorer results emit scores through `mastra.observability.addScore()` when a target trace is available. This also adds clearer score metadata for scorer name, target entity type, target scope, and scorer trace links. This PR keeps the legacy scores-store write path in place during the transition, but marks that path as deprecated and keeps the new observability score emission as the future direction. ## Related Issue(s) <!-- Link to the issue(s) this PR addresses, using hashtag notation: Fixes #123 --> ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Dedicated scorer tracing (run & step spans) with Sentry visibility; scores can be emitted to the observability bus when a target trace is available. * **Improvements** * Exported score payloads now include scorer name, target entity type/scope, optional trace/span anchors, and ground-truth metadata; observability API accepts optional trace anchoring and safer trace-id handling. * **Tests** * Added tests for score emission, trace linking, step tracing, and emission-failure handling. * **Chores** * Marked legacy scores-store helper as deprecated while keeping the legacy write path during transition. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
10 tasks
intojhanurag
added a commit
that referenced
this pull request
May 8, 2026
## Description Wires Mastra eval scores through the unified observability bus so every exporter implementing `onScoreEvent` automatically receives them. Replaces the deprecated per-exporter `addScoreToTrace` side-channel that bypassed the pipeline. Also unblocks Mastra spans appearing under Braintrust experiment eval spans (#11097) as a side effect of the unified path. - Producer alignment: `packages/core/src/mastra/hooks.ts` no longer iterates exporters with `addScoreToTrace`. `MastraScorer.run()` (already on main from PR #14920) remains the single producer of `ScoreEvent`s through `mastra.observability.addScore()`. - Consumer side: implemented `onScoreEvent` on Langfuse (`LangfuseClient.score.create`), Laminar (`/v1/evaluators/score`), Braintrust (`logger.logFeedback`), Datadog (`tracer.llmobs.submitEvaluation`), and LangSmith (`Client.createFeedback`). - Backwards compatibility: Langfuse `addScoreToTrace` and Laminar `_addScoreToTrace` are preserved as `@deprecated` wrappers that forward to a shared private `submitScore` helper. - Bug fix in `observability/mastra/src/recorded.ts`: `buildScoreEvent` now forwards `scorerName` and `targetEntityType` from `ScoreInput` to `ExportedScore` (without this, exporters silently fell back to `scorerId`). - Tests: rewrote core hook test to assert the hook no longer publishes a `ScoreEvent`; added per-exporter `onScoreEvent` tests; added a regression test for `buildScoreEvent` field forwarding. - Seven single-package changesets created (`@mastra/core`, `@mastra/observability`, `@mastra/langfuse`, `@mastra/laminar`, `@mastra/braintrust`, `@mastra/datadog`, `@mastra/langsmith`) per the project's changeset style guide. ## Related Issue(s) Fixes #10896 (also unblocks #11097) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [ ] Test update ## Checklist - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## ELI5 When Mastra assigns an evaluation score, this PR sends that score exactly once into a single observability "bus" so all monitoring/exporter integrations (Langfuse, Laminar, Braintrust, Datadog, LangSmith, etc.) automatically receive it — avoiding exporter-specific wiring and duplicate deliveries. ## Overview Routes Mastra evaluation scores through the unified observability ScoreEvent bus (mastra.observability.addScore) so any exporter implementing onScoreEvent receives them. MastraScorer.run() remains the single producer of ScoreEvent; the core scorer hook no longer iterates exporters to call addScoreToTrace. Per-exporter addScoreToTrace/_addScoreToTrace behaviors are preserved as @deprecated wrappers that forward to shared submit helpers where applicable. Fixes a bug in buildScoreEvent so scorerName and targetEntityType from ScoreInput are forwarded into emitted ExportedScore. ## Changes by Component ### Core (@mastra/core) - Removed exporter-iteration and addScoreToTrace republishing from createOnScorerHook; the hook now only persists scores to the legacy scores store. - Ensures MastraScorer.run() is the single producer of ScoreEvent. - Registers static workflow-step scorers synchronously during addWorkflow and resolves dynamic scorers asynchronously so scorers are registered before use. - Updated core test to assert the hook does not emit ScoreEvent. ### Observability (@mastra/observability) - Fix: buildScoreEvent (observability/mastra/src/recorded.ts) now includes scorerName and targetEntityType in emitted ScoreEvent. - Bus routing: route-event now prefers onScoreEvent; if an exporter lacks onScoreEvent but implements deprecated addScoreToTrace, the bus maps the ScoreEvent into the legacy addScoreToTrace payload and calls it (avoids duplicate delivery when both exist). - Added unit tests for score routing behavior and the buildScoreEvent regression. ### Exporter Integrations (new onScoreEvent handlers + tests) - Langfuse (@mastra/langfuse) - Added onScoreEvent → LangfuseClient.score.create with a shared submitScore helper and centralized error logging. - Preserved addScoreToTrace as a deprecated wrapper forwarding to submitScore. - Tests updated to cover onScoreEvent and mark addScoreToTrace deprecated. - Laminar (@mastra/laminar) - Added onScoreEvent delegating to a private submitScore that POSTS to /v1/evaluators/score. - submitScore converts trace/span IDs to Laminar/OTel UUIDs, merges metadata/reason, enforces timeouts via AbortController, and logs non-OK/timed-out responses. - Preserved _addScoreToTrace as a @deprecated wrapper forwarding to submitScore. - Tests verify POST payload mapping and wrapper behavior. - Braintrust (@mastra/braintrust) - Added onScoreEvent that calls logger.logFeedback using spanId (falling back to traceId) and skips when neither is present. - Maps scorer name/ID, reason, and metadata into the feedback payload and logs failures. - Tests cover span/trace fallback and skip behavior. - Datadog (@mastra/datadog) - Added onScoreEvent: validates traceId/spanId, looks up exported dd-trace span context (traceState), and calls tracer.llmobs.submitEvaluation with mapped fields (label/value/metricType/mlApp/timestamp/reason/metadata). - Drops scores when identifiers or exported span context are missing and warns when scores arrive before the span is exported. - Tests cover successful submission, unknown IDs (no-op), and dropping out-of-order scores. - LangSmith (@mastra/langsmith) - Added onScoreEvent that submits Client.createFeedback when a spanId→LangSmith runId mapping exists. - Introduced a configurable bounded LRU spanId→runId cache (runIdCacheMaxEntries) populated from RunTree building; evicts oldest entries when full and is cleared on shutdown. - Ensures authoritative scorerId/scoreSource are preserved in sourceInfo and ignores events when span/run mapping is absent. - Tests cover mapping semantics, eviction, metadata protection, and skipping unseen spans. ## Tests - Core hook test rewritten to assert the hook does not publish ScoreEvent. - Per-exporter onScoreEvent test suites added for Langfuse, Laminar, Braintrust, Datadog, and LangSmith covering field mapping, identifier fallbacks, ordering/dropping, timeout handling, and cache eviction. - Bus tests added to assert onScoreEvent preferred and legacy addScoreToTrace fallback behavior. - Regression test added to ensure buildScoreEvent forwards scorerName and targetEntityType. ## Backwards Compatibility - Deprecated addScoreToTrace/_addScoreToTrace preserved (where present) as wrappers that forward to the new onScoreEvent/submit helpers to avoid breaking existing callers. - Observability interface docs updated to deprecate addScoreToTrace in favor of onScoreEvent/ScoreEvent bus. - No breaking public API surface changes beyond documented additions and deprecations. ## Commits & Release Notes - Seven package-scoped changesets created for @mastra/core, @mastra/observability, @mastra/langfuse, @mastra/laminar, @mastra/langsmith, @mastra/datadog, and @mastra/braintrust documenting onScoreEvent support and the buildScoreEvent fix. - Exporter-specific notes documented (Datadog ordering caveat; Laminar timeout/error logging; LangSmith runId cache behavior). ## Related Issues Fixes #10896 and unblocks #11097. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
mbenhamd
added a commit
to mbenhamd/mastra
that referenced
this pull request
May 14, 2026
* chore: regenerate providers and docs [skip ci]
* fix(playground): stabilize streaming message rendering (#16331)
This fixes a couple Studio rendering issues that showed up during fast
streaming output.
Reasoning chunks now stay separated when a model interleaves reasoning,
tool calls, and text instead of appending everything back into the first
reasoning block.
Before:
```tsx
reasoning: "first thought second thought third thought"
tool call
text
```
After:
```tsx
reasoning: "first thought"
tool call
reasoning: "second thought"
text
reasoning: "third thought"
```
Autoscroll also keeps following rapid output from parallel tool calls
unless the user intentionally scrolls away, so content growth alone does
not disable follow mode.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR fixes how the Studio chat displays AI responses that stream in
very quickly. When an AI thinks, then uses a tool, then replies with
text, the fix keeps each "thought" separate instead of mixing them all
together. It also improves the auto-scroll feature so the chat keeps
showing new messages at the bottom, but stops automatically scrolling if
you manually scroll up.
## Changes Overview
This pull request addresses streaming render stability in the Studio
playground by improving how interleaved reasoning chunks are handled and
refining autoscroll behavior during rapid output.
### Reasoning Chunk Handling
The `toUIMessage` utility now preserves separate reasoning blocks when
`reasoning-delta` chunks arrive interleaved with other message parts
(tool calls, text). Previously, all reasoning content was merged into
the first reasoning block regardless of where it appeared in the message
sequence. The new behavior:
- Appends new reasoning-delta text to the most recent trailing reasoning
part if one exists
- Creates a new separate reasoning part if the last part is not
reasoning, preventing merged reasoning blocks
A new test case validates this behavior, ensuring reasoning parts remain
distinct and properly sequenced.
### Autoscroll Behavior Improvements
The `useAutoscroll` hook now better handles user interaction during
rapid streaming:
- Detects user scroll intent via wheel, touch, pointer, and keyboard
events
- Temporarily disables auto-follow when explicit user scroll-up is
detected
- Cancels pending animation frames to prevent forced viewport
repositioning after user scrolling
- Uses `requestAnimationFrame` for scroll animations instead of smooth
scroll behavior
- Introduces a `SCROLL_END_THRESHOLD` constant (8px) to determine
whether content should trigger auto-scrolling
- Expands scroll triggers from mutation observation to include both
`MutationObserver` (content changes) and `ResizeObserver` (size changes)
### Files Modified
- `.changeset/quiet-geese-scroll.md` - Changelog entry describing the
fixes for `@mastra/react` and `@mastra/playground-ui`
- `client-sdks/react/src/lib/ai-sdk/utils/toUIMessage.ts` - Updated
reasoning-delta handling logic
- `client-sdks/react/src/lib/ai-sdk/utils/toUIMessage.test.ts` - Added
test for interleaved reasoning preservation
- `packages/playground-ui/src/hooks/use-autoscroll.tsx` - Enhanced
autoscroll with user intent detection and improved animation handling
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>
* chore: version packages
* chore: regenerate providers and docs [skip ci]
* ci(e2e): migrate remaining matrix jobs to starsling-ubuntu-24.04 (#16330)
Co-authored-by: starslingdev[bot] <248995740+starslingdev[bot]@users.noreply.github.com>
* fix(core): fix requestContext serialization in persistStepUpdate (#16082)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: caleb <caleb@mastra.ai>
Co-authored-by: Caleb Barnes <caleb.d.barnes@gmail.com>
* fix(observability): resolve OpenRouter "vendor/model" ids in pricing lookup (#16206)
## Description
Adds two more model-id variants to the pricing lookup so OpenRouter ids
in `vendor/model` form resolve:
- slash-flattened (`xiaomi/mimo-v2-pro` → `xiaomi-mimo-v2-pro`)
- vendor-prefix-dropped (`openai/gpt-5-mini` → `gpt-5-mini`)
Date stripping is applied to each variant; the variant set is deduped so
non-prefixed inputs don't pay for redundant lookups. Native provider
lookups are unchanged.
Follows up on #14959 (dot-to-dash) and #15349 (date suffixes).
## Related Issue(s)
Closes #16205
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR fixes how the system finds prices for OpenRouter models that use
names like "vendor/model" by trying additional name variants (like
replacing "/" with "-" and dropping the vendor prefix) so the correct
cost is found and shown.
## Changes
### Core Logic (observability/.../pricing-registry.ts)
- Rewrote getModelVariants to produce a prioritized, deduplicated set of
lookup variants for a given model id.
- New variants generated:
- Slash-to-dash normalization (e.g., `xiaomi/mimo-v2-pro` →
`xiaomi-mimo-v2-pro`)
- Vendor-prefix-dropped for slash-containing ids (e.g.,
`openai/gpt-5-mini` → `gpt-5-mini`)
- Dot-to-dash normalization (existing behavior retained)
- Date-suffix stripping is applied to every variant.
- Variants are deduplicated to avoid redundant lookups.
- Doc comment expanded to explain variant strategy and ordering.
- Native provider lookups are unchanged.
### Tests (observability/.../estimator.test.ts)
- Added two Vitest cases to validate OpenRouter model normalization and
pricing resolution:
1. Resolves `xiaomi/mimo-v2-pro-...` to `xiaomi-mimo-v2-pro` when
pricing keeps the vendor prefix.
2. Resolves `openai/gpt-5-mini-...` to `gpt-5-mini` when pricing omits
the vendor prefix.
- Ensures pricing_id/tier_index and cost calculations match
expectations.
### Test Fixtures (observability/.../pricing-data-test.jsonl)
- Added JSONL fixture entries for OpenRouter models used in tests:
- `openrouter-xiaomi-mimo-v2-pro` with USD pricing under
model_pricing/v1
- `openrouter-gpt-5-mini` with USD pricing under model_pricing/v1
### Changelog (.changeset/gentle-maps-rest.md)
- Patch entry for @mastra/observability documenting the fix: Model Usage
& Cost now shows costs for OpenRouter IDs in `vendor/model` form that
previously appeared empty.
## Impact
- Fixes observation/metrics cost lookup failures for OpenRouter models
using `vendor/model` IDs by expanding normalization rules and applying
date-stripping to all variants.
- Backward-compatible: native provider lookups unchanged.
- Tests and fixtures included; related issue closed: #16205
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* fix(playground-ui): unbreak storybook build (codemirror lang-jinja missing peer deps) (#16346)
## Summary
- Vercel preview's `pnpm build-storybook` for `@mastra/playground-ui`
started failing on every PR with `[vite]: Rollup failed to resolve
import "@codemirror/view" from "@codemirror/lang-jinja"`.
- Root cause: upstream bug in `@codemirror/lang-jinja@6.0.0` — it
imports `@codemirror/view` and `@codemirror/state` but does not declare
them as deps. Pulled in transitively via `@codemirror/language-data`
lazy import (added to playground-ui in #9302).
- Latent until #16245 (`chore(deps): lock file maintenance`) regenerated
the pnpm graph (bumped `@codemirror/view` 6.41.0 → 6.41.1,
`@codemirror/search` 6.6.0 → 6.7.0). New hoisting layout no longer
placed `@codemirror/view` next to `lang-jinja`, so Rollup strict
resolution failed. Vite dev hides it via pre-bundling; Rollup prod build
does not.
- Fix: use `pnpm.packageExtensions` to inject the missing deps into
`lang-jinja`. Standard pnpm mechanism for upstream peer-dep gaps; no
revert needed.
## Test plan
- [x] `pnpm install` — lock regenerated cleanly
- [x] Verified
`node_modules/.pnpm/@codemirror+lang-jinja@6.0.0/node_modules/@codemirror/`
now contains `state` and `view`
- [x] `pnpm --filter @mastra/playground-ui build-storybook` — build
green locally
- [ ] Vercel preview build green on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
The package that needed two helper tools (@codemirror/view and
@codemirror/state) was fixed upstream, and this PR tells our package
manager to use that fixed version so Storybook can build again.
## Problem
Vercel preview builds for @mastra/playground-ui Storybook failed during
Rollup bundling because @codemirror/lang-jinja@6.0.0 imported
@codemirror/view and @codemirror/state but did not declare them as
dependencies. Lockfile maintenance (#16245) changed pnpm hoisting and
revealed the missing deps (Vite dev pre-bundling had hidden the issue).
## Solution
Use pnpm overrides to pin @codemirror/lang-jinja to ^6.0.1, which
upstream released with @codemirror/view and @codemirror/state properly
declared, removing the previous packageExtensions workaround.
## Changes
- package.json: added pnpm.overrides entry for @codemirror/lang-jinja ->
^6.0.1 (net +2/-1 lines)
## Verification
- `pnpm install` — lockfile regenerated cleanly
- Confirmed node_modules contains @codemirror/state and @codemirror/view
under @codemirror/lang-jinja@6.0.1
- `pnpm --filter @mastra/playground-ui build-storybook` — local build
succeeds
- Vercel preview build: pending
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* refactor(playground): redesign composer and polish DS interactions (#16269)
## Summary
- **Composer redesign**: split into a pill-shaped provider/model
switcher, an action row that wraps cleanly when space is tight
(`flex-wrap-reverse`, no breakpoint magic), and a dedicated send button.
Attachments now float above the composer. Smaller remove icon (`X` in
`Button` + `icon-sm`).
- **DS Combobox / Select**: added an `active:bg-surface4` state so
triggers feel like buttons, and fixed value truncation when a leading
badge (e.g. provider connection-status dot) is rendered — the truncate
now sits on the inner span so absolutely-positioned children stay
visible.
- **DS Threads**: dropped the default `bg-surface2` background so
consumers control the surface.
- **DS PanelSeparator**: enlarged hit area, refined hover/active fills,
added a focus-visible accent.
- **Agents layout polish**: removed the header bottom border, removed
`bg-surface2 px-4` from the page tabs, removed `py-4` from the main
panel; `LLMProvider`'s logo gets `shrink-0` + min size to keep the
connection-status dot anchored inside the new pill.
Before:
<img width="1731" height="993" alt="CleanShot 2026-05-06 at 17 06 37"
src="https://github.com/user-attachments/assets/86ad5f91-bdec-4f0f-9dad-5691ed2f1907"
/>
After:
<img width="1731" height="993" alt="CleanShot 2026-05-06 at 17 04 24"
src="https://github.com/user-attachments/assets/a4c0968c-f5f8-4ced-98a5-b9df15aeb1de"
/>
## Why
The composer felt cramped at narrow widths, the model switcher squashed
the provider logo and clipped its connection-status dot, and the
surrounding agent layout had stacked grey backgrounds (`Threads` + tabs
+ panel) that competed visually. This PR consolidates the chat surface,
gives the switcher a proper pill shape, and tightens DS triggers so they
read as buttons.
## Test plan
- [ ] Composer: type a message, attach files, verify attachments render
above the composer and remove (`X`) works.
- [ ] Composer model switcher: verify provider + model pills share a
single border, hover/active fills clip to the pill outline,
connection-status dot remains visible at all widths.
- [ ] Narrow viewport: verify the action row wraps so the switcher drops
below the buttons (`flex-wrap-reverse`) without overflow.
- [ ] Missing-API-key warning: select a non-connected provider — banner
should render above the action row, not inside the switcher pill.
- [ ] DS Combobox / Select: hover + active states show new fill; opening
triggers no longer flash; truncation works for long values.
- [ ] PanelSeparator: hover, drag, and focus-visible all render the new
affordance.
- [ ] `Threads` consumers (`chat-threads`, `workflow-run-list`): verify
the new transparent default still looks correct in their parents.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR redesigns how the message composer and related interface
elements look and work in Mastra Studio. The composer is split into
simpler, more compact pieces that rearrange cleanly on smaller screens,
attachments now float above the composer instead of being cramped inside
it, and various design system components get visual feedback
improvements (like buttons lighting up when clicked) and better text
handling.
---
## Composer Redesign & Layout
- **Composer structure overhaul**: Split into a pill-shaped
provider/model switcher, an action row that wraps cleanly without
breakpoint-specific hacks (using `flex-wrap-reverse`), and a dedicated
send button
- **Floating attachments**: Moved to absolute positioning above the
composer input, with removal controls reduced to a smaller X icon
(changed from CircleXIcon to X, size downgraded from icon-md to icon-sm)
- **ComposerActionRow**: New component consolidates model switcher,
attachment button, dictation input, and send/cancel controls with
cleaner conditional rendering
- **Missing API key banner**: Now positioned above the action row
instead of inside the switcher pill
---
## Design System Component Updates
**Combobox & Select**:
- Added `active:bg-surface4` state to trigger elements so they feel like
interactive buttons with visual feedback
- Fixed value truncation when a leading badge (e.g., provider
connection-status dot) is rendered by moving truncation logic to an
inner span wrapper, keeping absolutely-positioned children visible
**Threads**:
- Removed default `bg-surface2` background from root nav element,
allowing consumers to control the surface color
**PanelSeparator**:
- Enlarged hit area for dragging
- Refined hover and active fill states
- Added focus-visible accent for keyboard navigation affordance
- Now renders a resizing handle with improved structural markup
---
## Agent Layout Refinements
- **Agent header**: Added `border={false}` prop to Header component to
remove bottom border
- **Page tabs**: Removed `bg-surface2` and `px-4` from the wrapper,
leaving a plain flex container
- **Main panel**: Removed vertical padding (`py-4`) from the layout
- **Provider logo**: Added `shrink-0` and minimum size constraints to
keep the connection-status dot anchored inside the provider/model pill
at narrow widths; improved fallback UI with consistent sizing
---
## Testing Considerations
- Verify attachments float above composer and removal works as expected
- Confirm provider+model pill shares a single border and
connection-status dot remains visible at narrow widths
- Validate action row wraps with `flex-wrap-reverse` without content
overflow
- Ensure missing-API-key banner renders above the action row, not inside
the switcher pill
- Check Combobox/Select visual feedback (hover/active states) and
truncation behavior with badges
- Validate PanelSeparator hover, drag, and focus-visible affordances are
clear
- Confirm Threads consumers render correctly with the
transparent/controllable default background
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Add MODEL_INFERENCE span type for measuring model provider latency (#16267)
## Description
This PR introduces a new `MODEL_INFERENCE` span type that wraps the
model provider call within a `MODEL_STEP`, enabling precise measurement
of model latency separately from input/output processing and tool
executions.
### Key Changes
- **New Span Type**: Added `MODEL_INFERENCE` to `SpanType` enum with
corresponding attributes interface (`ModelInferenceAttributes`)
- **Span Hierarchy**: Updated from `MODEL_GENERATION -> MODEL_STEP ->
MODEL_CHUNK` to `MODEL_GENERATION -> MODEL_STEP -> MODEL_INFERENCE ->
MODEL_CHUNK`
- **Processor/Tool Handling**: Processors and tool executions remain as
siblings of `MODEL_INFERENCE` under `MODEL_STEP`, ensuring
`MODEL_INFERENCE` measures only pure model time
- **Feature Gate**: Implemented behind `model-inference-span` feature
flag for gradual rollout
- **Snapshot Updates**: Updated all test snapshots to reflect the new
span hierarchy
### Implementation Details
- Modified `ModelSpanTracker` to create `MODEL_INFERENCE` spans when the
feature is enabled
- Added `ModelInferenceContext` type for tracking inference-specific
data
- Updated span hierarchy in trace generation and visualization
- Added comprehensive test coverage for the new span type
### Files Modified
- `packages/core/src/observability/types/tracing.ts`: Added
`MODEL_INFERENCE` span type and attributes
- `observability/mastra/src/model-tracing.ts`: Implemented inference
span creation and tracking
- `observability/mastra/src/model-tracing.test.ts`: Added tests for
inference span behavior
- `packages/core/src/features/index.ts`: Added feature flag
- Multiple snapshot files: Updated to reflect new span hierarchy
- Supporting files: Updated span type handling in base spans, evals, and
processor tracing
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Testing
Added comprehensive unit tests in `model-tracing.test.ts` covering:
- Inference span creation as child of MODEL_STEP
- Chunk spans properly parented under inference spans
- Inference span attributes (stepIndex, model, provider, streaming,
parameters, availableTools)
- Multiple inference spans within a single step
- Feature flag behavior
All existing tests pass with updated snapshots reflecting the new span
hierarchy.
https://claude.ai/code/session_01GyZ9wem6FgwPECXpnMQsSK
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR adds a separate "MODEL_INFERENCE" timing span so we can measure
only the time the model provider spends generating responses—separate
from other work like tool calls or input/output handling.
## Summary
Introduces a new SpanType MODEL_INFERENCE and supporting types/logic to
isolate model provider latency by changing the span hierarchy to
MODEL_GENERATION → MODEL_STEP → MODEL_INFERENCE → MODEL_CHUNK. The
feature is behind a `model-inference-span` feature flag for gradual
rollout; when disabled, behavior falls back to the previous MODEL_STEP →
MODEL_CHUNK parenting.
### Key Changes
- Types & API
- Added SpanType.MODEL_INFERENCE and ModelInferenceAttributes,
ModelInferenceContext.
- Extended IModelSpanTracker with optional setInferenceContext(context).
- Added observabilityFeatures export containing 'model-inference-span'.
- Added 'model_inference' route-name literal to client-js route types.
- Core implementation
- ModelSpanTracker creates MODEL_INFERENCE spans under MODEL_STEP when
the feature flag is enabled and parents MODEL_CHUNK spans to the active
inference span.
- New setInferenceContext persists request-side parameters/provider
options/availableTools/toolChoice/responseFormat for subsequent
inference spans.
- wrapStream and chunk parent selection centralized to respect STEP →
INFERENCE → CHUNK lifecycle.
- Inference spans are ended eagerly on step finish (including
durable/deferred step flows) so later tool work doesn't inflate
inference duration.
- Usage and finishReason are duplicated onto both MODEL_INFERENCE and
MODEL_STEP for compatibility.
- Span internal classification and trajectory conversion updated to
treat MODEL_INFERENCE as internal / skipped type where appropriate.
- Integrations
- MastraLLMVNext.stream and durable LLM execution wiring call
setInferenceContext to propagate request context (including
structured-output responseFormat when applicable).
- Durable LLM execution adds inference context setup to ensure correct
attributes for persisted runs.
- Testing & snapshots
- Comprehensive unit tests added/updated to validate inference span
creation, parenting, attributes, context propagation, multiple
inferences per step, durable-mode behavior, and feature-flag fallback.
- ~29 observability snapshot files updated to reflect new span hierarchy
and adjusted metadata ordering (environment before modelMetadata).
- Test exporter gained getModelInferenceSpans() helper.
- Backward compatibility & rollout
- Feature gated by core feature flag 'model-inference-span'
(coreFeatures and observabilityFeatures).
- If the flag is absent (older packages), traces revert to prior
structure (chunks under MODEL_STEP).
- Changeset bumps for @mastra/core and @mastra/observability (minor).
Files of note modified:
packages/core/src/observability/types/tracing.ts,
observability/mastra/src/model-tracing.ts (+tests),
packages/core/src/features/index.ts, Mastra LLM loop & durable execution
files, many observability snapshots and test helpers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(core,server,cli,redis-streams): standalone workers + push-capable PubSub + Redis Streams broker (#16309)
## Summary
Introduces a first-class **MastraWorker** abstraction so a Mastra
deployment can split the workflow orchestrator, scheduler, and
background-task worker into separate processes deployed independently
from the HTTP server. Also adds a production Redis Streams PubSub
implementation, a push-capable PubSub interface, and the supporting
routes / CLI / docs.
The default (single-process) deployment is unchanged: when nothing is
configured, Mastra auto-creates the three workers in-process and runs as
before.
## What's in this PR
### Worker abstraction (packages/core, packages/cli, packages/server)
- `MastraWorker` base class with `OrchestrationWorker`,
`SchedulerWorker`, `BackgroundTaskWorker` concrete implementations.
- Mastra auto-creates the three workers by default; opt out per-worker
via `MASTRA_WORKERS` env (e.g. `MASTRA_WORKERS=orchestration,scheduler`
runs only those; `MASTRA_WORKERS=false` disables event processing
entirely).
- New `mastra worker` CLI command for running a worker process from a
bundled Mastra entry file. `WorkerBundler` reuses the existing build
pipeline.
- Two step-execution strategies for orchestrators that don't run in the
server process:
- `InProcessStrategy`: default, executes steps locally.
- `HttpRemoteStrategy`: ships step execution to the server's `POST
/api/workflows/:id/runs/:runId/steps/execute` route. Auth is delegated
to the framework's existing `experimental_auth` provider via standard
`Authorization: Bearer` headers — no separate worker-secret machinery.
- `BackgroundTaskWorker` reuses Mastra's `BackgroundTaskManager`
instance when present so cross-process workers share the same dispatch /
result / recovery semantics. When dispatch is received in a worker
process that doesn't hold the producer's per-task closure, executor
lookup falls back to a static `toolName → executor` registry populated
from the registered tools on Mastra.
### Push-capable PubSub (packages/core, packages/server)
- `PubSub` gains a `supportedModes: ('pull' | 'push')[]` declaration.
Existing third-party implementations default to `['pull']` for backwards
compatibility.
- `EventEmitterPubSub` declares `['pull', 'push']` (in-process, supports
both).
- `mastra.handleWorkflowEvent(event)` is the new public hook for push
delivery. When configured with a push-only PubSub, Mastra wires the
in-process subscription to call it directly and skips auto-creating an
`OrchestrationWorker`.
- New `POST /api/workers/events` route receives workflow events from
push-mode brokers (e.g. GCP Pub/Sub push subscriptions, EventBridge HTTP
targets) and forwards them to `handleWorkflowEvent`. This route is the
symmetric counterpart to the step-execute route and is gated by the same
auth provider mechanism.
### Redis Streams PubSub (pubsub/redis-streams)
New `@mastra/redis-streams` package implementing a production-grade
pull-mode broker:
- Consumer groups created at stream start (`'0'`) so late-joining
workers see historical events.
- `XAUTOCLAIM` reclaim loop for pending messages whose consumer died.
- Configurable `maxDeliveryAttempts` cap on `nack` (poison-pill
protection).
- Configurable `MAXLEN ~` stream trim on publish.
- Topic-scoped `unsubscribe` + clean shutdown.
- Structured logger pass-through; no silent `catch {}` blocks.
### Cross-process integration tests (pubsub/redis-streams/src)
Real multi-process tests using the bundled CLI worker artifact + Redis:
- `cross-process.test.ts` — server + standalone orchestrator
- `scheduler-cross-process.test.ts` — standalone scheduler
- `background-cross-process.test.ts` — standalone background-task worker
(proves cross-process executor resolution via the static-tool registry)
- `all-workers-split.test.ts` — server + all three workers in separate
processes
- `cli-bundler.test.ts` — `mastra worker` bundled artifact boots and
processes events
- `resilience.test.ts` — competing consumers, replica failover,
late-joining workers
- `auth-e2e.test.ts` — confirms `Authorization: Bearer` tokens are
validated by the framework auth provider on both `steps/execute` and
`workers/events` routes
### Docs
- `deployment/standalone-workers.mdx` — covers `MASTRA_WORKERS`, remote
step execution via `MASTRA_STEP_EXECUTION_URL` +
`MASTRA_WORKER_AUTH_TOKEN`, push vs pull pubsub, example server-side
`MastraAuthConfig`.
- `deployment/overview.mdx` — new section linking to standalone workers.
## What's NOT in this PR
- GCP Pub/Sub / SNS / EventBridge / SQS adapters — push-mode receive
route exists, but the broker-specific envelope decoders are
intentionally out of scope.
- `mastra worker temporal` — the existing `@mastra/temporal` package
stays as a thin library; no new worker subclass.
## Test plan
- `pnpm --filter ./packages/core check` (typecheck)
- `pnpm test:core`
- `pnpm --filter ./packages/server test`
- `pnpm --filter ./pubsub/redis-streams test` (against local Redis)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR lets Mastra split its work into separate programs: one that
schedules jobs, one that decides what to run, and one that runs
background tasks. It adds a CLI to build/run those workers, a
Redis‑Streams message broker to coordinate them, and a way for remote
workers to ask the server to execute individual workflow steps over HTTP
(with authentication).
---
## Overview
Adds first-class standalone worker support and a Redis Streams PubSub
broker so Mastra can run orchestration, scheduler, and background-task
workers as separate processes while preserving single-process defaults.
Includes worker lifecycle APIs, two step-execution strategies
(in-process and HTTP remote), CLI bundling/launcher, push-capable PubSub
hooks, a production-grade Redis Streams implementation, multi-process
integration tests, and deployment docs.
---
## Key Features & Changes
- Worker framework
- New MastraWorker base and concrete OrchestrationWorker,
SchedulerWorker, BackgroundTaskWorker.
- Auto-creation of workers by default; controllable via config.workers
and MASTRA_WORKERS (comma-separated selection or MASTRA_WORKERS=false).
- Mastra exposes workers getter and getWorker(name); startWorkers(name?)
/ stopWorkers(); legacy startEventEngine/stopEventEngine kept as
deprecated aliases.
- Step execution strategies
- InProcessStrategy (default) executes steps in-process.
- HttpRemoteStrategy posts to POST
/api/workflows/:id/runs/:runId/steps/execute with merged abort/timeout
semantics and bearer auth from MASTRA_WORKER_AUTH_TOKEN or explicit
config.
- Server adds the step-execution route with payload validation and
permissive JSON response.
- CLI & bundling
- mastra worker [name] command and WorkerBundler to emit/run bundled
worker artifacts, with dotenv handling, platform selection, and
signal-safe shutdown.
- PubSub delivery modes & push support
- PubSub now has supportedModes: ('pull'|'push')[] (default ['pull']);
EventEmitterPubSub supports ['pull','push'].
- Mastra.handleWorkflowEvent(event) public hook and POST
/api/workers/events for push-mode deliveries (auth gated).
- Orchestration worker creation skipped when pubsub lacks 'pull';
startWorkers wires push subscriptions when appropriate.
- Redis Streams broker (@mastra/redis-streams)
- RedisStreamsPubSub implementing pull-only delivery with consumer
groups (anchored at '0'), XAUTOCLAIM reclaim loop, configurable reclaim
intervals, optional MAXLEN trimming, and topic-scoped unsubscribe.
- Delivery attempt tracking with maxDeliveryAttempts (validation:
negative/NaN rejected; 0 treated as Infinity with a one-time warning;
Infinity disables cap).
- Ack via XACK (no xdel) so other groups can replay; nack
republish-redelivers with incremented deliveryAttempt; events dropped
and warned when attempts exhausted.
- Per-subscriber reader clients, flush/close semantics, clean shutdown,
structured logger passthrough, tests and docker-compose for Redis.
- Background tasks cross-process
- BackgroundTaskManager static executor registry
(registerStaticExecutor/unregisterStaticExecutor/getStaticExecutor) for
cross-process dispatch.
- BackgroundTaskWorker reuses Mastra's manager when present (preserves
per-task contexts) or owns a manager otherwise; subscribes on start();
start() throws if init() wasn't called.
- Transport & routing
- Worker transport interfaces (WorkerTransport, EventRouter) and
PullTransport implementation that subscribes grouped pull consumers and
nacks on handler promise rejections with injected logger support.
- WorkflowEventProcessor improvements
- Pluggable StepExecutionStrategy injection.
- New handle(event) returning { ok:true } | { ok:false, retry:boolean };
process(event, ack?) preserved as wrapper.
- Robustness and infra fixes
- Idempotent lifecycle operations, stricter validations, improved
logging, proper nacking on handler failures, unsubscribe scoping fixes,
CLI spawn error surfacing, test flake fixes, and related hardening.
---
## API Additions (high level)
- Core
- Mastra.handleWorkflowEvent(event), startWorkers, stopWorkers, workers
getter, getWorker(name).
- MastraWorker, OrchestrationWorker, SchedulerWorker,
BackgroundTaskWorker (+ configs).
- StepExecutionStrategy, StepExecutionParams, InProcessStrategy,
HttpRemoteStrategy, StepExecutionError.
- WorkerTransport/EventRouter interfaces and PullTransport.
- PubSubDeliveryMode type and supportedModes getter on PubSub.
- Server
- POST /api/workflows/:workflowId/runs/:runId/steps/execute (requires
auth when an auth provider is configured).
- POST /api/workers/events for push-mode broker deliveries.
- CLI
- mastra worker [name] command and WorkerBundler.
- New package
- @mastra/redis-streams: RedisStreamsPubSub and RedisStreamsPubSubConfig
plus test/build configs.
---
## Testing
Extensive Vitest integration tests exercise:
- Multi-process topologies (server + orchestration + scheduler +
background) against real Redis and shared storage.
- Cross-process step execution and auth (auth-e2e).
- Background-task cross-process dispatch and failure cases.
- Scheduler cross-process behavior and resilience (late-join recovery,
competing consumers, reclaim/XAUTOCLAIM).
- RedisStreams PubSub behaviors (fan-out, groups, ack/nack, redelivery
caps, unsubscribe scoping).
- CLI bundler output check (skipped-by-default gated test).
---
## Docs
- deployment/standalone-workers.mdx and deployment/overview.mdx
- explorations/orchestrator-report.md
- Changeset documenting push-capable PubSub, startWorkers usage, and
operational notes.
---
## Breaking Changes
No functional breaking changes—single-process defaults preserved.
startEventEngine/stopEventEngine are deprecated aliases for
startWorkers/stopWorkers.
---
## Migration / Usage Notes
- To run workers as separate processes: configure a Redis Streams
broker, build bundles with mastra worker, and run desired workers
(orchestration, scheduler, backgroundTasks) or use MASTRA_WORKERS to
select.
- For remote step execution: set MASTRA_STEP_EXECUTION_URL on workers
and provide MASTRA_WORKER_AUTH_TOKEN (or explicit HttpRemoteStrategy
auth) to authenticate server calls.
- For push-only PubSub providers, Mastra will call handleWorkflowEvent
for push delivery and skip auto-creating orchestration workers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-7) <noreply@mastra.ai>
* feat(observability): expose dropped observability events (#16111)
Adds structured drop events for observability data that
`DefaultExporter` cannot persist. Exporters and bridges can implement
`onDroppedEvent` to observe unsupported storage drops and retries being
exceeded with signal, reason, count, storage, exporter, timestamp, and
sanitized error details.
Before, unsupported logs or metrics could be dropped after a local
warning, and retry exhaustion inside the event buffer was not visible to
other observability sinks.
After:
```ts
class DropAlertExporter extends BaseExporter {
name = 'drop-alerts'
async onDroppedEvent(event: ObservabilityDropEvent) {
await fetch('https://monitoring.example.com/observability-drops', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({
count: event.count,
signal: event.signal,
reason: event.reason,
exporterName: event.exporterName,
}),
})
}
}
```
Includes focused tests for bus fan-out, `DefaultExporter`
unsupported-storage and retry-exhausted paths, and `EventBuffer`
dropped-event return values.
Fixes #16040
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
When observability systems (which collect logs, metrics, and traces from
applications) can't store or send data, they usually just silently fail.
This PR adds a way for exporters to be notified about these failures so
that external monitoring systems can alert when observability is only
partially working—for example, when traces are being captured but logs
aren't.
---
## Overview
This PR introduces a structured event notification system for dropped
observability data. When the observability pipeline cannot persist
events (due to unsupported storage or exhausted retries), it now emits
standardized `ObservabilityDropEvent` objects to exporters and bridges
implementing the new `onDroppedEvent` handler, enabling external
monitoring and alerting on partial observability setups.
---
## Key Changes
### New Type Definitions
Four new types are introduced in
`packages/core/src/observability/types/core.ts`:
- `ObservabilityDropSignal`: Identifies the signal type (`'tracing' |
'log' | 'metric' | 'score' | 'feedback'`)
- `ObservabilityDropReason`: Indicates why the event was dropped
(`'unsupported-storage' | 'retry-exhausted'`)
- `ObservabilityDropError`: Sanitized error details (`id`, `domain`,
`message`)
- `ObservabilityDropEvent`: Complete drop notification with signal,
reason, count, timestamp, exporter/storage names, and optional error
### API Extensions
- `ObservabilityExporter` interface now includes optional
`onDroppedEvent?(event: ObservabilityDropEvent): void | Promise<void>`
handler
- `InitExporterOptions` now includes optional `emitDropEvent` callback
for exporters to emit drop events
### Bus & Routing
- `ObservabilityBus` adds `emitDropEvent(event: ObservabilityDropEvent):
void` method to broadcast drop events to all registered exporters and
bridges
- New `routeDropToHandler` function in `route-event.ts` safely routes
drop events to exporter handlers with error handling and async support
### DefaultExporter Enhancements
`DefaultExporter` now:
- Tracks unsupported signals per storage instance using typed
`ObservabilityDropSignal`
- Emits drop events with `reason: 'unsupported-storage'` when a storage
adapter doesn't support a signal
- Emits drop events with `reason: 'retry-exhausted'` when flush retries
exceed the configured maximum
- Includes sanitized `MastraError` details in drop events for debugging
- Threads drop emission through create-flush and span update/end paths
- Uses singular signal names (`log`, `metric`, `score`) instead of
plural in drop events
### EventBuffer Updates
`EventBuffer.reAddCreates()` and `EventBuffer.reAddUpdates()` now return
`BufferedEvent[]` of events that exceeded `maxRetries` instead of
silently dropping them. This allows callers (DefaultExporter) to
explicitly emit drop events for exhausted retries.
### Documentation
- **Configuration examples** section added to DefaultExporter
documentation with a new "Dropped observability events" subsection
explaining the two drop reasons and providing a `DropAlertExporter`
example that forwards drop events to an external monitoring endpoint
- Reference documentation updated to describe the new drop event types,
callback signature, and metadata included in drop notifications
### Testing
- `ObservabilityBus` tests verify drop event routing to exporters and
bridges, async handler handling, and safe handling of missing
`onDroppedEvent` implementations
- `DefaultExporter` tests extend coverage to verify drop events for
unsupported-storage and retry-exhausted scenarios, including error
details and correct signal/reason values
- `EventBuffer` tests verify that `reAddCreates` and `reAddUpdates`
properly return dropped events when retry limits are exceeded
---
## Documentation Updates
Two changesets document the additions:
- `@mastra/observability`: DefaultExporter now emits drop-event
telemetry to custom exporters and integrations
- `@mastra/core`: New structured drop-event types and `onDroppedEvent`
hook for bridge/exporter integrations
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mohamed BEN HAMDOUNE <mbenhamd@Mohameds-MacBook-Pro.local>
* Fix and feat: add metadata filtering support to semantic recall (Issue #8610) (#9256)
Co-authored-by: Hunter Hagedorn <hunterhagedorn@Hunters-MacBook-Pro-2.local>
Co-authored-by: wardpeet <ward@coding-tech.com>
Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>
* chore: dedupe and clean up external dependencies (#16351)
Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix cache write tokens not being set in AI SDK v6 usage (#16354)
## Description
Fixed an issue where `inputTokenDetails.cacheWriteTokens` was always
`undefined` in the AI SDK v6 usage object. The field now correctly
reflects the prompt cache creation tokens reported by the provider (via
`cacheCreationInputTokens`). Previously, this value was only accessible
through `providerMetadata.anthropic.cacheCreationInputTokens`.
## Related Issue(s)
<!-- Link to the issue(s) this PR addresses, using hashtag notation:
Fixes #123 -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Changes
- Updated `normalizeV6Usage()` helper in
`client-sdks/ai-sdk/src/helpers.ts` to map
`usage.cacheCreationInputTokens` to `inputTokenDetails.cacheWriteTokens`
instead of leaving it undefined
## Checklist
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have addressed all Coderabbit comments on this PR
https://claude.ai/code/session_01Rhzjr9j1dvvo3qein2yeR9
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR fixes a bug where the AI SDK was losing track of how many tokens
were used for cache writing. Instead of that information disappearing,
it now properly captures and stores it in the right place so developers
can see it easily.
## Overview
Fixes cache write token tracking in AI SDK v6 usage objects. The
`inputTokenDetails.cacheWriteTokens` field was previously undefined and
unavailable through the standard API surface; cache token information
was only accessible via the provider-specific metadata. This PR maps the
provider-reported prompt cache creation tokens
(`usage.cacheCreationInputTokens`) directly into
`inputTokenDetails.cacheWriteTokens`.
## Changes
- **Updated `normalizeV6Usage()` in
`client-sdks/ai-sdk/src/helpers.ts`**: Added mapping from
`usage.cacheCreationInputTokens` to `inputTokenDetails.cacheWriteTokens`
to ensure cache write tokens are properly exposed in the standard usage
object.
- **Added changeset**: Documents the fix for the `@mastra/ai-sdk`
package (patch version bump).
## Scope
- 1 line changed in the main code file (`helpers.ts`)
- Metadata update in changeset documentation
- No public API changes
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude <noreply@anthropic.com>
* Add MAPPING span tracing for tool toModelOutput transforms (#16347)
## Description
This PR adds observability tracing for tool `toModelOutput`
transformations by emitting a `MAPPING` span type. When a tool defines a
`toModelOutput` function and it executes successfully, a child span is
created to track the transformation of the tool result before it's
passed to the model.
### Changes Made
1. **New Span Type**: Added `MAPPING` span type to `SpanType` enum in
`observability/types/tracing.ts` to represent inline data transforms
between pipeline stages.
2. **New Attributes Interface**: Created `MappingAttributes` interface
to capture mapping-specific metadata:
- `mappingType`: Identifier of the mapping (e.g., `toModelOutput`)
- `toolCallId`: Associated tool call ID when mapping operates on a tool
result
3. **Span Emission Logic**: Updated `llm-mapping-step.ts` to:
- Create a child `MAPPING` span when `toModelOutput` is defined and the
tool result is non-null
- Capture the tool result as span input and the transformed output as
span output
- Mark the span as errored if `toModelOutput` throws, then re-throw the
error
- Skip span creation if the tool has no `toModelOutput` or if the result
is null/undefined
4. **Comprehensive Tests**: Added four test cases covering:
- Successful mapping span emission with correct attributes and output
- No span emission when tool lacks `toModelOutput`
- No span emission when tool result is null/undefined
- Error handling when `toModelOutput` throws
## Related Issue(s)
Fixes #15486
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have addressed all Coderabbit comments on this PR
https://claude.ai/code/session_018bEDqEtGS2XkdsqBjye6JC
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
When a tool changes its result before the AI model sees it, the system
now records that transformation in traces so you can see both the
original tool output and what the model actually received.
## Overview
This PR adds observability for tool toModelOutput transformations by
introducing a MAPPING span type and emitting a child MAPPING span when a
tool defines toModelOutput and the tool result is non-null. The span
records the raw tool result as input and the transformed model-facing
output as span output, and marks the span errored if the transform
throws. No mapping span is created when toModelOutput is absent or the
tool result is null/undefined.
## Changes
- Added SpanType.MAPPING = 'mapping' and exported MappingAttributes in
packages/core/src/observability/types/tracing.ts to represent inline
data transforms and carry mapping-specific attributes (mappingType,
toolCallId).
- Updated
packages/core/src/loop/workflows/agentic-execution/llm-mapping-step.ts
to:
- create a child MAPPING span (EntityType.TOOL) when tool.toModelOutput
exists and the tool result is non-null,
- set span input to the raw tool result and end the span with the
transformed modelOutput,
- mark the span errored (endSpan: true) and re-throw if toModelOutput
throws,
- avoid creating a span when toModelOutput is undefined or tool result
is null/undefined,
- attach modelOutput into providerMetadata.mastra.modelOutput when
present.
- Added tests in
packages/core/src/loop/workflows/agentic-execution/llm-mapping-step.test.ts
covering:
- successful MAPPING span emission with expected attributes and output,
- no span when the tool lacks toModelOutput,
- no span when tool result is null/undefined,
- error handling when toModelOutput throws (span errored and error
re-thrown).
- Updated client SDK route types
(client-sdks/client-js/src/route-types.generated.ts) to include
'mapping' in relevant union types.
- Documented the change in the changeset (.changeset/loud-ads-double.md)
and regenerated client types as part of the commit.
## Issue Resolution
Addresses issue #15486 by capturing toModelOutput return values in
tracing so traces can distinguish "not invoked" vs "no-op" vs
"transformed" for tool-to-model mappings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(core): use toJSON for requestContext serialization in all workflow paths (#12573)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ward Peeters <ward@coding-tech.com>
* Evented workflow foreach fix (#16358)
## Description
<!-- Provide a brief description of the changes in this PR -->
- Fix evented workflow foreach timing out when payload is an empty array
- Write better test for foreach concurrency
## Related Issue(s)
<!-- Link to the issue(s) this PR addresses, using hashtag notation:
Fixes #123 -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR fixes a bug where evented workflows could get stuck when asked
to loop over an empty list, making the workflow properly treat an empty
list as "done" and continue. It also improves the foreach concurrency
test to verify parallelism reliably.
## Changes Overview
### Bug Fix: Empty Array Handling in Foreach Loops
- File:
packages/core/src/workflows/evented/workflow-event-processor/loop.ts
- processWorkflowForEach now treats a previous result with an empty
output array (prevResult.output.length === 0) as a completed foreach. It
constructs and persists a successful empty result, updates stepResults,
and advances to the next step. The workflowsStore acquisition was moved
earlier to support this flow. This prevents timeouts when the foreach
payload is an empty array.
### Test Updates
- File: packages/core/src/workflows/evented/evented-workflow.test.ts
- Removed skip entries for foreachPartialConcurrencyTiming and
emptyForeach so those tests run for the evented engine.
- File: workflows/_test-utils/src/domains/foreach.ts
- Reworked the "foreach-partial-concurrency" test setup to use a runtime
concurrency tracker (increment/decrement + peakActive) inside the mock
map step instead of wall-clock timing assertions. The test now asserts
concurrencyTracker.peakActive === 2 and keeps output verification;
resetMocks now resets the tracker.
### Changelog
- File: .changeset/silver-hounds-sink.md
- Added a changeset for @mastra/core (patch) documenting the foreach fix
for empty array payloads.
## Type of change
- Bug fix
- Test update
## Checklist (from PR)
- Documentation changes: not marked complete
- Tests added: not marked complete
- Addressed Coderabbit comments: not marked complete
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fix MODEL_INFERENCE span timing to exclude processor work (#16357)
## Description
Fixed `MODEL_INFERENCE` span timing so it accurately measures pure model
latency, excluding input processor and `prepareStep` work. Previously,
the inference span opened from `startStep()`, which the agentic loop
calls before processors run, inflating the span's duration.
The span now opens immediately before the model call via a new
`startInference()` method on the tracker. Chunk-arrival auto-creation
remains as a fallback for callers that don't explicitly call it.
Additionally, `setInferenceContext()` is now applied per-step after
input processors finalize the tool set, so `availableTools` and
`toolChoice` on `MODEL_INFERENCE` spans reflect per-step mutations
(input processors, `prepareStep`, `activeTools` filtering) instead of
the agent-run-level snapshot.
### Changes
- **ModelSpanTracker**: Removed automatic inference span creation from
`startStep()`. Added public `startInference()` method to open the span
immediately before model invocation, snapshotting the latest
`#inferenceContext`.
- **Auto-creation safety net**: Inference spans are auto-created on
first chunk arrival (or step-start chunk) if the caller didn't
explicitly call `startInference()`, ensuring chunks always parent under
`MODEL_INFERENCE` rather than `MODEL_STEP`.
- **LLM execution steps**: Both agentic and durable execution steps now
call `setInferenceContext()` + `startInference()` immediately before the
model call, after input processors and `prepareStep` have completed.
- **Utility function**: Added `getStepAvailableToolNames()` to compute
the post-processor tool set, applying `activeTools` filtering when
present.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Code refactoring
## Checklist
- [x] I have added tests that prove my fix is effective or that my
feature works
- Added 4 new test cases covering: span timing exclusion, inference
context snapshotting, auto-creation on chunk arrival, and explicit
`startInference()` behavior
- Added unit tests for `getStepAvailableToolNames()` utility function
- [x] Existing tests pass
https://claude.ai/code/session_01TURPcy3KRBWnmRV7WuQqun
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR makes the AI model timer more accurate by starting the
model-timing span immediately before the model runs (after input
processing), so measured latency reflects only the model call itself.
---
## Overview
Refactors MODEL_INFERENCE span lifecycle so inference spans are opened
and contextualized right before the model invocation (excluding input
processors and prepareStep work). Introduces an explicit
startInference() API, auto-creation safety-net for streaming chunks,
per-step inference-context snapshotting, and a utility to compute
available tools for a step.
---
## Key Changes
- ModelSpanTracker behavior
- startStep() now opens only MODEL_STEP.
- New public startInference(payload?) opens MODEL_INFERENCE immediately
before model invocation and snapshots the latest inferenceContext.
- Auto-creation safety-net: if chunks (including step-start chunks)
arrive before startInference() is called, the tracker auto-creates
missing MODEL_STEP and MODEL_INFERENCE so chunk spans parent under
MODEL_INFERENCE.
- Inference-context timing
- Agentic and durable execution paths now call setInferenceContext() and
startInference() immediately before the model call (after input
processors and prepareStep), so availableTools, toolChoice, and
responseFormat reflect post-processor mutations.
- Durable execution: responseFormat for inference context is derived
from the actual structuredOutput payload sent to execute(), avoiding
incorrect reporting when structuringModelConfig is used.
- Utility
- Added getStepAvailableToolNames(tools?, activeTools?) which:
- Returns a shallow copy of activeTools whenever activeTools is provided
(including an explicit empty array, allowing explicit disabling of
tools),
- Otherwise returns Object.keys(tools) when tools is provided (may be
[]),
- Otherwise returns undefined.
- Safety & consolidation
- Extracted ensureStepAndInference helper to consolidate auto-create
logic used by chunk handlers and wrapStream (which treats step-start as
a point to open inference if not yet opened).
---
## Public API Changes
- IModelSpanTracker (packages/core/src/observability/types/tracing.ts)
- Added optional startInference?(payload?: StepStartPayload): void with
docs noting it should be called immediately before invoking the model
and that it snapshots inference context and supports fallback
auto-creation.
- ModelSpanTracker (observability/mastra/src/model-tracing.ts)
- Added public startInference(payload?: StepStartPayload): void and
moved related logic into ensureStepAndInference helper.
- New exported utility
- getStepAvailableToolNames(tools?: Record<string, unknown>,
activeTools?: readonly string[]): string[] | undefined
---
## Tests
- Added four new tests in model-tracing.test.ts:
1. startStep() does not create MODEL_INFERENCE
2. startInference() snapshots latest inference context into the emitted
span
3. MODEL_INFERENCE.startTime is recorded at startInference() (excluding
earlier delays)
4. Receiving chunks auto-creates MODEL_INFERENCE if caller never called
startInference()
- Added unit tests for getStepAvailableToolNames covering activeTools
precedence (including explicit []), fallback to tools keys, empty-tool
cases, and undefined behavior.
---
## Files Modified
- .changeset/solid-snakes-double.md
- observability/mastra/src/model-tracing.ts
- observability/mastra/src/model-tracing.test.ts
- packages/core/src/observability/types/tracing.ts
- packages/core/src/observability/utils.ts
- packages/core/src/observability/utils.test.ts
- packages/core/src/agent/durable/workflows/steps/llm-execution.ts
-
packages/core/src/loop/workflows/agentic-execution/llm-execution-step.ts
- packages/core/src/llm/model/model.loop.ts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(observability): route eval scores through ScoreEvent bus (#16185)
## Description
Wires Mastra eval scores through the unified observability bus so every
exporter implementing `onScoreEvent` automatically receives them.
Replaces the deprecated per-exporter `addScoreToTrace` side-channel that
bypassed the pipeline. Also unblocks Mastra spans appearing under
Braintrust experiment eval spans (#11097) as a side effect of the
unified path.
- Producer alignment: `packages/core/src/mastra/hooks.ts` no longer
iterates exporters with `addScoreToTrace`. `MastraScorer.run()` (already
on main from PR #14920) remains the single producer of `ScoreEvent`s
through `mastra.observability.addScore()`.
- Consumer side: implemented `onScoreEvent` on Langfuse
(`LangfuseClient.score.create`), Laminar (`/v1/evaluators/score`),
Braintrust (`logger.logFeedback`), Datadog
(`tracer.llmobs.submitEvaluation`), and LangSmith
(`Client.createFeedback`).
- Backwards compatibility: Langfuse `addScoreToTrace` and Laminar
`_addScoreToTrace` are preserved as `@deprecated` wrappers that forward
to a shared private `submitScore` helper.
- Bug fix in `observability/mastra/src/recorded.ts`: `buildScoreEvent`
now forwards `scorerName` and `targetEntityType` from `ScoreInput` to
`ExportedScore` (without this, exporters silently fell back to
`scorerId`).
- Tests: rewrote core hook test to assert the hook no longer publishes a
`ScoreEvent`; added per-exporter `onScoreEvent` tests; added a
regression test for `buildScoreEvent` field forwarding.
- Seven single-package changesets created (`@mastra/core`,
`@mastra/observability`, `@mastra/langfuse`, `@mastra/laminar`,
`@mastra/braintrust`, `@mastra/datadog`, `@mastra/langsmith`) per the
project's changeset style guide.
## Related Issue(s)
Fixes #10896 (also unblocks #11097)
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
When Mastra assigns an evaluation score, this PR sends that score
exactly once into a single observability "bus" so all
monitoring/exporter integrations (Langfuse, Laminar, Braintrust,
Datadog, LangSmith, etc.) automatically receive it — avoiding
exporter-specific wiring and duplicate deliveries.
## Overview
Routes Mastra evaluation scores through the unified observability
ScoreEvent bus (mastra.observability.addScore) so any exporter
implementing onScoreEvent receives them. MastraScorer.run() remains the
single producer of ScoreEvent; the core scorer hook no longer iterates
exporters to call addScoreToTrace. Per-exporter
addScoreToTrace/_addScoreToTrace behaviors are preserved as @deprecated
wrappers that forward to shared submit helpers where applicable. Fixes a
bug in buildScoreEvent so scorerName and targetEntityType from
ScoreInput are forwarded into emitted ExportedScore.
## Changes by Component
### Core (@mastra/core)
- Removed exporter-iteration and addScoreToTrace republishing from
createOnScorerHook; the hook now only persists scores to the legacy
scores store.
- Ensures MastraScorer.run() is the single producer of ScoreEvent.
- Registers static workflow-step scorers synchronously during
addWorkflow and resolves dynamic scorers asynchronously so scorers are
registered before use.
- Updated core test to assert the hook does not emit ScoreEvent.
### Observability (@mastra/observability)
- Fix: buildScoreEvent (observability/mastra/src/recorded.ts) now
includes scorerName and targetEntityType in emitted ScoreEvent.
- Bus routing: route-event now prefers onScoreEvent; if an exporter
lacks onScoreEvent but implements deprecated addScoreToTrace, the bus
maps the ScoreEvent into the legacy addScoreToTrace payload and calls it
(avoids duplicate delivery when both exist).
- Added unit tests for score routing behavior and the buildScoreEvent
regression.
### Exporter Integrations (new onScoreEvent handlers + tests)
- Langfuse (@mastra/langfuse)
- Added onScoreEvent → LangfuseClient.score.create with a shared
submitScore helper and centralized error logging.
- Preserved addScoreToTrace as a deprecated wrapper forwarding to
submitScore.
- Tests updated to cover onScoreEvent and mark addScoreToTrace
deprecated.
- Laminar (@mastra/laminar)
- Added onScoreEvent delegating to a private submitScore that POSTS to
/v1/evaluators/score.
- submitScore converts trace/span IDs to Laminar/OTel UUIDs, merges
metadata/reason, enforces timeouts via AbortController, and logs
non-OK/timed-out responses.
- Preserved _addScoreToTrace as a @deprecated wrapper forwarding to
submitScore.
- Tests verify POST payload mapping and wrapper behavior.
- Braintrust (@mastra/braintrust)
- Added onScoreEvent that calls logger.logFeedback using spanId (falling
back to traceId) and skips when neither is present.
- Maps scorer name/ID, reason, and metadata into the feedback payload
and logs failures.
- Tests cover span/trace fallback and skip behavior.
- Datadog (@mastra/datadog)
- Added onScoreEvent: validates traceId/spanId, looks up exported
dd-trace span context (traceState), and calls
tracer.llmobs.submitEvaluation with mapped fields
(label/value/metricType/mlApp/timestamp/reason/metadata).
- Drops scores when identifiers or exported span context are missing and
warns when scores arrive before the span is exported.
- Tests cover successful submission, unknown IDs (no-op), and dropping
out-of-order scores.
- LangSmith (@mastra/langsmith)
- Added onScoreEvent that submits Client.createFeedback when a
spanId→LangSmith runId mapping exists.
- Introduced a configurable bounded LRU spanId→runId cache
(runIdCacheMaxEntries) populated from RunTree building; evicts oldest
entries when full and is cleared on shutdown.
- Ensures authoritative scorerId/scoreSource are preserved in sourceInfo
and ignores events when span/run mapping is absent.
- Tests cover mapping semantics, eviction, metadata protection, and
skipping unseen spans.
## Tests
- Core hook test rewritten to assert the hook does not publish
ScoreEvent.
- Per-exporter onScoreEvent test suites added for Langfuse, Laminar,
Braintrust, Datadog, and LangSmith covering field mapping, identifier
fallbacks, ordering/dropping, timeout handling, and cache eviction.
- Bus tests added to assert onScoreEvent preferred and legacy
addScoreToTrace fallback behavior.
- Regression test added to ensure buildScoreEvent forwards scorerName
and targetEntityType.
## Backwards Compatibility
- Deprecated addScoreToTrace/_addScoreToTrace preserved (where present)
as wrappers that forward to the new onScoreEvent/submit helpers to avoid
breaking existing callers.
- Observability interface docs updated to deprecate addScoreToTrace in
favor of onScoreEvent/ScoreEvent bus.
- No breaking public API surface changes beyond documented additions and
deprecations.
## Commits & Release Notes
- Seven package-scoped changesets created for @mastra/core,
@mastra/observability, @mastra/langfuse, @mastra/laminar,
@mastra/langsmith, @mastra/datadog, and @mastra/braintrust documenting
onScoreEvent support and the buildScoreEvent fix.
- Exporter-specific notes documented (Datadog ordering caveat; Laminar
timeout/error logging; LangSmith runId cache behavior).
## Related Issues
Fixes #10896 and unblocks #11097.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Fix flaky test by sorting thread IDs in memory handler test (#16360)
## Description
This PR fixes a flaky test in the memory handlers test suite. The test
was asserting thread IDs in a specific order without guaranteeing that
order from the API response. By sorting the thread IDs before
comparison, the test now reliably passes regardless of the order in
which threads are returned.
no changset required.
## Related Issue(s)
<!-- Link to the issue(s) this PR addresses, if applicable -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
## Test Plan
The existing test in
`packages/server/src/server/handlers/memory.test.ts` now passes
reliably. The change only affects test assertion order and does not
modify any production code behavior.
https://claude.ai/code/session_01RGNgHJmAGDZM8xEsSA7GW1
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
A test was failing unpredictably because it expected thread IDs in a
specific order, but the API doesn't guarantee a consistent order. The
fix sorts the thread IDs before comparing them, making the test
reliable.
## Changes
Updated the `Authorization - Reserved Context Keys` test in the memory
handlers test suite to make thread ID assertions order-independent by
sorting the returned thread IDs before comparison. This eliminates
flakiness caused by the `filterAccessible` FGA filter not guaranteeing
stable return order.
**Files changed:**
- `packages/server/src/server/hand…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds scorer observability and prepares scorer results for the new score-storage model.
The main change is that scorer execution is now traced for the first time.
scorer.run()createsSCORER_RUNspans, scorer pipeline steps createSCORER_STEPspans, and scorer results emit scores throughmastra.observability.addScore()when a target trace is available. This also adds clearer score metadata for scorer name, target entity type, target scope, and scorer trace links.This PR keeps the legacy scores-store write path in place during the transition, but marks that path as deprecated and keeps the new observability score emission as the future direction.
Related Issue(s)
Type of Change
Checklist
Summary by CodeRabbit
New Features
Improvements
Tests
Chores