Skip to content

Commit d468acb

Browse files
devin-ai-integration[bot]roaminroabhiaiyer91
authored
fix: ConsoleLogger.warn() severity + add rate-limit sleep span (#17623)
## Description Fixes two bugs surfaced in a community thread investigating high latency (~67s wall-clock) in a 10-step agent: 1. **`ConsoleLogger.warn()` called `console.info()` instead of `console.warn()`** — warn-level logs were misclassified as info/stdout, making rate-limit warnings invisible in log backends filtering by severity. 2. **Rate-limit sleep had no OTel span** — the hardcoded 10s `delay()` when `x-ratelimit-remaining-tokens < 2000` only emitted a warn log (which was invisible per bug #1). Added a `rate-limit-sleep` child span with `{ remainingTokens, delayMs }` metadata so the sleep is visible in traces. 3. **Fixed lockfile resolution** — `agent-sdks/openai` had a stale `eslint@10.3.0` lockfile entry after a merge; updated to `10.4.1`. ### Changes - `packages/_internal-core/src/logger/index.ts`: `console.info()` → `console.warn()` in `ConsoleLogger.warn()` - `packages/core/src/llm/model/model.ts`: Added `rate-limit-sleep` span in both `__text` and `__stream` paths - `packages/core/src/llm/model/model.loop.ts`: Added `rate-limit-sleep` span in agentic loop path - `packages/core/src/logger/console-logger.test.ts`: Updated 3 tests to assert `console.warn` - `packages/core/src/llm/model/model.test.ts`: New test verifying span creation with correct name/metadata when rate-limit fires - `observability/sentry/src/tracing.test.ts`: Updated test that was codifying the same `console.info` bug - `pnpm-lock.yaml`: Fixed stale `eslint@10.3.0` → `10.4.1` for `agent-sdks/openai` ## Related issue(s) Community-reported latency investigation (Slack thread). No GitHub issue. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [x] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR Link to Devin session: https://app.devin.ai/sessions/ce59a3645593455aa14f4397eda972d7 Requested by: @roaminro <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## ELI5 This PR fixes two observability problems: warning messages were being sent to the wrong console stream, and a repeated 10-second pause used for rate-limiting was invisible in traces—both are now fixed so logs and traces accurately show warnings and delay time. --- ## Overview This PR includes two fixes from a latency investigation: 1. ConsoleLogger.warn() now calls console.warn() (previously console.info()), so warn-level logs are classified correctly by log backends. 2. The hardcoded 10s rate-limit backpressure sleep (triggered when x-ratelimit-remaining-tokens < 2000) is now wrapped in a "rate-limit-sleep" OpenTelemetry child span with attributes remainingTokens and delayMs and the warn log now includes remainingTokens. --- ## Changes - packages/_internal-core/src/logger/index.ts - ConsoleLogger.warn() uses console.warn() instead of console.info(). - packages/core/src/llm/model/model.ts - In MastraLLMV1.__text and __stream: create a "rate-limit-sleep" child span around the 10s sleep, record remainingTokens and delayMs, end the span after sleep, and include remainingTokens in the warn message. - packages/core/src/llm/model/model.loop.ts - In MastraLLMVNext.stream (onStepFinish): add the same "rate-limit-sleep" child span and warn message including remainingTokens. - Tests - packages/core/src/logger/console-logger.test.ts: updated assertions to expect console.warn for warn-level logs. - observability/sentry/src/tracing.test.ts: updated to spy on console.warn for DSN-missing warning. - packages/core/src/llm/model/model.test.ts: added regression test verifying creation and lifecycle of the rate-limit-sleep span (attributes remainingTokens and delayMs), the warn log, and that the span ends after the 10s fake-timer sleep. - Documentation/changesets - .changeset/* files updated to document the ConsoleLogger fix and rate-limit span instrumentation. --- ## Impact - Warn-level logs will be routed and classified correctly by logging backends. - Rate-limit backpressure delays are now visible in distributed traces as "rate-limit-sleep" spans containing remainingTokens and delayMs, aiding latency investigations where repeated 10s sleeps caused unaccounted trace time. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Romain <97863888+roaminro@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
1 parent f5519fc commit d468acb

8 files changed

Lines changed: 121 additions & 12 deletions

File tree

.changeset/all-hounds-doubt.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@mastra/core': patch
3+
---
4+
5+
Fixed ConsoleLogger.warn() to correctly call console.warn() instead of console.info(), ensuring warn-level logs are routed to stderr and properly classified by log backends (e.g. Datadog)

.changeset/lazy-rings-punch.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@mastra/core': patch
3+
---
4+
5+
Added OTel span instrumentation for the hardcoded 10-second rate-limit backpressure sleep, making it visible in traces as a 'rate-limit-sleep' span with remainingTokens and delayMs metadata

observability/sentry/src/tracing.test.ts

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,7 @@ describe('SentryExporter', () => {
105105
const originalEnv = process.env.SENTRY_DSN;
106106
delete process.env.SENTRY_DSN;
107107

108-
// ConsoleLogger uses console.info for warn level logging
109-
const mockConsoleInfo = vi.spyOn(console, 'info').mockImplementation(() => {});
108+
const mockConsoleWarn = vi.spyOn(console, 'warn').mockImplementation(() => {});
110109

111110
// Clear mock to isolate this test from previous Sentry.init calls
112111
SentryMock.init.mockClear();
@@ -115,13 +114,13 @@ describe('SentryExporter', () => {
115114

116115
expect(exporterWithoutDsn.name).toBe('sentry');
117116
// Verify warning was logged about missing DSN
118-
expect(mockConsoleInfo).toHaveBeenCalledWith(expect.stringContaining('DSN'));
117+
expect(mockConsoleWarn).toHaveBeenCalledWith(expect.stringContaining('DSN'));
119118
// Verify exporter is disabled
120119
expect(exporterWithoutDsn.isDisabled).toBe(true);
121120
// Should not have initialized Sentry without DSN
122121
expect(SentryMock.init).not.toHaveBeenCalled();
123122

124-
mockConsoleInfo.mockRestore();
123+
mockConsoleWarn.mockRestore();
125124
process.env.SENTRY_DSN = originalEnv;
126125
});
127126

packages/_internal-core/src/logger/index.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ export class ConsoleLogger extends MastraLogger {
297297
(this.level === LogLevel.WARN || this.level === LogLevel.INFO || this.level === LogLevel.DEBUG) &&
298298
this.shouldLog(LogLevel.WARN, message, args)
299299
) {
300-
console.info(`${this.prefix()}${message}`, ...args);
300+
console.warn(`${this.prefix()}${message}`, ...args);
301301
}
302302
}
303303

packages/core/src/llm/model/model.loop.ts

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -270,8 +270,14 @@ export class MastraLLMVNext extends MastraBase {
270270

271271
const remainingTokens = parseInt(props?.response?.headers?.['x-ratelimit-remaining-tokens'] ?? '', 10);
272272
if (!isNaN(remainingTokens) && remainingTokens > 0 && remainingTokens < 2000) {
273-
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId });
273+
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId, remainingTokens });
274+
const rateLimitSpan = modelSpan?.createChildSpan({
275+
name: 'rate-limit-sleep',
276+
type: SpanType.GENERIC,
277+
metadata: { remainingTokens, delayMs: 10_000 },
278+
});
274279
await delay(10 * 1000);
280+
rateLimitSpan?.end();
275281
}
276282
},
277283

packages/core/src/llm/model/model.test.ts

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -877,4 +877,78 @@ describe('MastraLLM', () => {
877877
expect(errorMastra.logger.error).toHaveBeenCalled();
878878
});
879879
});
880+
881+
describe('rate-limit span instrumentation', () => {
882+
it('should create a rate-limit-sleep span when remaining tokens are below threshold in __text', async () => {
883+
const rateLimitMastra = {
884+
logger: {
885+
debug: vi.fn(),
886+
warn: vi.fn(),
887+
info: vi.fn(),
888+
error: vi.fn(),
889+
trackException: vi.fn(),
890+
} as any,
891+
};
892+
893+
const rateLimitModel = new MockLanguageModelV1({
894+
doGenerate: async () => ({
895+
rawCall: { rawPrompt: null, rawSettings: {} },
896+
finishReason: 'stop' as const,
897+
usage: { promptTokens: 10, completionTokens: 20 },
898+
text: 'hello',
899+
rawResponse: { headers: { 'x-ratelimit-remaining-tokens': '1500' } },
900+
}),
901+
doStream: async () => {
902+
throw new Error('not used');
903+
},
904+
});
905+
906+
const llm = new MastraLLMV1({ model: rateLimitModel });
907+
llm.__registerPrimitives(rateLimitMastra);
908+
909+
const mockRateLimitSpan = { end: vi.fn() };
910+
const mockLlmSpan = {
911+
createChildSpan: vi.fn().mockReturnValue(mockRateLimitSpan),
912+
end: vi.fn(),
913+
error: vi.fn(),
914+
update: vi.fn(),
915+
executeInContext: vi.fn(async (fn: any) => fn()),
916+
executeInContextSync: vi.fn((fn: any) => fn()),
917+
};
918+
const mockCurrentSpan = {
919+
createChildSpan: vi.fn().mockReturnValue(mockLlmSpan),
920+
};
921+
922+
const tracingCtx = {
923+
tracingContext: { currentSpan: mockCurrentSpan },
924+
};
925+
926+
// Use fake timers so the 10s delay completes instantly
927+
vi.useFakeTimers();
928+
929+
const textPromise = llm.__text({
930+
messages: [{ role: 'user', content: 'test' }],
931+
requestContext: new RequestContext(),
932+
...tracingCtx,
933+
});
934+
935+
// Advance past the 10s delay
936+
await vi.advanceTimersByTimeAsync(11_000);
937+
await textPromise;
938+
939+
vi.useRealTimers();
940+
941+
expect(rateLimitMastra.logger.warn).toHaveBeenCalledWith(
942+
'Rate limit approaching, waiting 10 seconds',
943+
expect.objectContaining({ remainingTokens: 1500 }),
944+
);
945+
expect(mockLlmSpan.createChildSpan).toHaveBeenCalledWith(
946+
expect.objectContaining({
947+
name: 'rate-limit-sleep',
948+
metadata: { remainingTokens: 1500, delayMs: 10_000 },
949+
}),
950+
);
951+
expect(mockRateLimitSpan.end).toHaveBeenCalled();
952+
});
953+
});
880954
});

packages/core/src/llm/model/model.ts

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -257,8 +257,14 @@ export class MastraLLMV1 extends MastraBase {
257257

258258
const remainingTokens = parseInt(props?.response?.headers?.['x-ratelimit-remaining-tokens'] ?? '', 10);
259259
if (!isNaN(remainingTokens) && remainingTokens > 0 && remainingTokens < 2000) {
260-
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId });
260+
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId, remainingTokens });
261+
const rateLimitSpan = llmSpan?.createChildSpan({
262+
name: 'rate-limit-sleep',
263+
type: SpanType.GENERIC,
264+
metadata: { remainingTokens, delayMs: 10_000 },
265+
});
261266
await delay(10 * 1000);
267+
rateLimitSpan?.end();
262268
}
263269
},
264270
experimental_output: schema
@@ -608,8 +614,14 @@ export class MastraLLMV1 extends MastraBase {
608614

609615
const remainingTokens = parseInt(props?.response?.headers?.['x-ratelimit-remaining-tokens'] ?? '', 10);
610616
if (!isNaN(remainingTokens) && remainingTokens > 0 && remainingTokens < 2000) {
611-
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId });
617+
this.logger.warn('Rate limit approaching, waiting 10 seconds', { runId, remainingTokens });
618+
const rateLimitSpan = llmSpan?.createChildSpan({
619+
name: 'rate-limit-sleep',
620+
type: SpanType.GENERIC,
621+
metadata: { remainingTokens, delayMs: 10_000 },
622+
});
612623
await delay(10 * 1000);
624+
rateLimitSpan?.end();
613625
}
614626
},
615627
onFinish: async props => {

packages/core/src/logger/console-logger.test.ts

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,15 @@ describe('ConsoleLogger', () => {
2222

2323
// Verify by checking the child only logs at WARN level (inherited from parent)
2424
const infoSpy = vi.spyOn(console, 'info').mockImplementation(() => {});
25+
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
2526
child.info('should not log');
2627
child.warn('should log');
2728

28-
expect(infoSpy).toHaveBeenCalledTimes(1);
29-
expect(infoSpy).toHaveBeenCalledWith('[AGENT] should log');
29+
expect(infoSpy).not.toHaveBeenCalled();
30+
expect(warnSpy).toHaveBeenCalledTimes(1);
31+
expect(warnSpy).toHaveBeenCalledWith('[AGENT] should log');
3032
infoSpy.mockRestore();
33+
warnSpy.mockRestore();
3134
});
3235

3336
it('inherits filter from parent', () => {
@@ -113,6 +116,7 @@ describe('ConsoleLogger', () => {
113116
const logger = new ConsoleLogger({ level: LogLevel.DEBUG, filter });
114117

115118
const infoSpy = vi.spyOn(console, 'info').mockImplementation(() => {});
119+
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
116120
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
117121

118122
logger.debug('d');
@@ -126,6 +130,7 @@ describe('ConsoleLogger', () => {
126130
expect(filter).toHaveBeenCalledWith(expect.objectContaining({ level: LogLevel.ERROR }));
127131

128132
infoSpy.mockRestore();
133+
warnSpy.mockRestore();
129134
errorSpy.mockRestore();
130135
});
131136

@@ -188,19 +193,22 @@ describe('ConsoleLogger', () => {
188193
const logger = new ConsoleLogger({ level: LogLevel.WARN });
189194

190195
const infoSpy = vi.spyOn(console, 'info').mockImplementation(() => {});
196+
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
191197
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
192198

193199
logger.debug('debug');
194200
logger.info('info');
195201
logger.warn('warn');
196202
logger.error('error');
197203

198-
expect(infoSpy).toHaveBeenCalledTimes(1); // only warn
199-
expect(infoSpy).toHaveBeenCalledWith('warn');
204+
expect(infoSpy).not.toHaveBeenCalled();
205+
expect(warnSpy).toHaveBeenCalledTimes(1);
206+
expect(warnSpy).toHaveBeenCalledWith('warn');
200207
expect(errorSpy).toHaveBeenCalledTimes(1);
201208
expect(errorSpy).toHaveBeenCalledWith('error');
202209

203210
infoSpy.mockRestore();
211+
warnSpy.mockRestore();
204212
errorSpy.mockRestore();
205213
});
206214
});

0 commit comments

Comments
 (0)