fix(aws-lambda): force placeholder resource so the Lambda Extension drops dd-tracer-serverless-span#11313
fix(aws-lambda): force placeholder resource so the Lambda Extension drops dd-tracer-serverless-span#11313zarirhamza wants to merge 1 commit intomasterfrom
Conversation
…rops dd-tracer-serverless-span The Datadog Lambda Extension filters the placeholder invocation span by comparing `span.resource == "dd-tracer-serverless-span"` (see `bottlecap/src/traces/trace_processor.rs::filter_span_from_lambda_library_or_runtime`). The intent is that the extension drops the placeholder so the inferred `aws.lambda` span (which the end-invocation handshake gives the same `span_id`) is the surviving record under that key in the trace store and remains parented to the inferred apigateway root. In practice the HTTP/JAX-RS instrumentation overwrites the placeholder's resource with the request route (e.g. "POST /") at `HTTP_FRAMEWORK_ROUTE` priority during the invocation, so the extension's filter no longer matches at end-of-invocation. Both records (the placeholder with `parent_id=0`, and the inferred `aws.lambda` with `parent_id=apigateway.span_id`) reach the backend under the same `(trace_id, span_id)` key. The trace store keeps the placeholder, and the rest of the trace is detached from the inferred apigateway root. Fix: in `LambdaHandlerInstrumentation.exit`, force the resource name back to the literal placeholder marker right before `span.finish()`, using `ResourceNamePriorities.MANUAL_INSTRUMENTATION` so the override beats whatever HTTP-framework priority the in-flight instrumentation has written. Verified end-to-end on Quarkus 3.15.4 / Java 21 / Lambda Extension v96: pre-fix the placeholder leaks to the backend with `parent_id=0` and the trace store reports orphans; post-fix zero `dd_tracer_serverless_span` rows reach the backend and the trace store reports a single 11-span chunk with no orphans. Refs: SLES-2837
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf170244a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // ignores writes whose priority is below the current resource priority, | ||
| // and the HTTP/JAX-RS instrumentation will already have written | ||
| // HTTP_FRAMEWORK_ROUTE (3) by this point. | ||
| span.setResourceName(INVOCATION_SPAN_NAME, ResourceNamePriorities.MANUAL_INSTRUMENTATION); |
There was a problem hiding this comment.
Overwriting the resource unconditionally at method exit with MANUAL_INSTRUMENTATION can clobber a resource name that application code intentionally set on the active Lambda span with the same priority during handler execution. AgentSpan#setResourceName uses last-write-wins for equal priority, so this line will replace those user values every time, which regresses resource naming whenever the placeholder span is ultimately emitted (for example if extension communication fails or is disabled). Consider only resetting when the current priority is framework-level (e.g., <= HTTP_FRAMEWORK_ROUTE) rather than overriding manual-level values.
Useful? React with 👍 / 👎.
1 participant
Summary
The Datadog Lambda Extension drops the placeholder invocation span by comparing
span.resource == "dd-tracer-serverless-span"(bottlecap/src/traces/trace_processor.rs::filter_span_from_lambda_library_or_runtime). The dedup is what lets the inferredaws.lambdaspan (which the end-invocation handshake gives the samespan_id) be the surviving record under that key in the trace store, parented to the inferred apigateway root.In practice the HTTP/JAX-RS instrumentation overwrites the placeholder's resource with the request route ("POST /") at
HTTP_FRAMEWORK_ROUTEpriority during the invocation, so by end-of-invocation the extension's filter no longer matches. Both records — the placeholder withparent_id=0and the inferredaws.lambdawithparent_id=apigateway.span_id— reach the backend under the same(trace_id, span_id)key. The trace store keeps the placeholder'sparent_id=0, detaching the rest of the trace from the inferred apigateway root.Fix
In
LambdaHandlerInstrumentation.exit, force the resource name back toINVOCATION_SPAN_NAME("dd-tracer-serverless-span") right beforespan.finish(), usingResourceNamePriorities.MANUAL_INSTRUMENTATION(6) so the override beats whatever priority the in-flight HTTP/JAX-RS instrumentation has written (HTTP_FRAMEWORK_ROUTEis 3).Files touched:
LambdaHandlerInstrumentation.java(+16/-0): importResourceNamePriorities, set the resource name in the exit advice.Test plan
LambdaHandlerInstrumentationTestwith a case that simulates HTTP-framework instrumentation overwritingresourceand asserts the final serialized span has resource"dd-tracer-serverless-span".parent_id=0(/api/v1/trace/<id>shows orphans,operation_name:dd_tracer_serverless_spanrows present in the spans index). Post-fix the spans index returns 0 placeholder rows and the trace store reports a single 11-span chunk with no orphans.Refs
SLES-2837
Notes
This complements an extension-side change (broaden the filter in
bottlecap/src/traces/trace_processor.rsto also checkspan.name) which I plan to file as a separate PR againstDataDog/datadog-lambda-extension. Either side alone closes the gap; both together is defense in depth.