Skip to content

Fixing workflow tracing + adding observability integration testing - #1870

Merged
WhitWaldo merged 6 commits into
masterfrom
workflow-tracing
Jul 12, 2026
Merged

WhitWaldo merged 6 commits into
masterfrom
workflow-tracing

Conversation

@WhitWaldo

Copy link
Copy Markdown
Contributor

Description

With the generous assistant in pinpointing the issue of otel tracing in Dapr.Workflows, this PR applies a patch to make such tracing automatic as part of the outbound gRPC calls.

But because this has been a recurring ticket, it also expands the Dapr.Testcontainers implementation to add a Zipkin endpoint to pull the telemetry from the Dapr runtime and validate that the tracing encompasses the appropriate spans for more than a dozen of the common Workflow scenarios to ensure that not only does this patch work, but to also prevent future regression.

This only targets Workflows because it's the only part of the SDK that shares work with the runtime. All other packages either defer entirely to the runtime or simply don't need their own tracing (preferring that the developer bring their own and auto-register transport, for example).

Once support for head-less workflows is implemented (as is planned for the 1.19 release), this will be expanded to accommodate that scenario as well.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #1867

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

…ng to `Dapr.Testcontainers` to more properly validate it.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…emetry from the Zipkin endpoint on the Dapr runtime to add to the integration testing suite

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…ure across various scenarios:

- Activity runtime-to-SDK span continuity
- Child workflow activity tracing
- Retry attempts
- Fan-out/fan-in parallel activities
- Timer plus external event before activity execution
- Multi-app remote activity
- Activity failure span status
- Continue-as-new
- Multi-app child workflow
- Downstream custom ActivitySource parented under the workflow activity span
- Suspend/resume (validate activity after resume)
- Re-run workflow (validating spans for rerun instance)
- History propagation with OwnHistory
- History propagation with Lineage
- Parallel child workflows, validating each child activity trace
- Terminate (validating for work completed before termination and terminated history)

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
… tracing:

- Caught activity failure/compensation (validate activity span is marked error
- Workflow failure after activity failure (validate trace when activity failure is not caught and workflow ends as failed)
- Child workflow + activity in both parent and child (validates activity in the child)

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…ith the expected dispaly name, the span has the same trace id as the runtime span, the span's ParentSpanId equals the runtime span Id and that the runtime exports a Zipkin activity span with the expected runtime name

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…ble accurate scenarios and requirements

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
@WhitWaldo WhitWaldo added this to the v1.19 milestone Jul 12, 2026
@WhitWaldo WhitWaldo self-assigned this Jul 12, 2026
@WhitWaldo
WhitWaldo requested review from a team as code owners July 12, 2026 19:59
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 4.21687% with 159 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.17%. Comparing base (f286ecb) to head (726dff8).

Files with missing lines Patch % Lines
...Testcontainers/Telemetry/DaprTelemetryCollector.cs 0.00% 107 Missing ⚠️
...c/Dapr.Testcontainers/Harnesses/WorkflowHarness.cs 0.00% 25 Missing ⚠️
.../Dapr.Testcontainers/Telemetry/CapturedActivity.cs 0.00% 12 Missing ⚠️
src/Dapr.Testcontainers/Telemetry/ZipkinSpan.cs 0.00% 10 Missing ⚠️
...rc/Dapr.Testcontainers/Telemetry/ZipkinEndpoint.cs 0.00% 4 Missing ⚠️
src/Dapr.Common/DaprClientUtilities.cs 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1870      +/-   ##
==========================================
- Coverage   63.19%   62.17%   -1.02%     
==========================================
  Files         330      334       +4     
  Lines        9534     9700     +166     
  Branches     1110     1127      +17     
==========================================
+ Hits         6025     6031       +6     
- Misses       3197     3356     +159     
- Partials      312      313       +1     
Flag Coverage Δ
net10.0 62.13% <4.21%> (-1.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@WhitWaldo
WhitWaldo merged commit 633fdc1 into master Jul 12, 2026
441 of 446 checks passed
@WhitWaldo
WhitWaldo deleted the workflow-tracing branch July 12, 2026 22:11
WhitWaldo added a commit that referenced this pull request Jul 13, 2026
…1873)

* Fixing workflow tracing + adding observability integration testing (#1870)

* Implementing better trace support in Dapr.Workflow + adding e2e testing to `Dapr.Testcontainers` to more properly validate it.
* Added functionality to `Dapr.Testcontainers` to support capturing telemetry from the Zipkin endpoint on the Dapr runtime to add to the integration testing suite
* Added a series of integration tests to validate proper telemetry capture across various scenarios:
- Activity runtime-to-SDK span continuity
- Child workflow activity tracing
- Retry attempts
- Fan-out/fan-in parallel activities
- Timer plus external event before activity execution
- Multi-app remote activity
- Activity failure span status
- Continue-as-new
- Multi-app child workflow
- Downstream custom ActivitySource parented under the workflow activity span
- Suspend/resume (validate activity after resume)
- Re-run workflow (validating spans for rerun instance)
- History propagation with OwnHistory
- History propagation with Lineage
- Parallel child workflows, validating each child activity trace
- Terminate (validating for work completed before termination and terminated history)
- Caught activity failure/compensation (validate activity span is marked error
- Workflow failure after activity failure (validate trace when activity failure is not caught and workflow ends as failed)
- Child workflow + activity in both parent and child (validates activity in the child)

* Corrected to validate that the `Dapr.Workflow` activity span exists with the expected display name, the span has the same trace id as the runtime span, the span's ParentSpanId equals the runtime span Id and that the runtime exports a Zipkin activity span with the expected runtime name

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>

* Improved existing orchestration tests to make them more closely resemble accurate scenarios and requirements

---------

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>

* Added mechanism to pass environment variables into daprd as part of integration testing with `Dapr.Testcontainers` (#1871)

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>

---------

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure Dapr Workflow activity trace context is available out of the box

1 participant