Error viewing traces with duplicate spans (yep)

[lmolkova]

Azure functions does a trick faking an activity with the same id in the worker - https://github.com/Azure/azure-functions-dotnet-worker/blob/2b83c355eaa21f508f9ff7a46bdc8e953e592527/src/DotNetWorker.Core/FunctionsApplication.cs#L80 (there are reasons).

Which in some cases results in two spans with the same trace id and span id.

The dashboard does not like it

Here the two top-level spans ("eventhubs process" and "InvokeFunctionAsync") have the same id, and the trees under them consist of the same spans shown twice.

blazor.web.js:1 [2024-10-23T01:47:34.335Z] Error: System.InvalidOperationException: More than one sibling of component 'Microsoft.FluentUI.AspNetCore.Components.FluentDataGridRow`1[Aspire.Dashboard.Model.Otlp.SpanWaterfallViewModel]' has the same key value, '45580ca3e66f43c6'. Key values must be unique.
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.ThrowExceptionForDuplicateKey(Object key, RenderTreeFrame& frame)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.BuildKeyToInfoLookup(DiffContext diffContext, Int32 oldStartIndex, Int32 oldEndIndexExcl, Int32 newStartIndex, Int32 newEndIndexExcl)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& diffContext, Int32 oldStartIndex, Int32 oldEndIndexExcl, Int32 newStartIndex, Int32 newEndIndexExcl)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForFramesWithSameSequence(DiffContext& diffContext, Int32 oldFrameIndex, Int32 newFrameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& diffContext, Int32 oldStartIndex, Int32 oldEndIndexExcl, Int32 newStartIndex, Int32 newEndIndexExcl)
   at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment, Exception& renderFragmentException)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()

The scenario is not a mainstream and not valid, but dashboard could handle it more gracefully. It just shows an error and does not let me expand anything.

[adamint]

Maybe the key should instead be {spanId}-{index}
edit: we can't include the row index, unfortunately. We should just remove the key

[JamesNK]

@lmolkova What do you suggest for how Aspire handles duplicate span IDs?

I think some code changes I made in the last couple of days will now reject the duplicate span when received over OTLP.

What do other systems do in this situation?

[lmolkova]

chatted offline, just for the record:

• Jaeger de-duplicates
  

• AzMon drops or rejects the one that came last (the other one does not show up in queries or on the UX)
  

Given that scenario is so edge-case, the only thing I would change is handling the error in some way so user can keep browsing valid parts of the telemetry.

[JamesNK]

Given that scenario is so edge-case, the only thing I would change is handling the error in some way so user can keep browsing valid parts of the telemetry.

Definitely.

New Aspire behavior will be to reject duplicate spans. You'll still be able to view the trace and first span with the id.

AzMon drops or rejects the one that came last (the other one does not show up in queries or on the UX)

To be honest it's hard to tell what is going on here. From the screen shot, it looks like AzMon is showing the second duplicate span. In other words, it's replacing rather than rejecting. i.e. InvokeFunctionAsync is second span in Jeager UI and dashboard (however we order by received date) but is the only one shown in AzMon.

Although reordering might be going on based dates. I don't know. It's hard to say.

[JamesNK]

This looks like a new change in functions: Azure/azure-functions-dotnet-worker#2765. I hope they're planning to fix it.