Span's event specification

[tonyredondo]

According the specification and proto file an event can have attributes as a key:value pairs, where the keys must be strings and the values may be string, booleans or numeric type.

I'm worried about the values types, the Opentracing specification leave this value types open as an "object", allowing the creation of events like error which have an object attribute error.object with the actual exception instance of the language as seen in the semantic conventions document.

Also we can use this object type to instrument loggers supporting structured events.

So i have two questions. How can I handle this kind of usage with the OpenTelemetry api? How the OpenTracing shim is going to work on those cases?

[SergeyKanzhelev]

loggers have not been designed yet. The idea behind Event is lightweight time mark in the span. Default limit on number of events is very aggressive to use it for logs.

With the assumption that logging API will be built separately and Events are used for simple time marks - do you still have a scenario when objects needs to be supported?

[austinlparker]

I think the way the shim would handle things like this would be turning span logs into OTel logs and creating a reference between the span and the log.

[tonyredondo]

@SergeyKanzhelev The common use case is the error event as the OpenTracing semantic conventions allows. In our case we are sending the exception object in the error.object field. We are planning to send other events with custom objects linked to a span.

If this can be handle by the logging api, I would be very interested to read about that api, even participate on those discussions.
Is this api will be available in the OTel first release? Any roadmap on that?

@austinlparker When you say creating a reference between the span and the log, are you planning that the logs are not going to be inside the span but rather in another collection?

[austinlparker]

I think it'd have to be... maybe @carlosalberto has some thoughts, since I know he's been working on the shim for Java, so he's probably closer to dealing with this in practice than I am right now.

[bogdandrutu]

@tonyredondo can you tell me how the backend that you use consumes these Objects?

[yurishkuro]

I second @tonyredondo concern. In OpenTracing the tracer decides how to marshal attributes passed as objects, usually by recognizing certain common types. Limiting the API to primitive values only means that the user code must do the marshaling, which is very inefficient if only <1% of spans are sampled.

Here's an example from Jaeger where an attribute of a span.log (aka event) is an object

or a stack trace

[bogdandrutu]

@yurishkuro nice example.

The current OpenTelemetry API allows to add lazy formatted events via the addEvent(Event) API. So marshaling happens only if the Span is serialized, but indeed needs to produce primitives as the end result.

If one type is important to support I think we should add support for it in our data model and allow users to explicitly set on critical path or delayed that type.

Also there is a discussion to support Exception/Stacktraces via the Status API.

[tonyredondo]

@bogdandrutu we have a custom tracer implementation that splits the span and logs into two collections, serialize it with msgpack and send the payload to our backend, where we store the log data inside a postgresql json column type.

The common types are object exceptions and structured data from loggers (so we don't know the schema). We are planning to add others events type.

Here is an example for structured data:

Here for exceptions:

[jmacd]

Added this issue #76 about supporting structured values, which would IMO be a better way to represent resources.

[carlosalberto]

@carlosalberto has some thoughts, since I know he's been working on the shim for Java, so he's probably closer to dealing with this in practice than I am right now.

So currently the OT Shim tries to convert to one of the base types (numerical, boolean), and fallbacks to String, which clearly is not enough, based on this issue.

I'm wondering if adding Object as a supported type in the API side is (for AttributeValue) would be enough...

[austinlparker]

I'm bumping this as another note of support for either changing the spec to accept object or something like it - as an example for what it looks like, we recently merged a PR to opentelemetry-dotnet that refactored the interface for attribute values into accepting a struct KeyValuePair<string, object> -- https://github.com/open-telemetry/opentelemetry-dotnet/blob/2183c425e80abcebd77cb2db6e1f1763d498c554/src/OpenTelemetry/Trace/Span.cs#L164

The advantage of this (in .NET at least) is that type information is preserved so we're able to use more idomatic methods of converting Attributes to whatever format we need for the exporter (see https://github.com/open-telemetry/opentelemetry-dotnet/blob/8aae2c02f1760668734d99e848ca780e7635b578/src/OpenTelemetry.Exporter.Ocagent/Implementation/SpanDataExtensions.cs#L108 for an example).

Obviously this pattern isn't going to work in every language, but supporting object would be very useful overall.