[logs] Add semantic conventions for durable identifiers

[CodeBlanch]

We have a need across Microsoft to support durable identifiers in logging. A durable identifier could be just an "id" or it could be an "id" + "name". This should be uniform across languages (C++, .NET, Rust, etc.) so that backends can be coded with rules to act on specific log messages.

Definitions

• id:
  A durable numerical identifier, that can be used to differentiate logs from each other. This is not guaranteed to be unique in anyway. This could be used for scenarios such as filtering based on identifier, triggering actions (for example, when a log with a given eventId is fired, take a memory dump) etc.

• name:
  Short event identifier that does not contain varying parts. Name describes what happened (e.g. "ProcessStarted"). Recommended to be no longer than 50 characters. Not guaranteed to be unique in any way. Typically used for filtering and grouping purposes in backends.

Proposal

I am going to propose 2 ways to go about this:

Proposal A

Add new fields on General log identification attributes: https://github.com/open-telemetry/semantic-conventions/compare/main...CodeBlanch:logs-id-and-name?expand=1

I think this is a nice fit.

Proposal B

Loosen the events semantic conventions for event.domain & event.name and add event.id: https://github.com/open-telemetry/semantic-conventions/compare/main...CodeBlanch:events-id?expand=1

I don't think this is a particularly nice fit as these are logs and not "events" as the working group originally intended, however it seems some languages have gone down this road:

• .NET: Add ability to export attributes corresponding to Logrecord.EventId opentelemetry-dotnet#4925
• Rust: According to @cijothomas they are currently planning this support.
• C++: https://github.com/open-telemetry/opentelemetry-cpp/blob/main/api/include/opentelemetry/logs/event_id.h

/cc @alanwest @reyang @trask @utpilla @vishweshbankwar @rajkumar-rangaraj @cijothomas @jack-berg @ThomsonTan @MSNev

[MSNev]

Loosen the events semantic conventions for event.domain & event.name and add event.id

While I "like" the idea of an id this is completely different from what the event.domain and event.name are for.

The event.domain and event.name ARE REQUIRED to identify that the contents of the payload conforms to the semantic conventions as defined by the domain / name combination.

And looking at a Log definition isn't there already a log.record.uid which can be used as a durable identifier???

In my opinion what you are trying to do "should" really be it's own event so something like

event.domain: "DotNet"
event.name: "LogEvent"

With the payload is then

"event.data":
{
    key: "",
    categoryName: "",
    source: "",
    id: "",
    data: "",
    message: ... // etc
} 

[cijothomas]

And looking at a Log definition isn't there already a log.record.uid which can be used as a durable identifier???

Thats does not look durable...

[CodeBlanch]

log.record.uid seems to be unique to every log written so I don't think it is a good fit for this.

[jack-berg]

Is this durable id essentially just a numeric representation of event.domain + event.name? Sort of like a hashcode / checksum applied to the event domain / event name combo, which uniquely identify the category / type of thing happening?

[MSNev]

Note: for "events" we do have an open issue on whether to "keep" both the event.domain and event.name vs just having the event.name Decide if event.domain separately from event.name is necessary #2994

So it may become something like
event.name: "DotNet.LogEvent"

So any input / preference you might have for identifying that uniqueness of an event would be welcome.

[CodeBlanch]

@jack-berg In .NET durable id is traditionally a manually-assigned thing. Eg:

new EventId(id: 18, name: "ProcessingFailure")
new EventId(id: 100)

There is a category name in .NET but it isn't part of the EventId contract. There really isn't any guarantee about the uniqueness, that is up to the app/dev to do correctly.

There is a new thing coming in .NET 8 (Nov 2023) to auto-generate Id as a hash of Name if the value wasn't provided: https://github.com/dotnet/runtime/pull/87892/files#diff-0a0edfbf348cc08e79a01ff6b0f9023793c86880d2cc9cbbb88c98bb7eb14575R236

@MSNev

I like the structure you are proposing for turning these into "events" but that is a probably a non-starter. For a couple of reasons:

• It makes these into .NET-specific things. We need this to be a standard across languages. If we wanted it to be .NET-specific we would just name these attributes something like dotnet.log.event_id + dotnet.log.event_name and call it a day 😄

• These are logs and people will want to see them in their log pipelines. Adding a couple of attributes/columns should be easy for backends to consume/handle. Creating a new pipeline/signal seems unnecessarily complex and will likely take a long time? There may also be a mismatch as far as volume. Logs will be high volume. Client events will be low volume (assuming?). I don't think backends will want to parse the complex event.data structure to get at the details in the high-volume path given the primary use case is to peek during ingestion. A primary goal is that access should be fast/cheap.

[MSNev]

It makes these into .NET-specific things.

I just gave the domain "dotnet" as an example, if this can be rationalized into something more generic then it would work.

Client events will be low volume (assuming?).

No, I would expect the reverse as every "client" (browser, android app, ios app, etc) could be sending thousands of (defined) events. eg. some websites track user clicks. And as part of "decoding" these values they have specific UI to handle these events.
No, they are actually the reverse. As every "client' will be sending the event.

I don't think backends will want to parse the complex event.data structure to get at the details in the high-volume path given the primary use case is to peek during ingestion.

But MS does this today, it doesn't necessarily need to happen at the first receiver but and there are cases of some sub-teams performing this at the point of querying the data -- but it DOES happen and is required for defining health alerts in relation to "some" events (such as client side JavaScript errors)

[MSNev]

I don't think backends will want to parse the complex event.data structure

This is actually one of the main sub-points for why we are pushing for defining the "shape" of an event, (domain, name, payload (event.data)) is so that venders / clients / receivers can make the decision on whether they event want to "parse" the payload (based on the "name") to perform either

• validation of the inbound event
• Store in a different table (for faster / better UX)
• drop / throttle the event (at the server side -- because even if every client only sends 1 event you could still be receiving 100's millions of those events (every browser, for every page load, etc)

[CodeBlanch]

Ya I don't know about the volume bit. I think a human doing things will always be much less than a cloud cranking out logs at scale. But let's put that aside for a second.

Why do we want these logs to be some other signal? No one in the .NET community has asked for events (so far). What they are asking for are these attributes on their logs.

What is the argument against...

General Logs Attributes

Attribute	Type	Description	Examples	Requirement Level
log.id	string	A durable identifier for the Log Record.	1; 0x100F	Optional
log.name	string	A name for the Log Record.	RequestProcessed; InvalidResponse	Optional
log.record.uid	string	A unique identifier for the Log Record. [1]	01ARZ3NDEKTSV4RRFFQ69G5FAV	Opt-In

[jack-berg]

log.record.uid already exists.

log.name seems conceptually similar to what we're trying to do with event.domain / event.name. That is, some sort of natural key representing the class / type of thing that happened. I.e. make it easy to query for logs WHERE log.name = 'RequestProcessed and find all the records which have a similar shape. The event.domain field just adds a namespace to avoid collisions of event.name, but I think the idea is otherwise the same.

log.id - I still don't get this. From what I'm reading here, its sort of like an integer alternative to what you call log.name, but not all records with the same log.name have the same log.id. If this is an accurate description, I'm not sure what purpose this serves and how to generalize it outside of dotnet ILogger.

Why do we want these logs to be some other signal? No one in the .NET community has asked for events (so far). What they are asking for are these attributes on their logs.

Not sure what you mean by this. Currently, OpenTelemetry logs and events share the same data model. As of now, event is defined as:

Events are recorded as LogRecords that are shaped in a special way: Event LogRecords have the attributes event.domain and event.name (and possibly other LogRecord attributes).

Where event.domain / event.name are conceptually similar to what you suggest as log.name. That definition doesn't seem to be at odds with what the .NET community has asked for. (With the exception of the log.id field, which I'm still trying to understand.)

[CodeBlanch]

@jack-berg

log.id is by far the more important thing. My understanding is that something like Windows has thousands (millions?) of logs defined each with an assigned unique id (keep me honest here @reyang). In .NET if you supply the structure, the id is required and name is optional. The name is really just a convenience thing for UI display.

As of now, event is defined as...

I think this would be a perfectly suitable solution:

Attribute	Type	Description	Examples	Requirement Level
event.domain	string	The domain identifies the business context for the events. [1]	browser	Optional
event.name	string	The name identifies the event.	click; exception	Optional
event.id	string	The durable identifier for the event.	1; 0x100F	Optional

But it is the "as of now" bit that is the problem for me. If the future goal is...

"event.domain": "DotNet",
"event.name": "LogEvent",
"event.data":
{
    key: "",
    categoryName: "",
    source: "",
    id: "",
    data: "",
    message: ... // etc
} 

That seems like a non-starter. Because...

a) We can't have a .NET specific solution. It most suit C++, Rust, and others.
b) Usage of EventId structure will be mixed. Most devs in small shops I would wager do not take the time to manage or specify the structure, which is perfectly fine. But the runtime internals will. We don't want to bifurcate the users log streams and hope their backend will do the right thing, do we?

Let's say the user has a client application which calls out to a cloud service. Client app emits these RUM events and they show up in a nice event UI. The cloud app emits logs. Because of this bifurcation some logs go to a nice log UI and some logs go to the event UI and show mixed in with the client interactions. Good or bad experience?

[jack-berg]

Let's ignore the event payload / event.data for a second. I do have opinions on that but think that talking about them just muddies the waters.

Wondering if you can help me understand how I would use event.id outside of a Windows / .NET use case. An event has a event.domain and event.name. Together, these function as a key that allows you to identify the class / type of thing that happened. I can easily filter log records to find the ones with a particular event.domain / event.name.

Now let's consider a future where we also add a event.id, defined as a durable identifier for the event. This appears to be an alternative to the event.domain / event.name key. Let's consider a user calling a future event API where we have an event.id attribute. The event API requires that you specify the event.domain / event.name because those are key parts of the event semantic conventions. Presumably, it would also accept a new event.id parameter.

How do we describe to a user what value should be used for event.id? How does a user ensure that the event.id they select is unique to others recorded by other instrumentation? How do we describe the difference between event.id and the other event.domain / event.name key (they seem to serve the same function).

Let's say the user has a client application which calls out to a cloud service. Client app emits these RUM events and they show up in a nice event UI. The cloud app emits logs. Because of this bifurcation some logs go to a nice log UI and some logs go to the event UI and show mixed in with the client interactions. Good or bad experience?

IMO, the point of the Event semantic conventions isn't to be prescriptive to how backends should process / ingest these Events. Its to give backends and users alike a very clean way to identify all the times a particular class / type of thing happened. With spans we rely on duck typing: How do you identify a HTTP server span? You find all spans with span.kind = SERVER AND http.request.method != null. That's an ok heuristic, but we can do better. That's the point of event.domain / event.name. The combo gives you a no-nonsense way to select all the times that particular class / type of thing happened. Sure you could probably do pattern matching on the log body, but that is likely to be less performant (e.g. consider a regex match against a body vs. WHERE event.domain = foo AND event.name = bar) and a pretty crappy experience.

If a backend wants to do something fancy, and store a particular type of event in a special way, that's fine, but it should also be fine for the backend to just treat all the log records the same, whether or not they have event.domain / event.name attached. We should mean it when we use "log" and "event" interchangeably in the log data model.

[CodeBlanch]

Let's consider a user calling a future event API

This may be the wrong thinking. What I'm pushing here is for the spec to define where appenders/bridge components can drop this information should the originating framework support such a concept.

.NET ILogger usage is like this:

public class WeatherForecastController
{
	private readonly ILogger<WeatherForecastController> _logger;

	public WeatherForecastController(ILogger<WeatherForecastController> logger)
	{
		_logger = logger;
	}

	public IEnumerable<WeatherForecast> Get()
	{
		WeatherForecast[] forecasts = GenerateForecasts();

		_logger.LogInformation(
			eventId: LogEvents.WeatherForecastsGenerated,
			message: "WeatherForecasts generated {count}",
			args: forecasts.Length);

		return forecasts;
	}
}

internal static class LogEvents
{
	public static EventId WeatherForecastsGenerated { get; } = new(id: 4001);
}

Let's look at some different ways we could send this off.

[Parden the pseudo json here. Trying to sort of show how it would look for OTLP log data model but I didn't try to nail the structure perfectly.]

One thing we are looking at for the categoryName ("WebApplication1.Controllers.WeatherForecastController" in this case) is to send that as instrumentationScope.name. Something like this:

{
   "instrumentationscope": { "name": "WebApplication1.Controllers.WeatherForecastController" } },
   "logs": [
      {
         "body": "WeatherForecasts generated {count}",
         "attributes": [
             { "key": "log.id", "value": 4001 },
             { "key": "count", "value": 100 },
         ]
      }
   ]
}

Or using event stuff:

{
   "instrumentationscope": { "name": "OpenTelemetry.Bridge.ILogger" } },
   "logs": [
      {
         "body": "WeatherForecasts generated {count}",
         "attributes": [
             { "key": "event.domain", "value": "WebApplication1.Controllers.WeatherForecastController" },
             { "key": "event.id", "value": 4001 },
             { "key": "count", "value": 100 },
         ]
      }
   ]
}

EventSource (C++, .NET) usage is like this:

[EventSource(Name = "MyLibrary")]
internal class LibraryEventSource : EventSource
{
	public static LibraryEventSource Log = new();

	[Event(100_001, Message = "Unexpected request length: {0}", Level = EventLevel.Warning)]
	public void LogUnexpectedRequestLengthWarning(int requestLength)
	{
		WriteEvent(100_001, requestLength);
	}
}

public class SomeClass
{
	public bool ProcessRequest(int length)
	{
		if (length < 0 || length > 1024)
		{
			LibraryEventSource.Log.LogUnexpectedRequestLengthWarning(length);
			return false;
		}

		// Process request

		return true;
	}
}

Here is how that might look using the same styles:

{
   "instrumentationscope": { "name": "MyLibrary" } },
   "logs": [
      {
         "body": "Unexpected request length: {requestLength}",
         "attributes": [
             { "key": "log.id", "value": 100_001 },
             { "key": "log.name", "value": "LogUnexpectedRequestLengthWarning" },
             { "key": "requestLength", "value": 9000 },
         ]
      }
   ]
}

Or using event stuff:

{
   "instrumentationscope": { "name": "OpenTelemetry.Bridge.EventSource" } },
   "logs": [
      {
         "body": "Unexpected request length: {requestLength}",
         "attributes": [
             { "key": "event.domain", "value": "MyLibrary" },
             { "key": "event.id", "value": 4001 },
             { "key": "event.name", "value": "LogUnexpectedRequestLengthWarning" },
             { "key": "requestLength", "value": 9000 },
         ]
      }
   ]
}

How do we describe to a user what value should be used for event.id?

We shouldn't. Again, I'm not trying to design a new event API. I'm trying to bridge things. It is up to the logging framework to define that. We just need to give it a home.