Line number for .NET stack traces with server-side PDB support

[bruno-garcia]

Problem statement:

.NET since its inception had the default Debug and Release configurations emit debug information that was used in production, so that at runtime, exceptions could include stack traces with function names, line numbers and file paths.

That requires apps to ship pdbs together with the executable, which is common on server apps, and most desktop apps. Particularly internal ones where the download size and reverse engineering are not big concerns.
The only motivation for server-side pdb support in those days were from desktop apps (such as WinForms and WPF) that wanted to have a smaller installer, or were concerned with facilitating reverse engineering of their IP.

In 2011 mobile support for .NET was introduced through Xamarin, in mobile app size is critical to stay as small as possible, so Xamarin didn't include debug information in the final app. And to this day, Xamarin users don't have line numbers on stack traces in Sentry, or any other competitor (to the best of my knowledge).

This feature was requested back in 2015 but the reason this became a bigger priority now is due to .NET MAUI. And also because ASP.NET Core since .NET 3.1 (or 5?) does not include PDBs anymore in the runtime installation. So 'system' frames no longer have line numbers on stack traces. Symbols (portable pdbs) are made available on the nuget.org's symbol server.

Goals:

Support portable PDB (ppdb) with the focus of giving line numbers for InApp frames

1. For that, SDK needs to include data required by the server to look up data in portable PDBs
2. sentry-cli will need to understand and upload ppdb's
3. Sentry's .NET SDK can include msbuild tasks that use the wizard API for onboarding, and invoke sentry-cli

Line number of other frames such as system frames.

This requires fetching symbols from nuget.org

Rely on source link to render a link to the right sha/file:line

1. At this point we could fetch the source and show it as Source Context.

Technical details

.NET tooling, by default, generates exe or dll with an accompanying pdb.
At runtime, when you use an API to retrieve a line number or a file path, the framework makes a lookup to the pdb on the local directory, and find the appropriate line number and path through the pdb.

sentry-dotnet/src/Sentry/Internal/MonoSentryStackTraceFactory.cs

Lines 58 to 60 in 610dee8

	InstructionOffset = f.Offset != 0 ? f.Offset : (long?)null,
	Function = f.MethodSignature,
	LineNumber = GetLineNumber(f.Line),

sentry-dotnet/src/Sentry/Extensibility/SentryStackTraceFactory.cs

Lines 178 to 188 in 610dee8

	var lineNo = stackFrame.GetFileLineNumber();
	if (lineNo > 0)
	{
	frame.LineNumber = lineNo;
	}
	var colNo = stackFrame.GetFileColumnNumber();
	if (lineNo > 0)
	{
	frame.ColumnNumber = colNo;
	}

dotnet Sentry.Samples.Console.Basic.dll
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Program.<<Main>$>g__AnotherMethod|0_1() in /Users/bruno/git/sentry-dotnet/samples/Sentry.Samples.Console.Basic/Program.cs:line 14
   at Program.<<Main>$>g__SomeMethod|0_0() in /Users/bruno/git/sentry-dotnet/samples/Sentry.Samples.Console.Basic/Program.cs:line 9
   at Program.<Main>$(String[] args) in /Users/bruno/git/sentry-dotnet/samples/Sentry.Samples.Console.Basic/Program.cs:line 4

Outside of this, there's no (to my knowledge) straightforward way to get from an Exception instance to an over-the-wire format that a standard system can be plugged in to give line numbers and paths. In other words, Exception.ToString() will just result in the same stack trace, but without any line numbers.

rm -rf Sentry.Samples.Console.Basic.pdb 
dotnet Sentry.Samples.Console.Basic.dll
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Program.<<Main>$>g__AnotherMethod|0_1()
   at Program.<<Main>$>g__SomeMethod|0_0()
   at Program.<Main>$(String[] args)

The way Mono solved this in the past was through mono-symbolicate.
The runtime generates an exception in a different format than above. It encodes the mvid (module version id which is the id of the pdb) and the aotid which was added when the code as AOT (ahead of time compiled, for example on Xamarin.iOS). Mono symbolicate on the 'server' needs to prepare code in a way that is sort of a symbol server lookup protocol, by moves the pdbs to folders named by their mvid.

This PR added support to parse that in the Sentry SDK for .NET.

In order to get this done at Sentry, we'll need to change the Sentry SDK for .NET and add the data required by the backend to find the correct pdb, and lookup within it.

The data that has to be read at runtime, to add to the payload can be found in this PoC: https://github.com/bruno-garcia/simbolo/blob/d7340e46c45c87e4598b76f350336cb695fa00f1/Simbolo/Client.cs#L19
And an example of the lookup done this way (with .NET library): https://github.com/bruno-garcia/simbolo/blob/d7340e46c45c87e4598b76f350336cb695fa00f1/Simbolo.Backend/Symbolicate.cs#L14
The PoC above does call from native code, since it was supposed to serve as an example where we use FFI from Rust into .NET on the backend: https://github.com/bruno-garcia/simbolo/blob/d7340e46c45c87e4598b76f350336cb695fa00f1/Simbolo.NativeLib/app.c

The PoC relied on [Mono.Cecil](https://github.com/jbevain/cecil). This library is very popular, widely adopted and allows for modifying the IL. On the other hands it allocates memory when reading debug info. Alternatively, System.Reflection.Metadata can only read things, but does so in a much more optimized way. Simbolo has a branch using System.Reflection.Metadata.

The symbol lookup straight to portable pdb will be a solution to customers that upload their . NET pdbs to Sentry.
But this won't solve the use case for libraries published to NuGet. There, we'll need to make a symbol server lookup to nuget.org to fetch the relevant debug symbols. All .NET libraries (installed with .NET) have source link information so we know the commit sha, git repo link so we can render a link to the exact line number.

Resources

Relates to:

• Add Sentry support for .NET debug symbols (portable pdbs) to unblock app distribution to customer devices without pdbs. #193
• In .NET 5+ no line numbers for frames of the framework #604

PPDB

• docs: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbols-portable-pdb
• design: https://github.com/dotnet/core/blob/main/Documentation/diagnostics/portable_pdb.md
• spec: https://github.com/dotnet/runtime/blob/main/docs/design/specs/PortablePdb-Metadata.md

For Sentry employees: Internal video intro about this problem: https://drive.google.com/file/d/1nXOtB-ChTcuCj1fqgPrbHhx83A0bvLEw/view

[bruno-garcia]

And also because ASP.NET Core since .NET 3.1 (or 5?) does not include PDBs anymore in the runtime installation. So 'system' frames no longer have line numbers on stack traces. Symbols (portable pdbs) are made available on the nuget.org's symbol server.

For this case specifically, there's a work around. @SimonCropp built a library that is able to fetch the PDBs from the symbol server at build time and include them in our published app. This will make the deployed app larger though, which is usually fine (maybe not on FaaS/serverless environments though).

https://github.com/SimonCropp/Cymbal

[Swatinem]

Support for this was added in sentry-cli for Portable PDB uploads, and symbolication support is already live.
The last missing piece here is #1785.