Add telemetry to categorize build failure reasons

[Copilot]

Work item (Internal use):

Summary

MSBuild telemetry tracks build success/failure but lacks granularity on why builds fail. This adds error categorization to enable failure pattern analysis.

Changes:

• BuildTelemetry: Added FailureCategory (primary) and error count properties per category
• BuildErrorTelemetryTracker: New dedicated class that encapsulates error tracking and categorization logic using lock-free Interlocked operations for thread safety, improving separation of concerns and performance
• LoggingService: Uses BuildErrorTelemetryTracker instance to delegate telemetry operations
• Categorization: 8 categories based on error code patterns:
  • Compiler: CS*, VBC*, FS* (via code or subcategory field)
  • MSBuildEngine: MSB4001-MSB4999 (except MSB4236)
  • Tasks: MSB3001-MSB3999
  • SDKResolvers: MSB4236 (SDK resolution errors)
  • NETSDK: NETSDK* (.NET SDK diagnostics)
  • NuGet: NU*
  • BuildCheck: BC*
  • Other: uncategorized errors

Data available in both VS telemetry (GetActivityProperties) and SDK aggregated telemetry (GetProperties). Error count properties use namespaced keys (e.g., errorCounts.compiler, errorCounts.netsdk) for better Kusto query ergonomics, enabling easy bucketing of all error counts together.

Customer Impact

Enables data-driven prioritization of failure root causes. Telemetry consumers can identify which error categories affect users most frequently. The distinction between SDK resolvers and .NET SDK diagnostics provides more actionable insights for SDK-related failures. Namespaced property keys improve query performance and usability in telemetry analysis tools.

Regression?

• Yes
• No

New properties are only populated when builds fail. No changes to success path or existing telemetry data. Telemetry logic is isolated in a dedicated tracker class for better maintainability. Lock-free implementation using Interlocked operations ensures thread safety without blocking message processing.

Testing

• 16 unit tests for categorization logic (all error code patterns including new NETSDK and SDKResolvers categories)
• 7 unit tests for telemetry property serialization with namespaced keys
• Manual validation: built failing project, verified CS0103 categorized as Compiler
• All tests pass after refactoring to separate tracker class and implementing lock-free operations

Risk

Low. Additive telemetry change with no impact on build behavior or existing telemetry schema. Refactoring to dedicated tracker class with lock-free Interlocked operations improves both code maintainability and performance. Property namespacing is backward-compatible as it only affects the property keys in the telemetry dictionary, not the underlying data structure.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add telemetry to differentiate build failure reasons</issue_title>
<issue_description>Currently, MSBuild's BuildTelemetry class tracks general build success/failure via the BuildSuccess boolean property, but does not provide granularity on why a build failed. This makes it difficult to analyze failure patterns and improve the developer experience.

Add telemetry data to categorize build failure reasons when BuildSuccess = false. The failure categories should include at minimum:

Success Criteria

When a build fails, telemetry should report the primary failure category
Error counts by category should be included
The first error code encountered could optionally be included for debugging
Data should be available in both VS telemetry and SDK aggregated telemetry</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Add telemetry to differentiate build failure reasons #13006

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[dotnet-policy-service]

Hello @@copilot, I noticed that you’re changing an .swr file or any file under src/Package/MSBuild.VSSetup.. Please make sure to validate this change by an experimental VS insertion. This is accomplished by pushing to an exp/* branch, which requires write permissions to this repo.

[Copilot]

Pull request overview

This PR adds telemetry to categorize build failure reasons, enabling data-driven analysis of why builds fail. The implementation introduces a new BuildErrorTelemetryTracker class that tracks errors across seven categories (Compiler, MSBuildEngine, Tasks, SDK, NuGet, BuildCheck, Other) and populates telemetry properties with error counts and the primary failure category.

Changes:

• Added error categorization properties to BuildTelemetry (FailureCategory and 7 error count properties)
• Created BuildErrorTelemetryTracker class to encapsulate error tracking logic with optimized categorization
• Integrated error tracking into LoggingService to track errors as they're logged
• Updated BuildManager to populate telemetry with error data on build failure
• Added comprehensive test coverage for all error categories and telemetry serialization
• Updated telemetry documentation

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Framework/Telemetry/BuildTelemetry.cs	Added failure category and error count properties; updated telemetry serialization methods
src/Build/Microsoft.Build.csproj	Added BuildErrorTelemetryTracker.cs to project compilation
src/Build/BackEnd/Components/Logging/BuildErrorTelemetryTracker.cs	New class implementing error categorization logic with performance optimizations
src/Build/BackEnd/Components/Logging/LoggingService.cs	Integrated error tracking and added method to populate telemetry
src/Build/BackEnd/Components/Logging/ILoggingService.cs	Added PopulateBuildTelemetryWithErrors to interface
src/Build/BackEnd/BuildManager/BuildManager.cs	Added call to populate error telemetry on build failure
src/Build.UnitTests/BackEnd/MockLoggingService.cs	Added empty mock implementation of new interface method
src/Build.UnitTests/BackEnd/KnownTelemetry_Tests.cs	Added tests for telemetry serialization of failure data
src/Build.UnitTests/BackEnd/BuildTelemetryErrorCategorization_Tests.cs	New comprehensive test suite for error categorization
documentation/wiki/CollectedTelemetry.md	Added documentation for primary failure category property