Repository Quality: BannedSymbols Assertion Enforcement Gaps in 3 Unit Test Projects

[Evangelink]

🎯 Repository Quality Improvement Report — Test Assertion Enforcement Consistency

Analysis Date: 2026-06-18
Focus Area: BannedSymbols Assertion Enforcement Consistency (Custom)
Strategy Type: Custom

Executive Summary

The repository enforces test assertion style via BannedSymbols.txt files paired with Microsoft.CodeAnalysis.BannedApiAnalyzers (globally installed in eng/Analyzers.props). This mechanism works well for 9 out of 12 unit test projects. However, three projects currently have no BannedSymbols.txt, leaving their assertion style unguarded by tooling.

The two most actionable gaps are MSTest.SourceGeneration.UnitTests and MSTest.AotReflection.SourceGeneration.UnitTests: they use MSTest as the test runner (EnableMSTestRunner=true) combined with AwesomeAssertions for verification — the opposite pattern from the closely-related MSTest.Analyzers.UnitTests project (which bans AwesomeAssertions). There is currently no file documenting, let alone enforcing, this divergence.

TestFramework.UnitTests is the third affected project. It uses the internal TestContainer-based framework and AwesomeAssertions. Because it tests the Assert class directly, a blanket MSTest-Assert ban is not appropriate; a targeted file documenting the approved style would still add clarity.

Full Analysis Report

Focus Area: BannedSymbols Assertion Enforcement Consistency

Current State Assessment

Metrics Collected:

Metric	Value	Status
Unit test projects total	12	i️
Projects WITH BannedSymbols.txt	9 (75%)	⚠️
Projects WITHOUT BannedSymbols.txt	3 (25%)	❌
Projects banning AwesomeAssertions	7	✅
Projects banning MSTest Assert family	2 (adapter tests)	✅
Projects with no assertion enforcement	3	❌
Accidental assertion style leakage in enforced projects	0	✅

The three gaps:

Project	Test Runner	Assertions Used	BannedSymbols.txt
TestFramework.UnitTests	Internal TestContainer	AwesomeAssertions	❌ Missing
MSTest.SourceGeneration.UnitTests	MSTest (EnableMSTestRunner=true)	AwesomeAssertions	❌ Missing
MSTest.AotReflection.SourceGeneration.UnitTests	MSTest (EnableMSTestRunner=true)	AwesomeAssertions	❌ Missing

Comparison with closest peer project:

Project	Test Runner	Assertions Enforced
MSTest.Analyzers.UnitTests	MSTest	Bans AwesomeAssertions (N:AwesomeAssertions)
MSTest.SourceGeneration.UnitTests	MSTest	Nothing enforced
MSTest.AotReflection.SourceGeneration.UnitTests	MSTest	Nothing enforced

Findings

Strengths

• BannedApiAnalyzers is included globally for all projects — no per-project package installation needed.
• All 7 MTP-side unit test projects that ban AwesomeAssertions have zero AwesomeAssertions references (enforcement is working).
• Both adapter unit test projects that ban MSTest's Assert family are clean (zero violations).
• Consistent BannedSymbols.txt content across all 16 src/Platform/ projects.

Areas for Improvement

• [HIGH] MSTest.SourceGeneration.UnitTests and MSTest.AotReflection.SourceGeneration.UnitTests use AwesomeAssertions with no enforcement. If a contributor migrates to MSTest assertions (Assert.AreEqual, etc.), the CI will not catch the inconsistency.
• [HIGH] The divergence between MSTest.Analyzers.UnitTests (bans AwesomeAssertions) and the two source generation test projects (uses AwesomeAssertions without a ban) is undocumented. A new contributor reading the project instructions ("Most MTP unit-test projects and MSTest.Analyzers.UnitTests ban AwesomeAssertions") would expect the source generation tests to follow the same rule.
• [MEDIUM] TestFramework.UnitTests has no BannedSymbols.txt. Because it exercises the Assert API, it cannot ban MSTest's Assert class, but a file documenting "use AwesomeAssertions for test verification; direct Assert.* calls are only permitted when testing the Assert API itself" would prevent style drift.

---

🤖 Suggested Improvement Tasks

The following actionable tasks address the findings above.

Task 1: Add BannedSymbols.txt to MSTest.SourceGeneration.UnitTests

Priority: High
Estimated Effort: Small

Decide on and enforce the assertion style. Since the project uses MSTest as the runner and already contains AwesomeAssertions assertions, the simplest path is to mirror the adapter projects (MSTestAdapter.UnitTests, MSTestAdapter.PlatformServices.UnitTests) and ban the MSTest assertion types. Create test/UnitTests/MSTest.SourceGeneration.UnitTests/BannedSymbols.txt:

T:Microsoft.VisualStudio.TestTools.UnitTesting.Assert; Use AwesomeAssertions for testing in this project.
T:Microsoft.VisualStudio.TestTools.UnitTesting.StringAssert; Use AwesomeAssertions for testing in this project.
T:Microsoft.VisualStudio.TestTools.UnitTesting.CollectionAssert; Use AwesomeAssertions for testing in this project.

Alternatively, align with MSTest.Analyzers.UnitTests by migrating the assertions to MSTest style and banning N:AwesomeAssertions. Either choice is valid, but one must be made and enforced.

---

Task 2: Add BannedSymbols.txt to MSTest.AotReflection.SourceGeneration.UnitTests

Priority: High
Estimated Effort: Small

Apply the same decision made in Task 1 to test/UnitTests/MSTest.AotReflection.SourceGeneration.UnitTests/BannedSymbols.txt. The two projects are structurally identical (AOT reflection source generator vs. regular source generator) and should share the same assertion policy.

---

Task 3: Add BannedSymbols.txt to TestFramework.UnitTests

Priority: Medium
Estimated Effort: Small

Create test/UnitTests/TestFramework.UnitTests/BannedSymbols.txt to document that AwesomeAssertions is the approved verification style. Since this project tests the Assert class itself, it cannot ban Assert; however, it can ban T:Microsoft.VisualStudio.TestTools.UnitTesting.StringAssert and T:Microsoft.VisualStudio.TestTools.UnitTesting.CollectionAssert (which are not under test here) to prevent mixed styles:

T:Microsoft.VisualStudio.TestTools.UnitTesting.StringAssert; Use AwesomeAssertions for testing in this project.
T:Microsoft.VisualStudio.TestTools.UnitTesting.CollectionAssert; Use AwesomeAssertions for testing in this project.

At a minimum, add the file as a documented artifact even if it is currently empty, so that the presence of a policy is explicit.

---

Task 4: Document the source-generator test assertion style divergence in the contributing guide

Priority: Medium
Estimated Effort: Small

The contributing instructions (embedded in .github/copilot-instructions.md or CONTRIBUTING.md) state: "Most MTP unit-test projects (and MSTest.Analyzers.UnitTests, MSTest.SelfRealExamples.UnitTests) ban AwesomeAssertions and require MSTest Assert/StringAssert/CollectionAssert." This description covers MSTest.Analyzers.UnitTests but not the source-generator tests. Update the instructions to explicitly name MSTest.SourceGeneration.UnitTests and MSTest.AotReflection.SourceGeneration.UnitTests and state which assertion style they use, so contributors don't need to infer it.

---

Task 5: Add a CI lint step to catch future BannedSymbols.txt omissions in new test projects

Priority: Low
Estimated Effort: Medium

Add a simple check (e.g., a build target or a shell script in CI) that scans test/UnitTests/ for .csproj files and verifies that each project directory has a corresponding BannedSymbols.txt. This prevents the pattern from silently breaking again when a new test project is added. The check can be a lightweight find + diff comparison, or an MSBuild target appended to test/Directory.Build.targets.

---

📊 Historical Context

Previous Focus Areas (last 5 runs)

Date	Focus Area	Type
2026-06-17	localization-pipeline-health	Custom
2026-06-16	target-framework-support-matrix	Custom
2026-06-15	experimental-api-lifecycle-governance	Custom
2026-06-12	dependency-health	Standard
2026-06-11	public-api-xml-documentation	Standard

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add BannedSymbols.txt to MSTest.SourceGeneration.UnitTests — Priority: High
2. Add BannedSymbols.txt to MSTest.AotReflection.SourceGeneration.UnitTests — Priority: High

Short-term Actions (This Month)

1. Add BannedSymbols.txt to TestFramework.UnitTests — Priority: Medium
2. Update contributing guide to document source-gen test assertion style — Priority: Medium
3. Add CI lint check for missing BannedSymbols.txt in new test projects — Priority: Low

Next analysis: 2026-06-19 — Focus area selected based on diversity algorithm

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Repository Quality Improver workflow. · 589.6 AIC · ⌖ 14.7 AIC · [◷]( · ◷)

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repository-quality-improver.md@main

• expires on Jun 20, 2026, 11:03 PM UTC