Skip to content

Commit 7a25749

Browse files
AndriySvyrydAndriySvyryd
authored
Merge branch 'main' into TestHelixCosmos
2 parents 82ca830 + a81cc59 commit 7a25749

87 files changed

Lines changed: 6437 additions & 952 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/ci-analysis/SKILL.md‎

Lines changed: 253 additions & 0 deletions
Large diffs are not rendered by default.

‎.agents/skills/ci-analysis/references/azdo-helix-reference.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Azure DevOps and Helix Reference
2+
3+
## Build Definition IDs (Example: dotnet/efcore)
4+
5+
Each repository has its own build definition IDs. Here are common ones for dotnet/efcore:
6+
7+
| Definition ID | Name | Description |
8+
|---------------|------|-------------|
9+
| `17` | efcore-ci | Main PR validation build |
10+
11+
**Note:** The script auto-discovers builds for a PR, so you rarely need to know definition IDs.
12+
13+
## Azure DevOps Organizations
14+
15+
**Public builds (default):**
16+
- Organization: `dnceng-public`
17+
- Project: `cbb18261-c48f-4abb-8651-8cdcb5474649`
18+
19+
**Internal/private builds:**
20+
- Organization: `dnceng`
21+
- Project GUID: Varies by pipeline
22+
23+
Override with:
24+
```powershell
25+
./scripts/Get-CIStatus.ps1 -BuildId 1276327 -Organization "dnceng" -Project "internal-project-guid"
26+
```
27+
28+
## Common Pipeline Names (Example: dotnet/efcore)
29+
30+
| Pipeline | Description |
31+
|----------|-------------|
32+
| `efcore-ci` | Main PR validation build |
33+
34+
Other repos have different pipelines - the script discovers them automatically from the PR.
35+
36+
## Useful Links
37+
38+
- [Helix Portal](https://helix.dot.net/): View Helix jobs and work items (all repos)
39+
- [Helix API Documentation](https://helix.dot.net/swagger/): Swagger docs for Helix REST API
40+
- [Build Analysis](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/LandingPage.md): Known issues tracking (arcade infrastructure)
41+
- [dnceng-public AzDO](https://dev.azure.com/dnceng-public/public/_build): Public builds for all dotnet repos
42+
43+
## Test Execution Types
44+
45+
### Helix Tests
46+
Tests run on Helix distributed test infrastructure. The script extracts console log URLs and can fetch detailed failure info with `-ShowLogs`.
47+
48+
### Local Tests (Non-Helix)
49+
Some repositories (e.g., dotnet/efcore and dotnet/sdk) run tests directly on the build agent. The script detects these and extracts Azure DevOps Test Run URLs.
50+
51+
## Known Issue Labels
52+
53+
- `Known Build Error` - Used by Build Analysis across all dotnet repositories
54+
- Search syntax: `repo:<owner>/<repo> is:issue is:open label:"Known Build Error" <test-name>`
55+
56+
Example searches (use `search_issues` when GitHub MCP is available, `gh` CLI otherwise):
57+
```bash
58+
# Search in runtime
59+
gh issue list --repo dotnet/runtime --label "Known Build Error" --search "FileSystemWatcher"
60+
61+
# Search in efcore
62+
gh issue list --repo dotnet/efcore --label "Known Build Error" --search "SaveChanges"
63+
```

‎.agents/skills/ci-analysis/references/azure-cli.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Deep Investigation with Azure CLI
2+
3+
The AzDO MCP tools handle most pipeline queries directly. This reference covers the Azure CLI fallback for cases where MCP tools are unavailable or the endpoint isn't exposed (e.g., downloading artifacts, inspecting pipeline definitions).
4+
5+
When the CI script and GitHub APIs aren't enough (e.g., investigating internal pipeline definitions or downloading build artifacts), use the Azure CLI with the `azure-devops` extension.
6+
7+
> 💡 **Prefer `az pipelines` / `az devops` commands over raw REST API calls.** The CLI handles authentication, pagination, and JSON output formatting. Only fall back to manual `Invoke-RestMethod` calls when the CLI doesn't expose the endpoint you need (e.g., build timelines). The CLI's `--query` (JMESPath) and `-o table` flags are powerful for filtering without extra scripting.
8+
9+
## Checking Authentication
10+
11+
Before making AzDO API calls, verify the CLI is installed and authenticated:
12+
13+
```powershell
14+
# Ensure az is on PATH (Windows may need a refresh after install)
15+
$env:Path = [System.Environment]::GetEnvironmentVariable("Path", "Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path", "User")
16+
17+
# Check if az CLI is available
18+
az --version 2>$null | Select-Object -First 1
19+
20+
# Check if logged in and get current account
21+
az account show --query "{name:name, user:user.name}" -o table 2>$null
22+
23+
# If not logged in, prompt the user to authenticate:
24+
# az login # Interactive browser login
25+
# az login --use-device-code # Device code flow (for remote/headless)
26+
27+
# Get an AAD access token for AzDO REST API calls (only needed for raw REST)
28+
$accessToken = (az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)
29+
$headers = @{ "Authorization" = "Bearer $accessToken" }
30+
```
31+
32+
> ⚠️ If `az` is not installed, use `winget install -e --id Microsoft.AzureCLI` (Windows). The `azure-devops` extension is also required — install or verify it with `az extension add --name azure-devops` (safe to run if already installed). Ask the user to authenticate if needed.
33+
34+
> ⚠️ **Do NOT use `az devops configure --defaults`** — it sets user-wide defaults that may not match the organization/project needed for dotnet repositories. Always pass `--org` and `--project` (or `-p`) explicitly on each command.
35+
36+
## Querying Pipeline Definitions and Builds
37+
38+
```powershell
39+
$org = "https://dev.azure.com/dnceng"
40+
$project = "internal"
41+
42+
# Find a pipeline definition by name
43+
az pipelines list --name "dotnet-unified-build" --org $org -p $project --query "[].{id:id, name:name, path:path}" -o table
44+
45+
# Get pipeline definition details (shows YAML path, triggers, etc.)
46+
az pipelines show --id 1330 --org $org -p $project --query "{id:id, name:name, yamlPath:process.yamlFilename, repo:repository.name}" -o table
47+
48+
# List recent builds for a pipeline (replace {TARGET_BRANCH} with the PR's base branch, e.g., main or release/9.0)
49+
az pipelines runs list --pipeline-ids 1330 --branch "refs/heads/{TARGET_BRANCH}" --top 5 --org $org -p $project --query "[].{id:id, result:result, finish:finishTime}" -o table
50+
51+
# Get a specific build's details
52+
az pipelines runs show --id $buildId --org $org -p $project --query "{id:id, result:result, sourceBranch:sourceBranch}" -o table
53+
54+
# List build artifacts
55+
az pipelines runs artifact list --run-id $buildId --org $org -p $project --query "[].{name:name, type:resource.type}" -o table
56+
57+
# Download a build artifact
58+
az pipelines runs artifact download --run-id $buildId --artifact-name "TestBuild_linux_x64" --path "$env:TEMP\artifact" --org $org -p $project
59+
```
60+
61+
## REST API Fallback
62+
63+
Fall back to REST API only when the CLI doesn't expose what you need:
64+
65+
```powershell
66+
# Get build timeline (stages, jobs, tasks with results and durations) — no CLI equivalent
67+
$accessToken = (az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)
68+
$headers = @{ "Authorization" = "Bearer $accessToken" }
69+
$timelineUrl = "https://dev.azure.com/dnceng/internal/_apis/build/builds/$buildId/timeline?api-version=7.1"
70+
$timeline = (Invoke-RestMethod -Uri $timelineUrl -Headers $headers)
71+
$timeline.records | Where-Object { $_.result -eq "failed" -and $_.type -eq "Job" }
72+
```
73+
74+
## Examining Pipeline YAML
75+
76+
All dotnet repos that use arcade put their pipeline definitions under `eng/pipelines/` or in the root. Use `az pipelines show` to find the YAML file path, then fetch it:
77+
78+
```powershell
79+
# Find the YAML path for a pipeline
80+
az pipelines show --id 1330 --org $org -p $project --query "{yamlPath:process.yamlFilename, repo:repository.name}" -o table
81+
82+
# Fetch the YAML from the repo (example: dotnet/efcore's CI pipeline)
83+
# Read the pipeline YAML from the repo to understand build stages and conditions
84+
# e.g., azure-pipelines-public.yml in dotnet/efcore
85+
86+
# For VMR unified builds, the YAML is in dotnet/dotnet:
87+
# eng/pipelines/unified-build.yml
88+
89+
# Templates are usually in eng/pipelines/common/ or eng/pipelines/templates/
90+
```
91+
92+
This is especially useful when:
93+
- A job name doesn't clearly indicate what it builds
94+
- You need to understand stage dependencies (why a job was canceled)
95+
- You want to find which template defines a specific step
96+
- Investigating whether a pipeline change caused new failures

‎.agents/skills/ci-analysis/references/binlog-comparison.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Deep Investigation: Binlog Comparison
2+
3+
When a test **passes on the target branch but fails on a PR**, comparing MSBuild binlogs from both runs reveals the exact difference in task parameters without guessing.
4+
5+
## When to Use This Pattern
6+
7+
- Test assertion compares "expected vs actual" build outputs (e.g., CSC args, reference lists)
8+
- A build succeeds on one branch but fails on another with different MSBuild behavior
9+
- You need to find which MSBuild property/item change caused a specific task to behave differently
10+
11+
## The Pattern: Delegate to Subagents
12+
13+
> ⚠️ **Do NOT download, load, and parse binlogs in the main conversation context.** This burns 10+ turns on mechanical work. Delegate to subagents instead.
14+
15+
### Step 1: Identify the two work items to compare
16+
17+
Use `Get-CIStatus.ps1` to find the failing Helix job + work item, then find a corresponding passing build (recent PR merged to the target branch, or a CI run on that branch).
18+
19+
**Finding Helix job IDs from build artifacts (binlogs to find binlogs):**
20+
When the failing work item's Helix job ID isn't visible (e.g., canceled jobs, or finding a matching job from a passing build), the IDs are inside the build's `SendToHelix.binlog`:
21+
22+
1. Download the build artifact with `az`:
23+
```
24+
az pipelines runs artifact list --run-id $buildId --org "https://dev.azure.com/dnceng-public" -p public --query "[].name" -o tsv
25+
az pipelines runs artifact download --run-id $buildId --artifact-name "TestBuild_linux_x64" --path "$env:TEMP\artifact" --org "https://dev.azure.com/dnceng-public" -p public
26+
```
27+
2. Load the `SendToHelix.binlog` and search for `Sent Helix Job` to find the GUIDs.
28+
3. Query each Helix job GUID with the CI script:
29+
```
30+
./scripts/Get-CIStatus.ps1 -HelixJob "{GUID}" -FindBinlogs
31+
```
32+
33+
**For Helix work item binlogs (the common case):**
34+
The CI script shows binlog URLs directly when you query a specific work item:
35+
```
36+
./scripts/Get-CIStatus.ps1 -HelixJob "{JOB_ID}" -WorkItem "{WORK_ITEM}"
37+
# Output includes: 🔬 msbuild.binlog: https://helix...blob.core.windows.net/...
38+
```
39+
40+
### Step 2: Dispatch parallel subagents for extraction
41+
42+
Launch two `task` subagents (can run in parallel), each with a prompt like:
43+
44+
```
45+
Download the msbuild.binlog from Helix job {JOB_ID} work item {WORK_ITEM}.
46+
Use the CI skill script to get the artifact URL:
47+
./scripts/Get-CIStatus.ps1 -HelixJob "{JOB_ID}" -WorkItem "{WORK_ITEM}"
48+
Download the binlog, load it, find the {TASK_NAME} task, and extract CommandLineArguments.
49+
Normalize paths (see table below) and sort args.
50+
Parse into individual args using regex: (?:"[^"]+"|/[^\s]+|[^\s]+)
51+
Report the total arg count prominently.
52+
```
53+
54+
**Important:** When diffing, look for **extra or missing args** (different count), not value differences in existing args. A Debug/Release difference in `/define:` is expected noise — an extra `/analyzerconfig:` or `/reference:` arg is the real signal.
55+
56+
### Step 3: Diff the results
57+
58+
With two normalized arg lists, `Compare-Object` instantly reveals the difference.
59+
60+
## Common Binlog Search Patterns
61+
62+
When investigating binlogs, these search query patterns are most useful:
63+
64+
- Search for a property: `analysislevel`
65+
- Search within a target: `under($target AddGlobalAnalyzerConfigForPackage_MicrosoftCodeAnalysisNetAnalyzers)`
66+
- Find all properties matching a pattern: `GlobalAnalyzerConfig`
67+
68+
## Path Normalization
69+
70+
Helix work items run on different machines with different paths. Normalize before comparing:
71+
72+
| Pattern | Replacement | Example |
73+
|---------|-------------|---------|
74+
| `/datadisks/disk1/work/[A-F0-9]{8}` | `{W}` | Helix work directory (Linux) |
75+
| `C:\h\w\[A-F0-9]{8}` | `{W}` | Helix work directory (Windows) |
76+
| `Program-[a-f0-9]{64}` | `Program-{H}` | Runfile content hash |
77+
| `dotnetSdkTests\.[a-zA-Z0-9]+` | `dotnetSdkTests.{T}` | Temp test directory |
78+
79+
### After normalizing paths, focus on structural differences
80+
81+
> ⚠️ **Ignore value-only differences in existing args** (e.g., Debug vs Release in `/define:`, different hash paths). These are expected configuration differences. Focus on **extra or missing args** — a different arg count indicates a real build behavior change.
82+
83+
## Example: CscArguments Investigation
84+
85+
A merge PR (release/10.0.3xx → main) had 208 CSC args vs 207 on main. The diff:
86+
87+
```
88+
FAIL-ONLY: /analyzerconfig:{W}/p/d/sdk/11.0.100-ci/Sdks/Microsoft.NET.Sdk/analyzers/build/config/analysislevel_11_default.globalconfig
89+
```
90+
91+
### What the binlog properties showed
92+
93+
Both builds had identical property resolution:
94+
- `EffectiveAnalysisLevel = 11.0`
95+
- `_GlobalAnalyzerConfigFileName = analysislevel_11_default.globalconfig`
96+
- `_GlobalAnalyzerConfigFile = .../config/analysislevel_11_default.globalconfig`
97+
98+
### The actual root cause
99+
100+
The `AddGlobalAnalyzerConfigForPackage` target has an `Exists()` condition:
101+
```xml
102+
<ItemGroup Condition="Exists('$(_GlobalAnalyzerConfigFile_...)')">
103+
<EditorConfigFiles Include="$(_GlobalAnalyzerConfigFile_...)" />
104+
</ItemGroup>
105+
```
106+
107+
The merge's SDK layout **shipped** `analysislevel_11_default.globalconfig` on disk (from a newer roslyn-analyzers that flowed from 10.0.3xx), while main's SDK didn't have that file yet. Same property values, different files on disk = different build behavior.
108+
109+
### Lesson learned
110+
111+
Same MSBuild property resolution + different files on disk = different build behavior. Always check what's actually in the SDK layout, not just what the targets compute.
112+
113+
## Anti-Patterns
114+
115+
> ❌ **Don't manually split/parse CSC command lines in the main conversation.** CSC args have quoted paths, spaces, and complex structure. Regex parsing in PowerShell is fragile and burns turns on trial-and-error. Use a subagent.
116+
117+
> ❌ **Don't assume the MSBuild property diff explains the behavior diff.** Two branches can compute identical property values but produce different outputs because of different files on disk, different NuGet packages, or different task assemblies. Compare the actual task invocation.
118+
119+
> ❌ **Don't load large binlogs and browse them interactively in main context.** Use targeted searches rather than browsing interactively. Get in, get the data, get out.

0 commit comments

Comments
 (0)