describe-ui misses SwiftUI toolbar Picker and NavigationBar back button that XCUITest can see

[aomoriringo]

Summary

On iOS 26+ simulator, axe describe-ui returns an incomplete accessibility tree for SwiftUI apps that use .toolbar + Picker(.segmented) and auto-generated NavigationStack back button. The same elements are visible to XCUITest, meaning the data exists in the simulator's accessibility APIs — it seems axe / FBSimulatorControl's accessibility traversal is stopping short.

This matters because agents driving apps via describe-ui + tap --label (e.g. via XcodeBuildMCP) cannot interact with these elements without resorting to coordinate-based taps.

Environment

• macOS 26.3.1
• Xcode 26.3, iOS 26.3 simulator (iPhone 17)
• AXe v1.5.2 (bundled with XcodeBuildMCP 2.3.2)
• App: SwiftUI on iOS 26 target, NavigationStack root, .toolbar { ToolbarItem(placement: .topBarTrailing) { Picker(... , selection: ...) { ... }.pickerStyle(.segmented) } }
• com.apple.Accessibility defaults AccessibilityEnabled=1, ApplicationAccessibilityEnabled=1

Reproduction

SwiftUI snippet:

NavigationStack {
    List { ... }
        .navigationTitle("賞名")
        .toolbar {
            ToolbarItem(placement: .topBarTrailing) {
                Picker("フィルタ", selection: $filter) {
                    ForEach(WorkListFilter.allCases, id: \.self) { f in
                        Text(f.label).tag(f)
                    }
                }
                .pickerStyle(.segmented)
            }
        }
}

Navigate into a pushed view (so the back button is shown), then run:

axe describe-ui --udid <UDID>

Observed (axe describe-ui)

Application  label=ZembuYomi  children=2
  Group      label=-   id=芥川龍之介賞    role=Nav bar    children=0   <-- back button missing
  Group      label=-   id=-              role=Group      children=9
    PopUpButton  label=古い順
    Button       label=火花, 又吉直樹, 第152回 (2015年上半期)
    ...

• SegmentedControl / Picker is absent entirely — the toolbar item is not traversed
• Nav bar Group has children: [] — the back button is not exposed as a child

Expected (XCUIApplication)

Running XCUITest with app.debugDescription / app.segmentedControls / app.navigationBars on the same screen shows:

SegmentedControl
  ├ "すべて" Button
  ├ "未読" Button
  └ "既読" Button

NavigationBar (identifier: "芥川龍之介賞")
  └ "BackButton"
  ├ "すべて" Button  (segmented duplicated here in XCUITest)
  ├ "未読" Button
  └ "既読" Button

i.e. iOS does expose these elements via its accessibility API, but axe / FBSimulatorControl does not surface them.

Impact

Agent-driven UI automation (e.g. via XcodeBuildMCP) cannot target these widgets by label/id. Users have to fall back to coordinate taps, which are fragile.

Notes / Possible cause

• axe uses FBSimulator's accessibilityElements(withNestedFormat: true) from FBSimulatorControl
• Apple's XCUIAutomation uses a different private API (XCAccessibility framework?) that traverses deeper into toolbar containers / system-provided navigation chrome
• Unclear whether FBSimulatorControl's accessibilityElements API can be extended, or whether axe should use a different API

Happy to help test fixes / provide more repro data if useful.

[coderabbitai]

🔗 Related PRs

#9 - Add tap by accessibility id/label [merged]
#20 - fix(touch): make long press gestures reliable [merged]
#39 - fix(describe-ui): Add point option support [merged]
#40 - feat(tap): add --wait-timeout, --value, --element-type to standalone tap [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[m13v]

hit this same class of gap doing AX-based macOS automation. SwiftUI toolbar items and NavigationStack back buttons don't land in AXChildren until something forces the tree to hydrate. XCUITest gets past it because XCAXClient talks to an in-process agent that materializes lazy containers; FBSimulatorControl's accessibilityElements just reads whatever Core Accessibility already cached. ugly workaround that stuck for us: call AXUIElementCopyElementAtPosition over the suspected region first, then re-read AXChildren. wakes up Picker(.segmented) and the back button almost every time.