chore(BA-4377): Increase Tempo max trace size to 50 MB

[hhoikoo]

resolves #8774 (BA-4377)

Overview

Raises Tempo's max_bytes_per_trace from the default 5 MB to 50 MB so that large OTel GraphQL traces are ingested instead of being rejected with TRACE_TOO_LARGE.

Problem Statement

• With OTel distributed tracing enabled in the Manager (BA-4330) and per-resolver spans in GraphQL middleware (BA-4376), complex queries produce traces exceeding Tempo's 5 MB default limit
• Tempo rejects these with TRACE_TOO_LARGE and drops the entire trace, making it impossible to debug the queries that need investigation most
• The 50 MB limit provides generous headroom given that BA-4376's leaf-type span suppression already eliminates ~92% of resolver spans

---

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue
• Installer updates including:
  • Fixtures for db schema changes
  • New mandatory config options
• Update of end-to-end CLI integration tests in ai.backend.test
• API server-client counterparts (e.g., manager API -> client SDK)
• Test case(s) to:
  • Demonstrate the difference of before/after
  • Demonstrate the flow of abstract/conceptual models with a concrete implementation
• Documentation
  • Contents in the docs directory
  • docstrings in public interfaces and type annotations

[Copilot]

Pull request overview

Updates the local Grafana Tempo configuration to allow ingesting significantly larger traces, preventing OTel GraphQL traces from being dropped with TRACE_TOO_LARGE.

Changes:

• Increase Tempo per-trace ingestion limit by setting overrides.defaults.global.max_bytes_per_trace to 50,000,000 bytes (50 MB).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.