Add optional retry logic to RabbitMQ publisher for transient connection failures

[Copilot]

Workers crash when publishing status after encountering Connection dead, no heartbeat or data received in >= 60s, terminating game servers unnecessarily on transient network issues.

Changes

Enhanced RabbitPublisher with optional retry

• Added retry_config parameter to __init__() that defaults to None (no retry by default)
• When retry_config is provided, publish() operations retry with exponential backoff
• When retry_config is None (default), publish fails immediately without retry to prevent duplicate messages
• Added _ensure_channel() to validate channel health before publish operations
• Leverages existing is_transient_rmq_error filter from retry module

Added ResilientConnection wrapper (optional for future use)

• Validates connection health and auto-reconnects on stale connections
• Invalidates failed connections to force clean reconnection on retry
• Supports context manager pattern for resource cleanup

Usage

Default behavior (no retry) - prevents duplicate messages:

# Default: Fails immediately on connection timeout (no retry)
publisher = RabbitPublisher(connection, binding_configs)
publisher.publish("status: RUNNING")

Opt-in retry behavior - explicitly enable when duplicate message handling is in place:

from libs.python.retry import RetryConfig, is_transient_rmq_error

# Explicitly enable retry with configuration
retry_config = RetryConfig(
    max_attempts=3,
    initial_delay=0.5,
    max_delay=5.0,
    exponential_base=2.0,
    exception_filter=is_transient_rmq_error,
)
publisher = RabbitPublisher(connection, binding_configs, retry_config=retry_config)
publisher.publish("status: RUNNING")  # Retries up to 3 times on transient failures

Testing

• 11 tests for ResilientConnection (using pytest)
• 8 tests for enhanced RabbitPublisher retry logic (using pytest)
• All 19 tests passing
• Includes test for default no-retry behavior

Original prompt

Add a lightweight wrapper for the AMQPStorm connection to handle timeouts/retries

The worker successfully started the CS2 dedicated server process but immediately encountered a fatal AMQP connection error (Connection dead, no heartbeat or data received in >= 60s) while attempting to publish its "RUNNING" status, leading to the immediate shutdown of the game server service.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.