gRPC Throughput Drops to Zero Under High Load

[suzmue]

Description

When running high-throughput benchmarks with the Rust Pub/Sub client, the publisher exhibits a "burst and stall" behavior when being published in a tight loop. Specifically, after an initial burst of high message volume, all outgoing activity—including both publishing and acknowledgement responses—drops to zero and does not recover.

This stall typically occurs when a large number of messages are outstanding. Logs indicate that while the application continues to attempt publishing, the transport layer stops sending data, resulting in a complete throughput collapse.

Solution / Workaround

We believe the root cause of this stall is related to a bug in the underlying h2 crate (related bug: hyperium/h2#853 and fix: hyperium/h2#860).

Updating the h2 dependency to the latest version (v0.4.13) resolves the issue.

[suzmue]

Upgrading doesn't seem to fully resolve the problem, though is more difficult to trigger.

Setting the concurrency_limit on the GrpcClient endpoint to 100 in gax-internal seems to resolve the problem. While this may limit max throughput due to additional overhead this will make our client libraries more robust.