Skip to content

Add vLLM Production Stack integration - #3272

Open
lbarrosby wants to merge 1 commit into
DataDog:masterfrom
lbarrosby:add-vllm-production-stack-integration
Open

lbarrosby wants to merge 1 commit into
DataDog:masterfrom
lbarrosby:add-vllm-production-stack-integration

Conversation

@lbarrosby

Copy link
Copy Markdown

What does this PR do?

Adds a new Agent-based integration for the router of vLLM Production Stack, the reference Kubernetes deployment for vLLM.

The check scrapes the router's Prometheus endpoint (/metrics on port 8000) and maps its metrics to vllm_production_stack.*:

  • Requests seen by the router: incoming requests per model, end-to-end latency histogram (tagged by status), errors by error_type, and input and output tokens.
  • Per-backend load from the router's sliding window: QPS, running, prefill, decoding and swapped requests, average latency and inter-token latency, average decoding length.
  • Per-backend engine stats scraped by the router (prefix cache hit rate, hits and queries) and backend health.
  • Router CPU, memory and disk usage.

The router labels per-backend series with server (the engine URL); it is renamed to backend so it does not clash with host-level tags. vllm:gpu_prefix_cache_hits_total and vllm:gpu_prefix_cache_queries_total are gauges despite the suffix, so they are submitted as gauges.

Motivation

The vllm integration in integrations-core monitors each engine, but there is no integration for the Production Stack router, which is where request routing, per-backend load and backend failures show up. The router reuses the vllm: prefix for some names (for example vllm:num_requests_running) with a different meaning and an extra server label, so pointing the vllm check at it does not work well. This integration is meant to be used next to vllm.

I'm happy to be listed as the maintainer of this integration.

Review checklist

  • PR has a meaningful title
  • Feature or bugfix has tests
  • Git history is clean
  • If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
  • If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

  • The fixture in tests/fixtures/metrics.txt is a real scrape from the router at tag vllm-stack-0.1.13, running with static service discovery and round-robin routing in front of two OpenAI-compatible backends, after streaming and non-streaming requests, an unknown model, and one backend going down (which produced the ClientConnectorError errors and the status:error latency series).
  • Because the backends were not vLLM engines, the engine-derived gauges (backend.prefix_cache.*) are zero in the fixture, and avg_itl/avg_decoding_length show the router's -1 "no data" value; the README mentions both.
  • vllm:num_requests_waiting is declared by the router but never set, so it is not mapped.
  • Tested locally with ddev test vllm_production_stack (unit), ddev test --lint, and ddev validate (config, models, metadata, service-checks, readmes, openmetrics, typos, codeowners, package). I also ran the check against the live router and all 22 metrics plus the health service check came through.

Collect the Prometheus metrics exposed by the vLLM Production Stack router: incoming requests, end-to-end latency, errors and token counts, per-backend load and prefix cache hit rate, backend health, and router resource usage.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby
lbarrosby requested review from a team as code owners November 5, 2026 17:49
@joepeeples joepeeples self-assigned this Nov 5, 2026
@joepeeples joepeeples removed their assignment Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants