Skip to content

Add Xinference integration - #3274

Open
lbarrosby wants to merge 1 commit into
DataDog:masterfrom
lbarrosby:add-xinference-integration
Open

lbarrosby wants to merge 1 commit into
DataDog:masterfrom
lbarrosby:add-xinference-integration

Conversation

@lbarrosby

Copy link
Copy Markdown

What does this PR do?

Adds a new Agent-based integration for Xinference (Xorbits Inference), which serves LLM, embedding, rerank, image and audio models behind an OpenAI-compatible API, on one machine or as a supervisor plus workers.

The check scrapes the Prometheus endpoints Xinference exposes, one instance per endpoint, and maps the metrics to xinference.*:

  • Worker (--metrics-exporter-port): model requests, failures, latency histogram, requests being served and the concurrency limit, last model load time, and for LLMs time to first token (histogram) and input, output and generated tokens.
  • Supervisor (API port, 9997 by default): online workers, loaded models by type, model lifecycle status, worker CPU and memory, API key requests by key, user, model and result, API key request latency, active and expired keys, banned IPs and keys, and HTTP API requests and responses by path and status code.

Motivation

Xinference is a widely used self-hosted inference platform (vLLM, SGLang, llama.cpp, MLX and Transformers engines under one API), and there is no integration for it today. Its own docs point users to OpenTelemetry for Datadog; this gives them the Prometheus endpoints through the Agent instead.

I'm happy to be listed as the maintainer of this integration.

Review checklist

  • PR has a meaningful title
  • Feature or bugfix has tests
  • Git history is clean
  • If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
  • If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

  • Both fixtures are real scrapes from Xinference 4.1.0 (xinference-local with --metrics-exporter-port, llama.cpp engine, small GGUF model registered as a custom LLM), after chat, completion and streaming requests, a failing request, and requests with two API keys (one allowed, one denied for the model). The only edit is the xinference_home label value, replaced with /root/.xinference. supervisor.txt and worker.txt carry different metric sets, so the metadata symmetry test runs the check against both.
  • Not mapped yet: the GPU metrics (worker_gpu_*, model_gpu_*), because the scrape comes from a machine without NVIDIA GPUs and I did not want to add metadata I could not check against real output; model_unexpected_termination, which only shows up after a worker failure; the token router metrics, which belong to a separate feature; and the info-style build_info, config_info and model_info. Happy to add the GPU ones in a follow-up with a scrape from a GPU host.
  • Tested locally with ddev test xinference (unit), ddev test --lint, and ddev validate (config, models, metadata, service-checks, readmes, openmetrics, typos, package). I also ran the check against the live supervisor and worker endpoints and all 32 metrics plus the health service check came through.

Collect the Prometheus metrics exposed by the Xinference supervisor and workers: model requests, errors, latency and concurrency, time to first token and token counts, model load time and status, loaded models and workers, worker CPU and memory, API key usage, and HTTP API requests.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby
lbarrosby requested review from a team as code owners November 6, 2026 03:02
@git-thuerk-done git-thuerk-done added the editorial review Waiting on a more in-depth review from a docs team editor label Nov 6, 2026
@git-thuerk-done

git-thuerk-done commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opened DOCS-16334 to assign a Docs writer and follow up with editorial review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

editorial review Waiting on a more in-depth review from a docs team editor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants