Repository navigation
Conversation
Collect the Prometheus metrics exposed by the Xinference supervisor and workers: model requests, errors, latency and concurrency, time to first token and token counts, model load time and status, loaded models and workers, worker CPU and memory, API key usage, and HTTP API requests. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Contributor
|
Opened DOCS-16334 to assign a Docs writer and follow up with editorial review. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a new Agent-based integration for Xinference (Xorbits Inference), which serves LLM, embedding, rerank, image and audio models behind an OpenAI-compatible API, on one machine or as a supervisor plus workers.
The check scrapes the Prometheus endpoints Xinference exposes, one instance per endpoint, and maps the metrics to
xinference.*:--metrics-exporter-port): model requests, failures, latency histogram, requests being served and the concurrency limit, last model load time, and for LLMs time to first token (histogram) and input, output and generated tokens.Motivation
Xinference is a widely used self-hosted inference platform (vLLM, SGLang, llama.cpp, MLX and Transformers engines under one API), and there is no integration for it today. Its own docs point users to OpenTelemetry for Datadog; this gives them the Prometheus endpoints through the Agent instead.
I'm happy to be listed as the maintainer of this integration.
Review checklist
Additional Notes
xinference-localwith--metrics-exporter-port, llama.cpp engine, small GGUF model registered as a custom LLM), after chat, completion and streaming requests, a failing request, and requests with two API keys (one allowed, one denied for the model). The only edit is thexinference_homelabel value, replaced with/root/.xinference.supervisor.txtandworker.txtcarry different metric sets, so the metadata symmetry test runs the check against both.worker_gpu_*,model_gpu_*), because the scrape comes from a machine without NVIDIA GPUs and I did not want to add metadata I could not check against real output;model_unexpected_termination, which only shows up after a worker failure; the token router metrics, which belong to a separate feature; and the info-stylebuild_info,config_infoandmodel_info. Happy to add the GPU ones in a follow-up with a scrape from a GPU host.ddev test xinference(unit),ddev test --lint, andddev validate(config, models, metadata, service-checks, readmes, openmetrics, typos, package). I also ran the check against the live supervisor and worker endpoints and all 32 metrics plus the health service check came through.