Repository navigation
Conversation
Collect the Prometheus metrics exposed by LocalAI: API request latency, token usage and billed requests by model, requests without a usage record, model failover target health and switches, and chat context compression. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Contributor
|
Opened DOCS-16335 to assign a Docs writer and follow up with editorial review. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a new Agent-based integration for LocalAI, the open source OpenAI-compatible server for running LLM, embedding, image and audio models on your own hardware.
The check scrapes the Prometheus endpoint LocalAI serves at
/metricsand maps the metrics tolocalai.*:api_callhistogram, in seconds).The
endpointlabel of the usage metrics is renamed toapi_endpoint, sinceendpointis the tag the Agent sets for the scraped URL, and theotel_scope_*labels added by the OpenTelemetry exporter are dropped.Motivation
LocalAI is a popular self-hosted alternative to the OpenAI API, and there is no integration for it today. The vLLM, SGLang and TGI integrations cover single inference engines; LocalAI sits in front of several backends, so token usage and failover are the useful signals here.
I'm happy to be listed as the maintainer of this integration.
Review checklist
Additional Notes
tests/fixtures/metrics.txtis a real scrape from LocalAI v4.12.1 (darwin-arm64 release, llama-cpp backend) serving a small GGUF model, with a failover chain whose first target is broken and a model with context compression enabled, so every mapped metric shows up./metricsresponse twice (the HTTP compression middleware wraps the already compressed promhttp output), which breaks parsing. The check sendsAccept-Encoding: identityby default; there is a unit test for it.localai_cost_usd_totalis not mapped: it is only emitted when a pricing version is attached to the usage record, and nothing sets one yet. The agent, PII and motion counters are left out for now as well, since they belong to separate features.ddev test localai(unit),ddev test --lint, andddev validate(config, models, metadata, service-checks, readmes, openmetrics, typos, package). I also ran the check against the live server and all 15 metrics plus the health service check came through.