When using Grafana's Tempo plugin against a VictoriaTraces datasource, leaving an empty value on a filter chip in the Search tab causes the request to fail with 503 Service Temporarily Unavailable instead of a structured 400 Bad Request with a parser error message. The 503 surfaces in Grafana's error toast and is opaque to the user.
Environment
- VictoriaTraces:
v0.8.1
- Deployment shape:
VTCluster (operator) — vtinsert + vtselect + vtstorage + requestsLoadBalancer
- Grafana datasource: type
tempo, pointing at vtselect:10471/select/tempo
Steps to reproduce
- In Grafana Explore, select a Tempo datasource backed by VictoriaTraces.
- Switch to Search query type.
- Add a tag filter chip and leave its value empty (e.g.,
span resource.service.name= with no value).
- Run the query.
Expected
A 400 Bad Request (or another 4xx) with a structured error body indicating a TraceQL parser/validation problem. The user should see what about the query is invalid.
Actual
A 503 Service Temporarily Unavailable response. The error body in our case is the nginx default error page (<center>nginx</center>), which strongly suggests the upstream VT response was either an unhandled error / panic / non-graceful timeout that our ingress wrapped — or VT genuinely returned a 5xx.
Open question for triage
We have not yet confirmed whether the 5xx originates inside VictoriaTraces or whether some intermediate layer is converting a structured 4xx into a 503. If the VT pod logs show a panic, error, or non-4xx response on receiving a query with an empty filter value, this is a VT bug. If VT cleanly returns 4xx, this is an infrastructure issue on our end and we'll close the issue.
Either way, a maintainer comment confirming what vtselect returns for a query with an empty filter value would unblock investigation.
Reporter context: Originally surfaced in internal feedback against our VictoriaTraces evaluation; not blocking but degrades the UX of the Search tab significantly.
When using Grafana's Tempo plugin against a VictoriaTraces datasource, leaving an empty value on a filter chip in the Search tab causes the request to fail with
503 Service Temporarily Unavailableinstead of a structured400 Bad Requestwith a parser error message. The 503 surfaces in Grafana's error toast and is opaque to the user.Environment
v0.8.1VTCluster(operator) — vtinsert + vtselect + vtstorage + requestsLoadBalancertempo, pointing atvtselect:10471/select/tempoSteps to reproduce
span resource.service.name=with no value).Expected
A
400 Bad Request(or another 4xx) with a structured error body indicating a TraceQL parser/validation problem. The user should see what about the query is invalid.Actual
A
503 Service Temporarily Unavailableresponse. The error body in our case is the nginx default error page (<center>nginx</center>), which strongly suggests the upstream VT response was either an unhandled error / panic / non-graceful timeout that our ingress wrapped — or VT genuinely returned a 5xx.Open question for triage
We have not yet confirmed whether the 5xx originates inside VictoriaTraces or whether some intermediate layer is converting a structured 4xx into a 503. If the VT pod logs show a panic, error, or non-4xx response on receiving a query with an empty filter value, this is a VT bug. If VT cleanly returns 4xx, this is an infrastructure issue on our end and we'll close the issue.
Either way, a maintainer comment confirming what
vtselectreturns for a query with an empty filter value would unblock investigation.Reporter context: Originally surfaced in internal feedback against our VictoriaTraces evaluation; not blocking but degrades the UX of the Search tab significantly.