_stream field must start with '{'

[mnow-cd]

Describe the bug

Hello,

I have a rather simple log panel with the following query:

kubernetes.pod_labels.app.kubernetes.io/name:in($name)
| sort by (_time) desc
| limit $limit

but this randomly fails with the below error:

{
    "results": {
        "A": {
            "error": "_stream field must start with '{'",
            "errorSource": "plugin",
            "status": 500,
            "frames": []
        }
    }
}

request body:

{
    "queries": [
        {
            "datasource": {
                "type": "victoriametrics-logs-datasource",
                "uid": "cekafjet83mdca"
            },
            "editorMode": "code",
            "expr": "kubernetes.pod_labels.app.kubernetes.io/name:in(\"api\")\n| sort by (_time) desc\n| limit 100",
            "queryType": "instant",
            "refId": "A",
            "maxLines": 1000,
            "legendFormat": "",
            "extraFilters": "",
            "datasourceId": 6,
            "intervalMs": 2000,
            "maxDataPoints": 1588
        }
    ],
    "from": "1771338455898",
    "to": "1771342055898"
}

^{Note: In the request the value of `queryType` is set to `instant`, but in the panel the selected type option is `Raw logs`.}

Grafana logs

logger=plugin.victoriametrics-logs-datasource t=2026-02-17T16:05:23.16514051Z level=error msg="_stream field must start with '{'"
logger=plugin.victoriametrics-logs-datasource t=2026-02-17T16:05:23.165334232Z level=error msg="Partial data response error" pluginId=victoriametrics-logs-datasource refID=A statusSource=plugin uname=admin dsName=victoriametrics-logs-datasource dsUid=cekafjet83mdca error="_stream field must start with '{'" pluginVersion=0.24.1 status=500 endpoint=queryData
logger=plugin.victoriametrics-logs-datasource t=2026-02-17T16:05:23.165375845Z level=error msg="Plugin Request Completed" endpoint=queryData pluginId=victoriametrics-logs-datasource pluginVersion=0.24.1 status=error duration=38.775047ms statusSource=plugin uname=admin dsName=victoriametrics-logs-datasource dsUid=cekafjet83mdca

Changing the time range often helps. Strangely, reverting to the previous value no longer triggers the error until it randomly comes back later.
Removing | limit $limit also helps.

The problem started with the update of:

• VL from v1.28.0 to the newest version (but I think it is a plugin problem - the query executed directly in VL never fails)
• datasource plugin to newest version, but unfortunately I don't remember which version it was before the update

To Reproduce

Change time range or limit value

Versions of VictoriaLogs datasource and VictoriaLogs backend

• VictoriaLogs datasource: 0.24.1
• VictoriaLogs backend: v1.45.0

Link to dashboard in Victoria Metrics

No response

Please provide dashboard JSON if it is possible

{
  "__inputs": [
    {
      "name": "DS_VICTORIAMETRICS-LOGS-DATASOURCE",
      "label": "victoriametrics-logs-datasource",
      "description": "",
      "type": "datasource",
      "pluginId": "victoriametrics-logs-datasource",
      "pluginName": "VictoriaLogs"
    }
  ],
  "__elements": {},
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "12.3.2"
    },
    {
      "type": "panel",
      "id": "logs",
      "name": "Logs",
      "version": ""
    },
    {
      "type": "datasource",
      "id": "victoriametrics-logs-datasource",
      "name": "VictoriaLogs",
      "version": "0.24.1"
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "datasource": {
        "type": "victoriametrics-logs-datasource",
        "uid": "${DS_VICTORIAMETRICS-LOGS-DATASOURCE}"
      },
      "gridPos": {
        "h": 18,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "dedupStrategy": "none",
        "enableInfiniteScrolling": true,
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showControls": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "pluginVersion": "12.3.2",
      "targets": [
        {
          "datasource": {
            "type": "victoriametrics-logs-datasource",
            "uid": "${DS_VICTORIAMETRICS-LOGS-DATASOURCE}"
          },
          "editorMode": "code",
          "expr": "kubernetes.pod_labels.app.kubernetes.io/name:in($name)\n| sort by (_time) desc\n| limit $limit",
          "queryType": "instant",
          "refId": "A"
        }
      ],
      "title": "Logs (newest first)",
      "type": "logs"
    }
  ],
  "preload": false,
  "refresh": "",
  "schemaVersion": 42,
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": "*",
        "allowCustomValue": true,
        "current": {},
        "datasource": {
          "type": "victoriametrics-logs-datasource",
          "uid": "${DS_VICTORIAMETRICS-LOGS-DATASOURCE}"
        },
        "definition": "",
        "includeAll": true,
        "label": "Service",
        "multi": true,
        "name": "name",
        "options": [],
        "query": {
          "field": "kubernetes.pod_labels.app.kubernetes.io/name",
          "limit": 100,
          "query": "",
          "refId": "VictoriaLogsVariableQueryEditor-VariableQuery",
          "type": "fieldValue"
        },
        "refresh": 1,
        "regex": "",
        "sort": 1,
        "type": "query"
      },
      {
        "current": {
          "text": "100",
          "value": "100"
        },
        "label": "Limit",
        "name": "limit",
        "options": [
          {
            "selected": true,
            "text": "100",
            "value": "100"
          },
          {
            "selected": false,
            "text": "200",
            "value": "200"
          },
          {
            "selected": false,
            "text": "500",
            "value": "500"
          },
          {
            "selected": false,
            "text": "1000",
            "value": "1000"
          }
        ],
        "query": "100, 200, 500, 1000",
        "type": "custom"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Logs Dashboard",
  "uid": "cekdblgv9nxtsz",
  "version": 2,
  "weekStart": ""
}

Additional information

No response

[arturminchukov]

@mnow-cd thanks for bug reporting!
I'm trying to reproduce your steps, but it's working as expected for me. Could you please also specify which Grafana version you use?

[mnow-cd]

Im using Grafana v12.3.2, but it also worked in Grafana v11.5.*.

Some videos:
1)
https://github.com/user-attachments/assets/76d3d942-af03-44a0-ba0c-777733f9985a
2)
https://github.com/user-attachments/assets/9b474e81-bdd6-4ff1-8eff-d408cb9181f2

[arturminchukov]

@mnow-cd
Could you create a new datasource with our playground backend VictoriaLogs - https://play-vmlogs.victoriametrics.com. I think it still should work with your current dashboard, and try reproduce the problem with this new datasource. Is the problem still exist?
I just want to figure out the problem, because the query looks valid, and I still cannot reproduce your problem.

[mnow-cd]

@arturminchukov
I changed the data source to the backend you mentioned, and I was also unable to reproduce the problem.
To rule out the problem with a incorrectly configured data source, I added it again, but that didn't help either.
This problem occurs in 3 different clusters.

I can't quite figure out how to reproduce it, it seems to be random. However, removing | limit $limit from the query always helps.

Thanks for the help.

[arturminchukov]

@mnow-cd
Do you have some services before the VictoriaLogs backend? Is it possible that these services/apps can change the query?

[mnow-cd]

No, nothing in between.
Datasource URL directly points to the VL Kubernetes service:
http://victoria-logs-server.logs.svc.cluster.local:9428

[arturminchukov]

@mnow-cd Is it possible to also check the logs of the VL pod in Kubernetes during the request that is broken?

[mnow-cd]

Yes, I checked the logs but unfortunately such requests do not produce any logs.