app/vlagent/kubernetescollector: add metadata-based filtering for Kubernetes logs

[vadimalekseev]

Describe Your Changes

Fixes #923

Checklist

The following checks are mandatory:

• My change adheres to VictoriaMetrics contributing guidelines.
• My change adheres to VictoriaMetrics development goals.

[valyala]

@vadimalekseev , thank you for the pull request!

[vadimalekseev]

@valyala, I've been giving this some thought: do you think we should invert the logic of the filter flag? For example, which would be more intuitive for a user who wants to avoid collecting logs from the frontend namespace:
-kubernetesCollector.filter="namespace:=frontend"
or
-kubernetesCollector.filter="not namespace:=frontend"?

Current logic is the first one.

[valyala]

@vadimalekseev , I renamed -kubernetesCollector.filter command-line flag to -kubernetesCollector.excludeFilter in the commit 612d8af in order to make it less confusing for users to use. This flag accepts filters for exclusion of the logs, not for inclusion of the logs.

[func25]

How about pod annotations? I was expecting to be able to filter similarly to Prometheus-style scrape annotations :s

[valyala]

How about pod annotations? I was expecting to be able to filter similarly to Prometheus-style scrape annotations

Agreed :) @vadimalekseev , what do you think about adding the ability to filter by node / pod annotations? There is no need to add these annotations to the set of log stream labels.

[vadimalekseev]

Yes, the is a good idea, already working on it

[vadimalekseev]

The implementation might take longer than I planned because we store commonFields on disk in checkpointsDB without any mechanism to update them.

As a result, even if we add this information, it won't work for existing (old) pods. I want to move away from storing commonFields on disk. This is better architectural solution, but we need to come up with a workaround for restoring file reading in restarts case and still being able to remove checkpoints for deleted files. I have an idea and I'm currently working on it.

I will create an issue for this.

[valyala]

I think it should be OK to assume that labels and annotations are immutable for already deployed pods. In this case you don't need to update this data in the checkpointsDB. This can be documented. This should be easier to do than dynamic update of the labels and annotations for already discovered pods.

[vadimalekseev]

@valyala agree. Created PR to include and exclude Node/Pod labels and annotations: #934
I will continue working on the metadata update issue.