Support for Multi-tenant Kubernetes setup

[Sharathmk99]

Hello,
I'm trying to setup Grafana in Kubernetes cluster. Kubernetes cluster will be used by multiple project team and one project team can't access other project team apps. We managed to setup Kubernetes by creating namespace per project and controlling the access level to specific namespace only. When i installed Grafana and Loki it gets installed cluster level. Now how can i setup Grafana and Loki to provide access at namespace level logs.

Thank you for your time!.

[tombrk]

Okay, in theory some support is there.

Loki

Take a look at https://github.com/grafana/loki/blob/master/docs/operations.md#multi-tenancy, it describes how multi-tenancy is supposed to work in Loki.

In reality this probably means you are going to run a proxy which exposes the Loki endpoint for every tenant and adds the X-Scope-OrgID header based on the tenant used. You will also handle auth (e.g. mTLS using ServiceAccounts) there.

Grafana

Regarding Grafana, you will probably want to use Organizations (https://grafana.com/docs/guides/basic_concepts/#organization) to provide multi tenancy. Every organization would add it's own LokiProxyEndpoint as a loki datasource.

If this is not enough, every tenant could use it's own grafana instance.

This should give you some hints to get started, let me know if any questions pop up!

[Sharathmk99]

Hello @sh0rez ,
Loki is installed as kubernetes cluster wide right. When you tell run proxy in front of Loki server. How can i control the pods inside namespace to use proxy host?

Do i need to use sidecar for supporting multi-tenant?
Can i configure "scrape_configs" to send data to proxy server?

[tombrk]

Okay, I basically imagined the following:

Pod
+-------------+
|             |  auth
|    nginx <---------+
|             |      |
|    +  ^     |    +-+---------+    +------------+
|    |  |     |    |           |    |            |
|    v  +     |    |  :3100 <---------+ promtail |
|             |    |           |    |            |
|    loki     |    +-----------+    +------------+
|             |        Service
+-------------+

Basically, nginx and loki are running the same pod with the nginx sidecar being the one who has the Service port.

AuthN is handled by the nginx, for example using BasicAuth (note you should also implement TLS termination when using BasicAuth). As soon as the user is authenticated, the request is passed to loki with the special X-Scope-OrgID header set to the tenant.

nginx.conf

worker_processes 2;

events {
    use           epoll;
    worker_connections  128;
}

http {
    server {
        location / {
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/htpasswd;

            proxy_pass         http://echo:8080;
            proxy_redirect     default;
            proxy_set_header   X-Scope-OrgID $remote_user;
        }
    }
}

.htpasswd

team1:$apr1$KpL3FXAw$Lqd5qTLO.qghnNwEDyl230
team2:$apr1$LqDYD4JH$0iwaLSLO1.cYoUHGSohsj/

Using that information, the Loki server is able to do AuthZ and decide which data to return to the specific tenant.

Can i configure "scrape_configs" to send data to proxy server?

Yes, you actually have to, as the loki server does is not reachable outside of its pod:

clients:
  - url: http://loki.default.svc.cluster.local:3100/api/prom/push

From https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config:

# Sets the `Authorization` header on every scrape request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
  [ username: <string> ]
  [ password: <secret> ]
  [ password_file: <string> ]

Using this setup, Loki is fully encapsulated behind an auth layer without any information leaking through.

[tombrk]

And don't forget to make loki honor the header using the -auth_enabled=true flag

[Sharathmk99]

Hi @sh0rez, I have implemented little different. I have deployed Loki and promtail to separate namespace("monitoring") and deployed nginx reverse proxy in separate namespace("project1") and added org id header. I have configured Promtail Loki service URL as nginx URL.

From Grafana i'm adding datasource URL as nginx URL instead of Loki URL.

In this approach i'm using same Loki and Promtail for multiple tenant.

But now i'm not sure how can i configure multiple client.url for promtail so that job A can push data to nginxA and job B can push data to nginx B. Any idea?

[cyriltovena]

We don't support this currently but this could be interesting to configure a different client per promtail job.

[Sharathmk99]

@Kuqd Thank you for your reply. Is there any current alternative? Can sidecar approach be alternative?

[tombrk]

Yes, take a look at https://github.com/grafana/loki/blob/master/docs/promtail-setup.md#sidecar-method

[cyriltovena]

Yes I would say the sidecar might be a good alternative, never tried it. Let us know.

[Sharathmk99]

@cyriltovena I was unsuccessful to configure Promtail as sidecar. Can you please me with basic example, it would be great to get started. Please help.

[Sharathmk99]

@cyriltovena can you help me to implement different client per promtail job with org Id. I can change the source code as need with you guide. Please help to point which module I have change under promtail source code.

[cyriltovena]

I'd would like to think about a solution for everyone, give me some time to think about it.