Broker initialization error when using global Zookeeper #602
Description
Describe the bug
2025-04-29T05:42:01,615+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2025-04-29T05:42:01,624+0000 [main-SendThread(pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local/10.0.1.61:2181.
2025-04-29T05:42:01,625+0000 [main-SendThread(pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025-04-29T05:42:01,633+0000 [main-SendThread(pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.0.0.144:43116, server: pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local/10.0.1.61:2181
2025-04-29T05:42:01,645+0000 [main-SendThread(pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local/10.0.1.61:2181, session id = 0x200757815ff0028, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null zxid: -1
Node does not exist: /admin/clusters/cluster3
2025-04-29T05:42:01,657+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 1
pulsar cluster cluster3 isn't initialized yet ... check in 3 seconds ...
To Reproduce
Steps to reproduce the behavior:
- helm install pulsar-cs apache/pulsar -f config-store-values.yaml
- helm install cluster3 apache/pulsar -f cluster3-values.yaml --namespace pulsar-cluster3 --create-namespace --set affinity.anti_affinity=false --set kube-prometheus-stack.enabled=false --set clusterName=cluster3
[root@master examples]# cat cluster3-values.yaml
metadataPrefix: "/configuration-store"
pulsar_metadata:
configurationStore: "pulsar-cs-zookeeper.pulsar-cs.svc.cluster.local"
configurationStoreMetadataPrefix: "/configuration-store"
configurationStorePort: 2181
components:
zookeeper: true
bookkeeper: true
broker: true
proxy: true
[root@master examples]#
- See error
[root@master examples]# kubectl -n pulsar-cluster3 get po
NAME READY STATUS RESTARTS AGE
cluster3-grafana-6bc457ddcf-6xngh 1/1 Running 0 5m28s
cluster3-kube-state-metrics-57c86d4bb6-qx2jm 1/1 Running 0 5m28s
cluster3-prometheus-node-exporter-5g5zz 0/1 Pending 0 5m28s
cluster3-prometheus-node-exporter-hjzs6 0/1 Pending 0 5m28s
cluster3-prometheus-node-exporter-t85qj 0/1 Pending 0 5m28s
cluster3-pulsar-bookie-0 1/1 Running 0 5m27s
cluster3-pulsar-bookie-1 1/1 Running 0 5m26s
cluster3-pulsar-bookie-2 1/1 Running 0 5m27s
cluster3-pulsar-bookie-3 1/1 Running 0 5m26s
cluster3-pulsar-bookie-init-6f29m 0/1 Completed 0 5m27s
cluster3-pulsar-broker-0 0/1 Init:0/2 0 5m27s
cluster3-pulsar-broker-1 0/1 Init:0/2 0 5m27s
cluster3-pulsar-broker-2 0/1 Init:0/2 0 5m27s
cluster3-pulsar-proxy-0 0/1 Init:1/2 2 (22s ago) 5m27s
cluster3-pulsar-proxy-1 0/1 Init:1/2 2 (22s ago) 5m27s
cluster3-pulsar-proxy-2 0/1 Init:1/2 2 (23s ago) 5m27s
cluster3-pulsar-pulsar-init-x2f8v 0/1 Completed 0 5m27s
cluster3-pulsar-recovery-0 1/1 Running 0 5m27s
cluster3-pulsar-toolset-0 1/1 Running 0 5m27s
cluster3-pulsar-zookeeper-0 1/1 Running 0 5m27s
cluster3-pulsar-zookeeper-1 1/1 Running 0 5m27s
cluster3-pulsar-zookeeper-2 1/1 Running 0 5m27s
cluster3-victoria-metrics-operator-745db79bb9-6xrds 1/1 Running 0 5m28s
vmagent-cluster3-victoria-metrics-k8s-stack-6ccdd854bb-rd25l 2/2 Running 0 5m25s
vmsingle-cluster3-victoria-metrics-k8s-stack-dd5c5fdf5-rn6mh 1/1 Running 0 5m9s
[root@master examples]#
Expected behavior
broker, proxy running
Desktop (please complete the following information):
[root@master examples]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.7", GitCommit:"07a61d861519c45ef5c89bc22dda289328f29343", GitTreeState:"clean", BuildDate:"2023-10-18T11:42:32Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.7", GitCommit:"07a61d861519c45ef5c89bc22dda289328f29343", GitTreeState:"clean", BuildDate:"2023-10-18T11:33:23Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
[root@master examples]#
chart version: pulsar-4.0.0
Additional context
get {{ .Values.configurationStoreMetadataPrefix }}/admin/clusters/{{ template "pulsar.cluster.name" . }}
should be
get {{ .Values.pulsar_metadata.configurationStoreMetadataPrefix }}/admin/clusters/{{ template "pulsar.cluster.name" . }}
get {{ .Values.metadataPrefix }}/admin/clusters/{{ template "pulsar.cluster.name" . }}
should be
get {{ .Values.pulsar_metadata.configurationStoreMetadataPrefix }}/admin/clusters/{{ template "pulsar.cluster.name" . }}
in cluster3-values.yaml, for the proxy to function, samemetadataPrefix and configurationStoreMetadataPrefix are set the same
metadataPrefix for local Zookeeper，configurationStoreMetadataPrefix for global Zookeeper
I'm not sure if this understanding is correct