Repository navigation
Closed
Labels
Description
Expected Behavior
No error, connection should be closed.
Actual Behavior
RuntimeError: Invalid error code
Snippet to Reproduce the Problem
zk = KazooClient(hosts = f'{ip}', verify_certs = False, timeout = 10)
zk.start()
zk.get_children(target_path)
It is happening when zookeeper running with a few extra jvm parameters
-Djava.security.auth.login.config=/etc/kafka/zookeeper_jaas.conf
-Dzookeeper.allowSaslFailedClients=false
-Dzookeeper.sessionRequireClientSASLAuth=true
Default library doesn't handle well error code -124 which declares that client should authorise himself. Our application is ok with closing connection altogether and not messing with it.
I've fixed it with monkey patching:
def new_read_response(self, header, buffer, offset):
client = self.client
request, async_object, xid = client._pending.popleft()
if header.zxid and header.zxid > 0:
client.last_zxid = header.zxid
if header.xid != xid:
exc = RuntimeError('xids do not match, expected %r '
'received %r', xid, header.xid)
async_object.set_exception(exc)
raise exc
# Determine if it's an exists request and a no node error
exists_error = (header.err == NoNodeError.code and
request.type == Exists.type)
# Set the exception if it's not an exists error
if header.err and not exists_error:
# -124 error handler
if header.err == -124:
async_object.set(False)
async_object.set_exception(NoAuthError())
return CLOSE_RESPONSE
callback_exception = EXCEPTIONS[header.err]()
self.logger.debug(
'Received error(xid=%s) %r', xid, callback_exception)
if async_object:
async_object.set_exception(callback_exception)
elif request and async_object:
if exists_error:
# It's a NoNodeError, which is fine for an exists
# request
async_object.set(None)
else:
try:
response = request.deserialize(buffer, offset)
except Exception as exc:
self.logger.exception(
"Exception raised during deserialization "
"of request: %s", request)
async_object.set_exception(exc)
return
self.logger.debug(
'Received response(xid=%s): %r', xid, response)
# We special case a Transaction as we have to unchroot things
if request.type == Transaction.type:
response = Transaction.unchroot(client, response)
async_object.set(response)
# Determine if watchers should be registered
watcher = getattr(request, 'watcher', None)
if not client._stopped.is_set() and watcher:
if isinstance(request, (GetChildren, GetChildren2)):
client._child_watchers[request.path].add(watcher)
else:
client._data_watchers[request.path].add(watcher)
if isinstance(request, Close):
self.logger.log(BLATHER, 'Read close response')
return CLOSE_RESPONSE
kazoo.protocol.connection.ConnectionHandler._read_response = new_read_response
Logs with logging in DEBUG mode
[2023-08-08 13:59:25,212][INFO]: connection.py:639:_connect - Connecting to 192.xxx.xxx.xxx(192.xxx.xxx.xxx):2181, use_ssl: False
[2023-08-08 13:59:25,223][DEBUG]: connection.py:312:_submit - Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None)
[2023-08-08 13:59:25,236][INFO]: client.py:532:_session_callback - Zookeeper connection established, state: CONNECTED
[2023-08-08 13:59:25,236][DEBUG]: connection.py:312:_submit - Sending request(xid=1): GetChildren(path='/', watcher=None)
[2023-08-08 13:59:25,248][ERROR]: connection.py:631:_connect_attempt - Unhandled exception in connection loop
Traceback (most recent call last):
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
response = self._read_socket(read_timeout)
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 453, in _read_socket
return self._read_response(header, buffer, offset)
File "/opt/scanners/m_zookeeper.py", line 41, in new_read_response
callback_exception = EXCEPTIONS[header.err]()
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/exceptions.py", line 54, in _invalid_error_code
raise RuntimeError('Invalid error code')
RuntimeError: Invalid error code
[2023-08-08 13:59:25,248][INFO]: client.py:537:_session_callback - Zookeeper session closed, state: CLOSED
Exception in thread Thread-3 (zk_loop):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 512, in zk_loop
if retry(self._connect_loop, retry) is STOP_CONNECTING:
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/retry.py", line 126, in __call__
return func(*args, **kwargs)
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 552, in _connect_loop
status = self._connect_attempt(host, hostip, port, retry)
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
response = self._read_socket(read_timeout)
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 453, in _read_socket
return self._read_response(header, buffer, offset)
File "/opt/scanners/m_zookeeper.py", line 41, in new_read_response
callback_exception = EXCEPTIONS[header.err]()
File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/exceptions.py", line 54, in _invalid_error_code
raise RuntimeError('Invalid error code')
RuntimeError: Invalid error code
Specifications
- Kazoo version: 2.9.0
- Result of
pip listcommand: kazoo only - Zookeeper version: Confluent Zookeper latest
- Zookeeper configuration: put here any useful ZK configuration (authentication, encryption, number of ZK members, number of (concurrent?) clients, Java version, krb5 version, etc.)
- Python version: 3.9
- OS: ubuntu 22
Activity
StephenSorriaux commentedon Aug 9, 2023
Hello,
Thank you for the issue. Indeed the
ZSESSIONCLOSEDREQUIRESASLAUTHerror (-124) is not handled properly (reminds me of a PR where @ztzg said some errors were not handled at all in the lib).Some documentation for reference: https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md?plain=1#L1615
nkormakov commentedon Aug 9, 2023
Hi Stephen!
What is the proper way of handling it will be? I'm aware that my workaround is not up to standards yet, but it does not produces some secondary errors or halts
ceache commentedon Aug 9, 2023
nkormakov commentedon Aug 9, 2023
Ok, I'll look into that and make a PR for you to judge.
jeffwidman commentedon Feb 6, 2024
Fixed by #743
fix(protocol): defer CONNECTED state until auth completes and unify s…
fix(protocol): defer CONNECTED state until auth completes and unify s…
fix(protocol): defer CONNECTED state until auth completes and unify s…
fix(protocol): defer CONNECTED state until auth completes and unify s…
fix(protocol): defer CONNECTED state until auth completes and unify s…