TLS for MQTT

[jardleex]

Hello,

would it be possible to support TLS on the MQTT sink?
I wanted to lock down my home automation a bit more and as I got quite some stuff going on via MQTT it'll be great to secure that.
For some setup's it may also be required to skip certificate validation. I'm using Lets Encrypt here which should work with enabled verification.

Thank you for your work.
Do you have any way of donating to you?

[AnotherDaniel]

Hi @jardleex, I was afraid that someday, someone would ask for this...

Seriously though: I'm on it. Will result in a bunch of new config vars for this plugin, but that's what certificate handling gets you... ;-)

[AnotherDaniel]

Ok, just release v1.5.5 with TLS support for the MQTT sink plugin - give it a whirl, hope this works for you!

[jardleex]

Hello @AnotherDaniel

also this change works pretty good.
Not the 1h time shift between smahub and iobroker comes as smahub runs on UTC time while iobroker uses TZ: Europe/Berlin. smahub seems not to use this environment variable when set. Just a minor thing.

The only thin I noticed worth mentioning is the 2024-02-07 07:45:33 ERROR Missing CA certificate error. I'm using traefik with LetsEncrypt with a valid/public domain and thus can offer valid certificates. So in this case here it's no actual error for the setup.

# docker-compose service:
  smahub:
    image: ghcr.io/anotherdaniel/smahub:v1.5.5
    restart: unless-stopped
    depends_on:
      iobroker:
        condition: service_healthy
    environment:
      SMAHUB_VERBOSE: true
      SMAHUB_VERBOSER: true
      SHM2_ENABLED: false
      TRIPOWERX_ENABLED: true
      TRIPOWERX_ADDRESS: redacted
      TRIPOWERX_USER: smahub
      TRIPOWERX_PASSWORD: 'redacted'
      TRIPOWERX_PROTOCOL: https
      TRIPOWERX_VERIFYTLS: false
      TRIPOWERX_UPDATEFREQ: 20
      MQTT_ENABLED: true
      MQTT_ADDRESS: redacted
      MQTT_PORT: 2884
      MQTT_TLS: 2
      MQTT_USER: redacted
      MQTT_PASSWORD: 'redacted'
      MQTT_UPDATEFREQ: 20

# smahub logs
2024-02-07 07:45:33 INFO     SHM2 plugin disabled
2024-02-07 07:45:33 INFO     Starting Tripower X source
2024-02-07 07:45:33 INFO     EV Charger plugin disabled
2024-02-07 07:45:33 INFO     demo source plugin disabled
2024-02-07 07:45:33 INFO     Starting MQTT sink
2024-02-07 07:45:33 INFO     HA-MQTT sink plugin disabled
2024-02-07 07:45:33 INFO     gen_ha_sensors sink plugin disabled
2024-02-07 07:45:33 ERROR    Missing CA certificate
2024-02-07 07:45:33 INFO     demo sink plugin disabled
2024-02-07 07:45:33 DEBUG    Starting new HTTPS connection (1): redacted:443
2024-02-07 07:45:33 INFO     TLS enabled
2024-02-07 07:45:33 DEBUG    MQTT sink connected to redacted:2884
2024-02-07 07:45:33 INFO     Connected to MQTT Broker

# iobroker log
mqtt.0	2024-02-07 08:45:33.400	info	Client [smahub] connected with secret 1707291933386_9697

[AnotherDaniel]

Have another go - timezone setting support should be there now, and that log message was a mistake.

[jardleex]

Confirmed for 1.5.6 that TZ environment variables works and missing CA certificate error is gone.

Thank you!

[jardleex]

Hi,
today I've updated the MQTT instance in iobroker which caused a restart of it.
smahub was not able to reconnect though, so I restarted it in the end which fixed the error message below:

2024-02-14 11:42:21 INFO     Got 401 - trying reauth
2024-02-14 12:42:28 INFO     Got 401 - trying reauth
2024-02-14 13:42:36 INFO     Got 401 - trying reauth
2024-02-14 14:42:43 INFO     Got 401 - trying reauth
2024-02-14 14:43:37 WARNING  Disconnected from MQTT Broker
Exception in thread Thread-1 (_thread_main):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
         ^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
                   ^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1243, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427)
2024-02-14 14:47:26 INFO     SHM2 plugin disabled
2024-02-14 14:47:26 INFO     Starting Tripower X source
2024-02-14 14:47:26 INFO     EV Charger plugin disabled
2024-02-14 14:47:26 INFO     demo source plugin disabled
2024-02-14 14:47:26 INFO     Starting MQTT sink
2024-02-14 14:47:26 INFO     HA-MQTT sink plugin disabled
2024-02-14 14:47:26 INFO     gen_ha_sensors sink plugin disabled
2024-02-14 14:47:26 INFO     demo sink plugin disabled
2024-02-14 14:47:26 INFO     TLS enabled
2024-02-14 14:47:26 INFO     Connected to MQTT Broker

Does that say something to you?
Thank you!

[AnotherDaniel]

Does it work again after restarting all involved components, do I see that correctly?
The googling I did so far is inconclusive, other than hinting at the dumpster-fire robustness that is the Python ecosystem 🫤

[jardleex]

Yes, everything worked again after a restart cycle.
I just wanted to raise this as it might contain valuable information for fixing something.
Generally the MQTT reconnect works flawless as I did multiple restarts of my broker since the new smahub feature.

I'd be okay for me to close this one here in this state.

[jardleex]

Nah, we can't close this one.
The error comes up every time I restart the broker. The mentioned restarts must have happened when I ain't used TLS yet but after #27.
I'll do some more testing and will report what I found.

[AnotherDaniel]

Ok, I'll also try to reproduce this.

[jardleex]

I did some tests again and only got the issue with MQTTS. With plain MQTT the reconnect worked just fine.
I triggered the error message above by simply restarting the mqtt broker.

[AnotherDaniel]

@jardleex just to let you know I haven't forgotten about this issue - not that much free mind-time at the moment, and when I tried to pick this up my TLS config (that was fine the first time around) refused to work. As I said, not forgotten...

[AnotherDaniel]

Add-on: I just figured out what my recent issue was, and have no idea why this ever worked? Anyways - broker-restart-reconnect works for me, but that might not be related to your scenario. But please have a go with v1.6.2 and let me know how that looks for you!