Repository navigation
Paramiko 2.9.x breaking SSH operator for some OpenSSH configurations #22194
Description
Apache Airflow version
2.2.4 (latest released)
What happened
After upgrading from 2.2.3 to 2.2.4 the Paramiko library upgraded from 2.8.1 to 2.9.2. After that change one of our SSH connections started to fail with the following error:
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1334, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1460, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1516, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.9/site-packages/airflow/providers/ssh/operators/ssh.py", line 175, in execute
raise AirflowException(f"SSH operator error: {str(e)}")
airflow.exceptions.AirflowException: SSH operator error: encountered RSA key, expected OPENSSH key
After looking at Paramiko changelog we saw that since 2.9.0 they have made non-backward compatible changes to the way they manage algorithms. In our case we are trying to connect to a server that has the following version of openssh and openssl : OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013.
What you expected to happen
Our existing SSH connections to keep working regardless of the version of OpenSSH used.
How to reproduce
Connect through the 2.2.4 SSHOperator to a server that do not support either RSA2 or the server-sig-algs protocol extension.
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
apache-airflow-providers-ssh==1!2.4.0
Deployment
Astronomer
Deployment details
No response
Anything else
In their changelog Paramiko indicates that to workaround that issue it is possible to specify disabled_algorithms={'keys': ['rsa-sha2-256', 'rsa-sha2-512']} in either SSHClient <paramiko.client.SSHClient.init> or Transport <paramiko.transport.Transport.init>.
I'm wondering what could be the impact to add this to the SSHClient?
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Mar 11, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
ashb commentedon Mar 11, 2022
What version of the SSH provider do you have installed?
talnicolas commentedon Mar 11, 2022
Does this answer your question?
potiuk commentedon Mar 12, 2022
You are free to downgrade to earlier paramiko version. Current limitation of SSH Provider is
paramiko>=2.6.0- the fact that paramiko >=2.9.0 does not accept your key is likely for a good reason (security) and if you choose to continue using version that is older (and might contain some security vulnerabilities) rather than upgrading your key - you can still do it.Follow the examples here:
https://airflow.apache.org/docs/docker-stack/build.html#example-of-upgrading-airflow-provider-packages
You can build your own image using this Dockerfile (feel free to do it)
Expecting that newer version of Airflow is using an older and potentially vulnerable version of a dependency to allow you to use potentially vulnerable key format is a bit selfish.
talnicolas commentedon Mar 12, 2022
Thank you for your insightful comment @potiuk.
Just a comment if I may, insulting people when they are just coming for advice from « experts » won’t help building a safe and inclusive community.
potiuk commentedon Mar 12, 2022
Unfortunately, I have no control whether you feel insulted or not, that was certainly not my intention, My intention was to raise your awareness that your expectations were a bit selfish as they did not include wider thinking about the community but focused on your own needs. This is what I think is a bit selfish. I just expressed how I perceived it.
But I certainly had no intention to insult you - if you feel insulted - sorrry for that - but again, I have completely no control on your feelings when I express mine. I suggest you to look at it at a wider context.
talnicolas commentedon Mar 12, 2022
Indeed you do not have that control, just like I don’t have any control on your ability to understand people’s request.
If that doesn’t feel to you like someone who came here specifically to understand that wider context you are talking about, well I’m sorry too.
That conversation is sterile now, I wish you a good continuation.
potiuk commentedon Mar 12, 2022
By all means - please propose a PR to expose this. You are most welcome.
Indeed I missed that part.
I was focusing more on the expectations: "Our existing SSH connections to keep working regardless of the version of OpenSSH used". But If you marked something as a bug, where this is a rather a feature request to expose this feature to the operators - this bug will be closed as invalid. This is not a bug and your
expectationis more of a wish, than expectation and is not abugby any means. Maybe that's why the part where you proposed a feature (in otherwise a bug report) was simply, well, missed.No hard feelings - really :). And again - if you felt "insulted" - then that was a total misunderstanding. I simply closed the bug, believing that you "complain on a bug that you expected to work" rather than "proposed a feature to make it work better". The latter is absolutely welcome - actually you don't even need to make an issue for it - just create a PR and proceed as usual. Airflow has almost 2000 contributors - mostly volunteers so you are free to propose this (and I will be happy to review and approve it if it follows all the criteria from CONTRIBUTING.rst - and this is a great way to become one.
Again, apologies if you felt insulted, that was definitely not intention.
potiuk commentedon Mar 12, 2022
This is really what I consider as "non selfish" - Identify a problem that small part of others might have, and propose a PR so that not only my problem is fixed but also others. For an open-source software that you get for free this is absolutely the best way to proceed.
swapniel99 commentedon Apr 18, 2023
Hi,
Just putting this here in case someone else faces this issue.
https://askubuntu.com/questions/1409105/ubuntu-22-04-ssh-the-rsa-key-isnt-working-since-upgrading-from-20-04
In case the server is on 22.0.4 then RSA keys are considered obsolete.
For me applying this solved the issue.