Repository navigation
Migrate elasticsearch client to 8.* (or latest 7.*) #31256
Description
Body
The eleasticsearch version we use is old (from 2021). It is eleasticsearch 7.* and since we are using elasticsearch-dsl, we cannot used elasticsearch 8 currently (first version of that was also released in 2021).
We are currently limited to < 7.15.0 additionally to being limited to 7, because 7.15.0 breaks our tests as we found out when we released some of the dependencies as part of #30067
We should migrate to a newer version of elasticsearch. There are two options:
- Fix the tests and migrate to lates 7.* version - and bump to latest released 7* by fixing our tests (possible for short term).
- Migreate to 8 version (better long term)
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
[-]Migrate elasticsearch to 8[/-][+]Migrate elasticsearch client to 8.* (or latest 7.*)[/+]Owen-CH-Leung commentedon May 25, 2023
Since
elasticsearch-dsldoesn't support elasticsearch 8, if we are to migrate to the official elasticsearch 8, does it mean we need to deprecateelasticsearch-dslalso ?potiuk commentedon May 26, 2023
If we do yes. But that also might be reason why we don't do it.
Owen-CH-Leung commentedon May 27, 2023
Ok. Please assign this issue to me. I'll take some time to study the current usage of this package and propose whether to upgrade to 8.* or latest 7.*
potiuk commentedon May 27, 2023
Done.
Owen-CH-Leung commentedon May 29, 2023
I saw
elasticsearch_dslwas only used in 3 files as follows (one usage in taskhandler & 2 in testing) :airflow/airflow/providers/elasticsearch/log/es_task_handler.py
Line 291 in fc36d00
airflow/docker_tests/test_prod_image.py
Line 130 in fc36d00
airflow/tests/providers/elasticsearch/log/test_es_task_handler.py
Line 344 in fc36d00
Given the utilization of
elasticsearch_dslis fairly limited, I'll try to upgrade toelasticsearch8.* and deprecateelasticsearch_dslOwen-CH-Leung commentedon May 29, 2023
hmmmm - I just realize that we also use
elasticsearch-dbapi, which requireselasticsearch==7.13.4https://github.com/apache/airflow/blob/main/airflow/providers/elasticsearch/provider.yaml#L60
https://github.com/preset-io/elasticsearch-dbapi/blob/master/requirements.txt#L9
Owen-CH-Leung commentedon May 29, 2023
I feel like if we are to upgrade to elasticsearch 8, we need to divide it into chunk of works (coz we will be deprecating 2 packages). Since
elasticsearch-dbapiis used to construct theElasticsearchSQLHook, we might first need to refactor it using the existing officialelasticsearchlibrary. Then we can go further to refactor the taskhandler that useselasticsearch_dslto use the existingelasticsearchlibrary. Finally we can attempt to upgrade to 8, remove those 2 libraries, and fix the failing tests. What do you think ? @potiukpotiuk commentedon Jun 8, 2023
sure