Repository navigation
Druid Operator is not getting host #27029
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
We use Airflow 2.3.3. I see that this test is successful, but I take a this error. This is the picture
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 792, in get_adapter
raise InvalidSchema(f"No connection adapters were found for {url!r}")
I take dag like this
Also I tried this type but I failed
ingestion_2 = SimpleHttpOperator(
task_id='test_task',
method='POST',
http_conn_id=DRUID_CONN_ID,
endpoint='/druid/indexer/v1/task',
data=json.dumps(read_file),
dag=dag,
do_xcom_push=True,
headers={
'Content-Type': 'application/json'
},
response_check=lambda response: response.json()['Status'] == 200)I get this log
[2022-10-13, 06:16:46 UTC] {http.py:143} ERROR - {"error":"Missing type id when trying to resolve subtype of [simple type, class org.apache.druid.indexing.common.task.Task]: missing type id property 'type'\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 1]"}
I don't know this is bug or issue or networking problem but can we check this?
P.S - We use Airflow on Kubernetes so that we can not debug it.
What you think should happen instead
No response
How to reproduce
No response
Operating System
Kubernetes
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Oct 13, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
emincanoguz11 commentedon Oct 13, 2022
P.S I updated Airflow 2.4.0 but it is not work :(
o-nikolas commentedon Oct 13, 2022
It looks like the druid hook is getting the protocol to use from the conn type, and since you're using a Druid connection that ends up being
druid://in your url, butrequestsdoesn't find any adapters for that protocol:airflow/airflow/providers/apache/druid/hooks/druid.py
Lines 65 to 69 in 0d78ba5
It also looks like you're not providing an endpoint, the default seems to be
{"endpoint": "druid/indexer/v1/task"}I have no Druid experience so I can't advise much more than that unfortunately.
emincanoguz11 commentedon Oct 14, 2022
Hi Nikolas,
We saw that this type but I cannot change this. Because we are using Airflow on Kubernetes. But also we gave HTTP Operator to Druid Broker for this reason but it is not worked.
emincanoguz11 commentedon Oct 14, 2022
Hi Nikolas Again,
I tested with standalone Airflow and I tested only http but this is not worked :(
emincanoguz11 commentedon Oct 14, 2022
I solved problem. Thanks but this logs are so terrible to defining this problem. Maybe we will create new logs for defining problem.
o-nikolas commentedon Oct 14, 2022
Hey @emincanoguz11
That sounds great! Can you provide more details on what the actual problem was and what logging you think would improve things? That would be massively helpful to others in the future. You could even open a PR yourself and become one of the many contributors to the airflow project! 🚀
emincanoguz11 commentedon Oct 14, 2022
This is ingestion file problem but I see in debug mode for in details. We must see log in Airflow for about ingestion problem. I hope I take PR. Thanks for answering.
o-nikolas commentedon Oct 14, 2022
Okay, I've assigned this ticket to you for now :)
emincanoguz11 commentedon Oct 16, 2022
Hi Again,
I develop the new log system. First of all we see that
but in my development I see that clearly the main reason of error.
I take PR in few days. Thanks for all kinding supports.
emincanoguz11 commentedon Oct 16, 2022
Sorry for removed assigment. It my fault. @o-nikolas can you give assignment to me please again
o-nikolas commentedon Oct 16, 2022
No worries, done :)
emincanoguz11 commentedon Oct 21, 2022
@o-nikolas Hi Nikolas,
If you have a free time please check this PR. Thanks for your support
PR Link