Repository navigation
KubernetesPodOperator : TypeError: 'NoneType' object is not iterable #19369
Description
Apache Airflow Provider(s)
cncf-kubernetes
Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==2.0.2
Apache Airflow version
2.1.2
Operating System
GCP Container-Optimized OS
Deployment
Composer
Deployment details
No response
What happened
[2021-11-02 14:16:27,086] {pod_launcher.py:193} ERROR - Error parsing timestamp. Will continue execution but won't update timestamp
[2021-11-02 14:16:27,086] {pod_launcher.py:149} INFO - rpc error: code = NotFound desc = an error occurred when try to find container "8f8c2f3dce295f70ba5d60175ff847854e05ab288f7efa3ce6d0bd976d0378ea": not found
[2021-11-02 14:16:28,152] {taskinstance.py:1503} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1158, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1333, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1363, in _execute_task
result = task_copy.execute(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 367, in execute
final_state, remote_pod, result = self.create_new_pod_for_operator(labels, launcher)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 521, in create_new_pod_for_operator
final_state, remote_pod, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 154, in monitor_pod
if not self.base_container_is_running(pod):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 215, in base_container_is_running
status = next(iter(filter(lambda s: s.name == 'base', event.status.container_statuses)), None)
TypeError: 'NoneType' object is not iterable
in the GKE logs I see
airflow-XXX.XXXXX.b14f7e38312c4e83984d1d7a3987655a
"Pod ephemeral local storage usage exceeds the total limit of containers 10Mi. "
So the pod failed because I set to low limit for local storage , but the airflow operator should not raise an exception but fail normally.
What you expected to happen
The KubernetesPodOperator should managed this kind of error
How to reproduce
launch KubernetesPodOperator with a very small local disk limit and run a container using more than this limit
KubernetesPodOperator(
task_id="XXX",
name="XXXXXXXXX",
namespace="default",
resources={'limit_memory': "512M",
'limit_cpu': "250m",
'limit_ephemeral_storage': "10M"},
is_delete_operator_pod=True,
image="any_image_using_more_than_limit_ephemeral_storage")Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Nov 15, 2021
Sounds like an interesting edge case and rather simple to debug and track - would you lke maybe to make a PR with a fix for it @raphaelauv ?
ulisesojeda commentedon Nov 17, 2021
@raphaelauv could you provide an image example?
raphaelauv commentedon Nov 18, 2021
@ulisesojeda
minimal reproductive example :
ulisesojeda commentedon Nov 19, 2021
Thanks @raphaelauv.
I've opened this PR #19713
@potiuk could you check it please?