DockerOperator not working from containerized Airflow not recognizing /var/run/docker.sock

Apache Airflow version: 2.1.1
Docker Image: apache/airflow:2.1.1-python3.8

Kubernetes version (if you are using kubernetes) (use kubectl version): Not running on k8s.

Environment:

Cloud provider or hardware configuration: DigitalOcean
OS (e.g. from /etc/os-release): Ubuntu
Kernel (e.g. uname -a): Linux airflow 5.11.0-22-generic #23-Ubuntu SMP Thu Jun 17 00:34:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Install tools: Installed via Docker & Docker Compose following instructions from official docker-compose installation docs.
Others: N/A

What happened:

What you expected to happen:

How to reproduce it:

Running with the DockerOperator causes the following error:

*** Reading local file: /opt/airflow/logs/docker_eod_us_equities/docker_command_sleep/2021-07-04T23:19:48.431208+00:00/2.log
[2021-07-04 23:49:53,886] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]>
[2021-07-04 23:49:53,902] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [queued]>
[2021-07-04 23:49:53,903] {taskinstance.py:1087} INFO - 
--------------------------------------------------------------------------------
[2021-07-04 23:49:53,903] {taskinstance.py:1088} INFO - Starting attempt 2 of 2
[2021-07-04 23:49:53,903] {taskinstance.py:1089} INFO - 
--------------------------------------------------------------------------------
[2021-07-04 23:49:53,911] {taskinstance.py:1107} INFO - Executing <Task(DockerOperator): docker_command_sleep> on 2021-07-04T23:19:48.431208+00:00
[2021-07-04 23:49:53,914] {standard_task_runner.py:52} INFO - Started process 3583 to run task
[2021-07-04 23:49:53,920] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'docker_eod_us_equities', 'docker_command_sleep', '2021-07-04T23:19:48.431208+00:00', '--job-id', '52', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/eod_us_equities.py', '--cfg-path', '/tmp/tmp5iz14yg9', '--error-file', '/tmp/tmpleaymjfa']
[2021-07-04 23:49:53,920] {standard_task_runner.py:77} INFO - Job 52: Subtask docker_command_sleep
[2021-07-04 23:49:54,016] {logging_mixin.py:104} INFO - Running <TaskInstance: docker_eod_us_equities.docker_command_sleep 2021-07-04T23:19:48.431208+00:00 [running]> on host 0dff7922cb76
[2021-07-04 23:49:54,172] {taskinstance.py:1300} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=docker_eod_us_equities
AIRFLOW_CTX_TASK_ID=docker_command_sleep
AIRFLOW_CTX_EXECUTION_DATE=2021-07-04T23:19:48.431208+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-04T23:19:48.431208+00:00
[2021-07-04 23:49:54,205] {docker.py:231} INFO - Starting docker container from image alpine
[2021-07-04 23:49:54,216] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.30/containers/create?name=docker_command_sleep

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
    return self._run_image()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
    self.container = self.cli.create_container(
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
    return self.create_container_from_config(config, name)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
    return self._result(res, True)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.30/containers/create?name=docker_command_sleep: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmpw5gvv6dj")
[2021-07-04 23:49:54,222] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=docker_eod_us_equities, task_id=docker_command_sleep, execution_date=20210704T231948, start_date=20210704T234953, end_date=20210704T234954
[2021-07-04 23:49:54,297] {local_task_job.py:151} INFO - Task exited with return code 1

I have even tried with Docker API v1.41 (latest) and same issue. I have bound the /var/run/docker.sock as a bind mount into the container.

Docker Compose:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME           - Docker image name used to run Airflow.
#                                Default: apache/airflow:master-python3.8
# AIRFLOW_UID                  - User ID in Airflow containers
#                                Default: 50000
# AIRFLOW_GID                  - Group ID in Airflow containers
#                                Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME   - Username for the administrator account (if requested).
#                                Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD   - Password for the administrator account (if requested).
#                                Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
#                                Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
  user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
  depends_on:
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'airflow']
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 80:8080
    healthcheck:
      test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test:
        [
          'CMD-SHELL',
          'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
        ]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - 'CMD-SHELL'
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

volumes:
  postgres-db-volume:

DAG:

from datetime import datetime, timedelta

import pendulum
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.dummy_operator import DummyOperator

AMERICA_NEW_YORK_TIMEZONE = pendulum.timezone('US/Eastern')

default_args = {
    'owner': 'airflow',
    'description': 'Docker testing',
    'depend_on_past': False,
    'start_date': datetime(2021, 5, 1, tzinfo=AMERICA_NEW_YORK_TIMEZONE),
    'retries': 1,
    'retry_delay': timedelta(minutes=30),
}

with DAG(
    'docker_test',
    default_args=default_args,
    schedule_interval="15 20 * * *",
    catchup=False,
) as dag:
    start_dag = DummyOperator(task_id='start_dag')

    end_dag = DummyOperator(task_id='end_dag')

    t1 = BashOperator(task_id='print_current_date', bash_command='date')

    t2 = DockerOperator(
        task_id='docker_command_sleep',
        image='alpine',
        container_name='docker_command_sleep',
        api_version='1.30',
        auto_remove=True,
        command="/bin/sleep 3",
        docker_url="unix://var/run/docker.sock",
        network_mode="bridge",
        do_xcom_push=True,
    )

    start_dag >> t1

    t1 >> t2

    t2 >> end_dag

Anything else we need to know: Problem happens any time DockerOperator is being used. Not entirely sure why this happening given that the docker sock is fully permissive (has 777) and is bind mounted into the container. When I test via docker-py client in Python shell under airflow user inside the container, it works perfectly fine to run all docker-py operations like listing running containers and such confirming the mounted docker UNIX socket is available and working. However, even with the docker_url pointing to the docker socket in the above DAG, I am getting this error thrown in above trace.

For whatever strange reason the logs say it's trying to connect over http+docker://localhost/v1.30/containers/create instead of the UNIX docker socket that's bind mounted and explicitly specified via docker_url.

Thanks for opening your first issue here! Be sure to follow the issue template!

Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.

/var/run/docker.sock' is owned by the user that doesn't exist in the container, so we don't have access to it.

As a workaround, you can use a sidecar container with tecnativa/docker-socket-proxy:0.1.1. For details, see: #8605 (comment)

/var/run/docker.sock' is owned by the user that doesn't exist in the container, so we don't have access to it.

I think that's not it. The access right are good and confirmed they work with docker command (see the description). The real reason is described in #16808 - as I suspected, DockerOperator creates a folder in "/tmp" and tries to bind-mount it to the docker container it runs - which does not work because the "/tmp" file is created inside airflow container, not in the host. This also prevents to use DockerOperator from running with remote engine (because the host folder cannot be bind-mounted there).

The temporary solution for docker-compose I proposed (mounting /tmp directory) should work in this case, but we have to fix it differently - DockerOperator should create a separate volume in docker and copy all the files there and mount this volume in the Container it runs, rather than mount the "/tmp" host folder. This should solve both problem - running airflow as docker container in local docker-compose and running airflow with remote docker engine.

@potiuk

Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine.

@potiuk This ended up working with mounting /tmp into the container alongside the /var/run/docker.sock.

Docker Compose:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME           - Docker image name used to run Airflow.
#                                Default: apache/airflow:master-python3.8
# AIRFLOW_UID                  - User ID in Airflow containers
#                                Default: 50000
# AIRFLOW_GID                  - Group ID in Airflow containers
#                                Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME   - Username for the administrator account (if requested).
#                                Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD   - Password for the administrator account (if requested).
#                                Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
#                                Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
    - '/tmp:/tmp'
  user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
  depends_on:
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'airflow']
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 80:8080
    healthcheck:
      test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test:
        [
          'CMD-SHELL',
          'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
        ]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - 'CMD-SHELL'
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

volumes:
  postgres-db-volume:

Yeah. Thought so. We will likely have to fix it.

@potiuk

@potiuk Great, thank you for the quick response and pointing me to the temporary workaround.

I looked at this and turned the temp file mounting into optional feature that you might disable: see #16932

Seems it can also be workarounded by downgrading to 1.0.1 docker provider version.

Hi, I got the same issue when updating from 2.1.0 to 2.1.1.

Adds option to disable mounting temporary folder in DockerOperator (#…

sudo chmod 666 /var/run/docker.sock

/var/run/docker.sock' is owned by the user that doesn't exist in the container, so we don't have access to it.

I think that's not it. The access right are good and confirmed they work with docker command (see the description). The real reason is described in #16808 - as I suspected, DockerOperator creates a folder in "/tmp" and tries to bind-mount it to the docker container it runs - which does not work because the "/tmp" file is created inside airflow container, not in the host. This also prevents to use DockerOperator from running with remote engine (because the host folder cannot be bind-mounted there).

The temporary solution for docker-compose I proposed (mounting /tmp directory) should work in this case, but we have to fix it differently - DockerOperator should create a separate volume in docker and copy all the files there and mount this volume in the Container it runs, rather than mount the "/tmp" host folder. This should solve both problem - running airflow as docker container in local docker-compose and running airflow with remote docker engine.

Thank you. It's work!

Is this solution for running jobs using docker operator available in new releases?

Navigation Menu

Repository navigation

DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock` #16803

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DockerOperator not working from containerized Airflow not recognizing /var/run/docker.sock #16803

Description

Activity

boring-cyborg commented on Jul 5, 2021

potiuk commented on Jul 5, 2021

mik-laj commented on Jul 5, 2021

potiuk commented on Jul 5, 2021

ExecutionSpeed commented on Jul 5, 2021

potiuk commented on Jul 5, 2021

ExecutionSpeed commented on Jul 5, 2021

potiuk commented on Jul 11, 2021

potiuk commented on Jul 12, 2021

ledo01 commented on Jul 13, 2021

eracle commented on Sep 17, 2023

vohoang2002 commented on Jul 5, 2024

VctrEman commented on Sep 11, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

DockerOperator not working from containerized Airflow not recognizing `/var/run/docker.sock` #16803