Repository navigation
Git Sync containers are missing when Dag Processor is enabled #27080
Description
Official Helm Chart version
1.7.0 (latest released)
Apache Airflow version
2.3.4
Kubernetes Version
1.22.6
Helm Chart configuration
...
# Airflow Dag Processor Config
dagProcessor:
enabled: true
# Number of airflow dag processors in the deployment
replicas: 1
...
# Git sync
dags:
persistence:
# Enable persistent volume for storing dags
enabled: true
# Volume size for dags
size: 1Gi
# If using a custom storageClass, pass name here
storageClassName:
# access mode of the persistent volume
accessMode: ReadWriteOnce
## the name of an existing PVC to use
existingClaim: airflow-dags
## optional subpath for dag volume mount
subPath: ~
gitSync:
enabled: true
...
Docker Image customisations
No response
What happened
When Dag Processor is enabled, the Git Sync containers (git-sync-init and git-sync) are missing in airflow-scheduler pod.
They are neither part of the airflow-dag-processor pod.
What you think should happen instead
Git Sync containers (git-sync-init and git-sync) should be part of one of airflow-scheduler or airflow-dag-processor pods.
Below attaching parts of the helm chart log showing deployment of both airflow-scheduler and airflow-dag-processor pods.
################################
## Airflow Dag Processor Deployment
#################################
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-dag-processor
labels:
tier: airflow
component: dag-processor
release: airflow
chart: "airflow-1.7.0"
heritage: Helm
spec:
replicas: 1
selector:
matchLabels:
tier: airflow
component: dag-processor
release: airflow
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 50%
template:
metadata:
labels:
tier: airflow
component: dag-processor
release: airflow
annotations:
checksum/metadata-secret: 765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/airflow-config: a78767ecb4f7423a34e69c0add978a0ff15452ff0a54a6cb08fc93f06aaf5e7e
checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
nodeSelector:
nodepool-type: airflow
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
component: dag-processor
topologyKey: kubernetes.io/hostname
weight: 100
tolerations:
[]
topologySpreadConstraints:
[]
terminationGracePeriodSeconds: 60
restartPolicy: Always
serviceAccountName: airflow-dag-processor
securityContext:
runAsUser: 50000
fsGroup: 0
imagePullSecrets:
- name: ds-registry
initContainers:
- name: wait-for-airflow-migrations
resources:
limits:
cpu: 1000m
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
image: seebodsregistry.azurecr.io/airflow:2.3.4-extended
imagePullPolicy: IfNotPresent
volumeMounts:
- name: config
mountPath: "/opt/airflow/airflow.cfg"
subPath: airflow.cfg
readOnly: true
args:
- airflow
- db
- check-migrations
- --migration-wait-timeout=60
envFrom:
[]
env:
# Dynamically created environment variables
# Dynamically created secret envs
# Extra env
- name: AIRFLOW__CORE__DAGS_FOLDER
value: '/opt/airflow/all-dags'
- name: AIRFLOW__CORE__PLUGINS_FOLDER
value: '/opt/airflow/plugins/repo/deployment/plugins'
- name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
value: '600'
- name: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT
value: '3600'
- name: AIRFLOW__CORE__PARALLELISM
value: '128'
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: 'True'
- name: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__PARSING_PROCESSES
value: '15'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP
value: '100'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE
value: '100'
- name: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD
value: '1200'
- name: AIRFLOW__WEBSERVER__INSTANCE_NAME
value: 'Staging'
- name: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE
value: 'True'
# Hard Coded Airflow Envs
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: airflow-fernet-key
key: fernet-key
# For Airflow <2.3, backward compatibility; moved to [database] in 2.3
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: airflow-webserver-key
key: webserver-secret-key
containers:
- name: dag-processor
image: seebodsregistry.azurecr.io/airflow:2.3.4-extended
imagePullPolicy: IfNotPresent
args:
- bash
- -c
- exec airflow dag-processor
resources:
limits:
cpu: 1000m
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: logs
mountPath: "/opt/airflow/logs"
- name: config
mountPath: "/opt/airflow/airflow.cfg"
subPath: airflow.cfg
readOnly: true
- name: config
mountPath: "/opt/airflow/config/airflow_local_settings.py"
subPath: airflow_local_settings.py
readOnly: true
- name: dags
mountPath: /opt/airflow/dags
readOnly: True
envFrom:
[]
env:
# Dynamically created environment variables
# Dynamically created secret envs
# Extra env
- name: AIRFLOW__CORE__DAGS_FOLDER
value: '/opt/airflow/all-dags'
- name: AIRFLOW__CORE__PLUGINS_FOLDER
value: '/opt/airflow/plugins/repo/deployment/plugins'
- name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
value: '600'
- name: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT
value: '3600'
- name: AIRFLOW__CORE__PARALLELISM
value: '128'
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: 'True'
- name: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__PARSING_PROCESSES
value: '15'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP
value: '100'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE
value: '100'
- name: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD
value: '1200'
- name: AIRFLOW__WEBSERVER__INSTANCE_NAME
value: 'Staging'
- name: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE
value: 'True'
# Hard Coded Airflow Envs
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: airflow-fernet-key
key: fernet-key
# For Airflow <2.3, backward compatibility; moved to [database] in 2.3
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: airflow-webserver-key
key: webserver-secret-key
livenessProbe:
initialDelaySeconds: 10
timeoutSeconds: 20
failureThreshold: 5
periodSeconds: 120
exec:
command:
- sh
- -c
- |
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --hostname $(hostname)
volumes:
- name: config
configMap:
name: airflow-airflow-config
- name: dags
persistentVolumeClaim:
claimName: airflow-dags
- name: logs
persistentVolumeClaim:
claimName: airflow-logs
################################
## Airflow Scheduler Deployment/StatefulSet
#################################
# Are we using a local executor?
# Is persistence enabled on the _workers_?
# This is important because in $local mode, the scheduler assumes the role of the worker
# If we're using a StatefulSet
# We can skip DAGs mounts on scheduler if dagProcessor is enabled, except with $local mode
# If we're using elasticsearch logging
kind: Deployment
apiVersion: apps/v1
metadata:
name: airflow-scheduler
labels:
tier: airflow
component: scheduler
release: airflow
chart: "airflow-1.7.0"
heritage: Helm
executor: KubernetesExecutor
spec:
replicas: 1
selector:
matchLabels:
tier: airflow
component: scheduler
release: airflow
template:
metadata:
labels:
tier: airflow
component: scheduler
release: airflow
annotations:
checksum/metadata-secret: 765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
checksum/result-backend-secret: 74e3e99feee51248d44224665d60fab543dd6b25ba95f04e6fcb0e5758342056
checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/airflow-config: a78767ecb4f7423a34e69c0add978a0ff15452ff0a54a6cb08fc93f06aaf5e7e
checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
nodeSelector:
nodepool-type: airflow
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
component: scheduler
topologyKey: kubernetes.io/hostname
weight: 100
tolerations:
[]
topologySpreadConstraints:
[]
restartPolicy: Always
terminationGracePeriodSeconds: 10
serviceAccountName: airflow-scheduler
securityContext:
runAsUser: 50000
fsGroup: 0
imagePullSecrets:
- name: ds-registry
initContainers:
- name: wait-for-airflow-migrations
resources:
limits:
cpu: 8000m
memory: 8Gi
requests:
cpu: 500m
memory: 1Gi
image: seebodsregistry.azurecr.io/airflow:2.3.4-extended
imagePullPolicy: IfNotPresent
volumeMounts:
- name: config
mountPath: "/opt/airflow/airflow.cfg"
subPath: airflow.cfg
readOnly: true
args:
- airflow
- db
- check-migrations
- --migration-wait-timeout=60
envFrom:
[]
env:
# Dynamically created environment variables
# Dynamically created secret envs
# Extra env
- name: AIRFLOW__CORE__DAGS_FOLDER
value: '/opt/airflow/all-dags'
- name: AIRFLOW__CORE__PLUGINS_FOLDER
value: '/opt/airflow/plugins/repo/deployment/plugins'
- name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
value: '600'
- name: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT
value: '3600'
- name: AIRFLOW__CORE__PARALLELISM
value: '128'
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: 'True'
- name: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__PARSING_PROCESSES
value: '15'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP
value: '100'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE
value: '100'
- name: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD
value: '1200'
- name: AIRFLOW__WEBSERVER__INSTANCE_NAME
value: 'Staging'
- name: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE
value: 'True'
# Hard Coded Airflow Envs
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: airflow-fernet-key
key: fernet-key
# For Airflow <2.3, backward compatibility; moved to [database] in 2.3
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: airflow-webserver-key
key: webserver-secret-key
containers:
# Always run the main scheduler container.
- name: scheduler
image: seebodsregistry.azurecr.io/airflow:2.3.4-extended
imagePullPolicy: IfNotPresent
args:
- bash
- -c
- exec airflow scheduler
envFrom:
[]
env:
# Dynamically created environment variables
# Dynamically created secret envs
# Extra env
- name: AIRFLOW__CORE__DAGS_FOLDER
value: '/opt/airflow/all-dags'
- name: AIRFLOW__CORE__PLUGINS_FOLDER
value: '/opt/airflow/plugins/repo/deployment/plugins'
- name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
value: '600'
- name: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT
value: '3600'
- name: AIRFLOW__CORE__PARALLELISM
value: '128'
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: 'True'
- name: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL
value: '120'
- name: AIRFLOW__SCHEDULER__PARSING_PROCESSES
value: '15'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP
value: '100'
- name: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE
value: '100'
- name: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD
value: '1200'
- name: AIRFLOW__WEBSERVER__INSTANCE_NAME
value: 'Staging'
- name: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE
value: 'True'
# Hard Coded Airflow Envs
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: airflow-fernet-key
key: fernet-key
# For Airflow <2.3, backward compatibility; moved to [database] in 2.3
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
name: airflow-postgresql
key: connection
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: airflow-webserver-key
key: webserver-secret-key
livenessProbe:
initialDelaySeconds: 10
timeoutSeconds: 20
failureThreshold: 5
periodSeconds: 60
exec:
command:
- sh
- -c
- |
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --job-type SchedulerJob --hostname $(hostname)
resources:
limits:
cpu: 8000m
memory: 8Gi
requests:
cpu: 500m
memory: 1Gi
volumeMounts:
- name: config
mountPath: /opt/airflow/pod_templates/pod_template_file.yaml
subPath: pod_template_file.yaml
readOnly: true
- name: logs
mountPath: "/opt/airflow/logs"
- name: config
mountPath: "/opt/airflow/airflow.cfg"
subPath: airflow.cfg
readOnly: true
- name: config
mountPath: "/opt/airflow/config/airflow_local_settings.py"
subPath: airflow_local_settings.py
readOnly: true
- name: scheduler-log-groomer
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
image: seebodsregistry.azurecr.io/airflow:2.3.4-extended
imagePullPolicy: IfNotPresent
args:
- bash
- /clean-logs
env:
- name: AIRFLOW__LOG_RETENTION_DAYS
value: "15"
volumeMounts:
- name: logs
mountPath: "/opt/airflow/logs"
volumes:
- name: config
configMap:
name: airflow-airflow-config
- name: logs
persistentVolumeClaim:
claimName: airflow-logs
How to reproduce
Both Git Sync and Dag Processor should be enabled.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Jan 10, 2023
Related #27545 #27476 #27124
github-actions commentedon Feb 9, 2024
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
github-actions commentedon Mar 19, 2024
This issue has been closed because it has not received response from the issue author.