Skip to content

feat(docker): add minute-level log retention to clean-logs script #61855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jscheffl merged 6 commits into from
Mar 11, 2026
Merged

feat(docker): add minute-level log retention to clean-logs script #61855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0418328
feat(docker): add minute-level log retention support to clean-logs.sh
pugarte7 Feb 13, 2026
e8206b6
feat(docker): import coauthored changes
pugarte7 Mar 6, 2026
c93b924
update inlined scripts in Dockerfile
pugarte7 Mar 6, 2026
61d4562
comments fixed
pugarte7 Mar 11, 2026
d264e95
Merge branch 'main' into feat/docker-minute-log-retention
pugarte7 Mar 11, 2026
7d2945c
Merge branch 'main' into feat/docker-minute-log-retention
pugarte7 Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1670,7 +1670,8 @@ COPY <<"EOF" /clean-logs.sh
set -euo pipefail

readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
readonly RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
readonly RETENTION_DAYS="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
readonly RETENTION_MINUTES="${AIRFLOW__LOG_RETENTION_MINUTES:-0}"
readonly FREQUENCY="${AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES:-15}"
readonly MAX_PERCENT="${AIRFLOW__LOG_MAX_SIZE_PERCENT:-0}"

Expand All @@ -1695,10 +1696,12 @@ fi
retention_days="${RETENTION}"

while true; do
echo "Trimming airflow logs to ${retention_days} days."
total_retention_minutes=$(( (RETENTION_DAYS * 1440) + RETENTION_MINUTES ))
echo "Trimming airflow logs older than ${total_retention_minutes} minutes."

find "${DIRECTORY}"/logs \
-type d -name 'lost+found' -prune -o \
-type f -mtime +"${retention_days}" -name '*.log' -print0 | \
-type f -mmin +"${total_retention_minutes}" -name '*.log' -print0 | \
xargs -0 rm -f || true

if [[ "$MAX_SIZE_BYTES" -gt 0 && "$retention_days" -ge 0 ]]; then
Expand Down
4 changes: 4 additions & 0 deletions chart/templates/dag-processor/dag-processor-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,10 @@ spec:
- name: AIRFLOW__LOG_RETENTION_DAYS
value: "{{ .Values.dagProcessor.logGroomerSidecar.retentionDays }}"
{{- end }}
{{- if .Values.dagProcessor.logGroomerSidecar.retentionMinutes }}
- name: AIRFLOW__LOG_RETENTION_MINUTES
value: "{{ .Values.dagProcessor.logGroomerSidecar.retentionMinutes }}"
{{- end }}
{{- if .Values.dagProcessor.logGroomerSidecar.frequencyMinutes }}
- name: AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES
value: "{{ .Values.dagProcessor.logGroomerSidecar.frequencyMinutes }}"
Expand Down
4 changes: 4 additions & 0 deletions chart/templates/scheduler/scheduler-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,10 @@ spec:
- name: AIRFLOW__LOG_RETENTION_DAYS
value: "{{ .Values.scheduler.logGroomerSidecar.retentionDays }}"
{{- end }}
{{- if .Values.scheduler.logGroomerSidecar.retentionMinutes }}
- name: AIRFLOW__LOG_RETENTION_MINUTES
value: "{{ .Values.scheduler.logGroomerSidecar.retentionMinutes }}"
{{- end }}
{{- if .Values.scheduler.logGroomerSidecar.frequencyMinutes }}
- name: AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES
value: "{{ .Values.scheduler.logGroomerSidecar.frequencyMinutes }}"
Expand Down
4 changes: 4 additions & 0 deletions chart/templates/triggerer/triggerer-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,10 @@ spec:
- name: AIRFLOW__LOG_RETENTION_DAYS
value: "{{ .Values.triggerer.logGroomerSidecar.retentionDays }}"
{{- end }}
{{- if .Values.triggerer.logGroomerSidecar.retentionMinutes }}
- name: AIRFLOW__LOG_RETENTION_MINUTES
value: "{{ .Values.triggerer.logGroomerSidecar.retentionMinutes }}"
{{- end }}
{{- if .Values.triggerer.logGroomerSidecar.frequencyMinutes }}
- name: AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES
value: "{{ .Values.triggerer.logGroomerSidecar.frequencyMinutes }}"
Expand Down
4 changes: 4 additions & 0 deletions chart/templates/workers/worker-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,10 @@ spec:
- name: AIRFLOW__LOG_RETENTION_DAYS
value: "{{ .Values.workers.logGroomerSidecar.retentionDays }}"
{{- end }}
{{- if .Values.workers.logGroomerSidecar.retentionMinutes }}
- name: AIRFLOW__LOG_RETENTION_MINUTES
value: "{{ .Values.workers.logGroomerSidecar.retentionMinutes }}"
{{- end }}
{{- if .Values.workers.logGroomerSidecar.frequencyMinutes }}
- name: AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES
value: "{{ .Values.workers.logGroomerSidecar.frequencyMinutes }}"
Expand Down
7 changes: 6 additions & 1 deletion chart/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14139,10 +14139,15 @@
]
},
"retentionDays": {
"description": "Number of days to retain the logs when running the Airflow log groomer sidecar.",
"description": "Number of days to retain the logs when running the Airflow log groomer sidecar. Total retention time is retentionDays + retentionMinutes.",
"type": "integer",
"default": 15
},
"retentionMinutes": {
"description": "Number of minutes to retain the logs when running the Airflow log groomer sidecar. Total retention time is retentionDays + retentionMinutes.",
"type": "integer",
"default": 0
},
"frequencyMinutes": {
"description": "Number of minutes between attempts to groom the Airflow logs in log groomer sidecar.",
"type": "integer",
Expand Down
24 changes: 24 additions & 0 deletions chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -993,6 +993,12 @@ workers:
# Number of days to retain logs
retentionDays: 15

# Number of minutes to retain logs.
# This can be used for finer granularity than days.
# Total retention is retentionDays + retentionMinutes.
retentionMinutes: 0


# Frequency to attempt to groom logs (in minutes)
frequencyMinutes: 15

Expand Down Expand Up @@ -1472,6 +1478,12 @@ scheduler:
args: ["bash", "/clean-logs"]
# Number of days to retain logs
retentionDays: 15

# Number of minutes to retain logs.
# This can be used for finer granularity than days.
# Total retention is retentionDays + retentionMinutes.
retentionMinutes: 0

# frequency to attempt to groom logs, in minutes
frequencyMinutes: 15
# Max size of logs in bytes. 0 = disabled
Expand Down Expand Up @@ -2317,6 +2329,12 @@ triggerer:
args: ["bash", "/clean-logs"]
# Number of days to retain logs
retentionDays: 15

# Number of minutes to retain logs.
# This can be used for finer granularity than days.
# Total retention is retentionDays + retentionMinutes.
retentionMinutes: 0

# frequency to attempt to groom logs, in minutes
frequencyMinutes: 15
# Max size of logs in bytes. 0 = disabled
Expand Down Expand Up @@ -2548,6 +2566,12 @@ dagProcessor:
args: ["bash", "/clean-logs"]
# Number of days to retain logs
retentionDays: 15

# Number of minutes to retain logs.
# This can be used for finer granularity than days.
# Total retention is retentionDays + retentionMinutes.
retentionMinutes: 0

# frequency to attempt to groom logs, in minutes
frequencyMinutes: 15
# Max size of logs in bytes. 0 = disabled
Expand Down
9 changes: 6 additions & 3 deletions scripts/docker/clean-logs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
set -euo pipefail

readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
readonly RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
readonly RETENTION_DAYS="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
readonly RETENTION_MINUTES="${AIRFLOW__LOG_RETENTION_MINUTES:-0}"
readonly FREQUENCY="${AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES:-15}"
readonly MAX_PERCENT="${AIRFLOW__LOG_MAX_SIZE_PERCENT:-0}"

Expand All @@ -45,10 +46,12 @@ fi
retention_days="${RETENTION}"
Copy link
Copy Markdown
Member

@potiuk potiuk Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bug here = RETENTION is sttill used here @Wastelander777

Copy link
Copy Markdown
Contributor Author

@pugarte7 pugarte7 Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups, my bad 😣. What should be the approach? How can I fix that? Do I re-open the issue?

Copy link
Copy Markdown
Contributor

@n-badtke-cg n-badtke-cg Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, went over my head too :/

Do I re-open the issue?

no, the PR is already merged, there needs to be a new PR unfortunately


while true; do
echo "Trimming airflow logs to ${retention_days} days."
total_retention_minutes=$(( (RETENTION_DAYS * 1440) + RETENTION_MINUTES ))
echo "Trimming airflow logs older than ${total_retention_minutes} minutes."

find "${DIRECTORY}"/logs \
-type d -name 'lost+found' -prune -o \
-type f -mtime +"${retention_days}" -name '*.log' -print0 | \
-type f -mmin +"${total_retention_minutes}" -name '*.log' -print0 | \
xargs -0 rm -f || true

if [[ "$MAX_SIZE_BYTES" -gt 0 && "$retention_days" -ge 0 ]]; then
Expand Down
Loading