Skip to content

chore(pipeline): upgrade airflow to v3#466

Closed
vmttn wants to merge 21 commits intomainfrom
vmttn/chore/upgrade-airflow
Closed

chore(pipeline): upgrade airflow to v3#466
vmttn wants to merge 21 commits intomainfrom
vmttn/chore/upgrade-airflow

Conversation

@vmttn
Copy link
Copy Markdown
Contributor

@vmttn vmttn commented Jul 15, 2025

Comment thread pipeline/Dockerfile Outdated
# This image is the runtime
########
FROM apache/airflow:2.11.0-python3.12 AS runtime-image
FROM apache/airflow:3.0.2-python3.12 AS runtime-image
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docker-compose.yml

environment:
&airflow-common-environment
AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from flask_appbuilder.security.manager import AUTH_OAUTH

AUTH_TYPE = AUTH_OAUTH
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Viewer"
CSRF_ENABLED = True

OAUTH_PROVIDERS = [
    {
        'name': 'google',
        'icon': 'fa-google',
        'token_key': 'access_token',
        'remote_app': {
            'client_id': 'your-google-client-id.apps.googleusercontent.com',
            'client_secret': 'your-google-client-secret',
            'api_base_url': 'https://www.googleapis.com/oauth2/v2/',
            'client_kwargs': {'scope': 'email profile'},
            'request_token_url': None,
            'access_token_url': 'https://accounts.google.com/o/oauth2/token',
            'authorize_url': 'https://accounts.google.com/o/oauth2/auth'
        },
        'whitelist': ['@yourdomain.com']  # Optional: restrict to specific domains
    }
]

mais sûrement YAGNI 😄

@vperron
Copy link
Copy Markdown
Contributor

vperron commented Jul 15, 2025

D'après ChatGPT,

image

... ça va peut etre permettre d'affiner ma manip pour eviter les grosses AirflowException partout.

@vperron
Copy link
Copy Markdown
Contributor

vperron commented Jul 15, 2025

Pour info on annonce la fin de la dépendance à FAB pour très bientot (la MAJ à Py3.13 en dépend d'ailleurs):
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79%3A+Remove+Flask+AppBuilder+as+Core+dependency

Ah et puis oups pour Sentry 🤷
apache/airflow#52368

@vmttn vmttn self-assigned this Jul 21, 2025
@vmttn vmttn changed the title chore(pipeline): upgrade airflow chore(pipeline): upgrade airflow to v3 Jul 23, 2025
@vmttn vmttn force-pushed the vmttn/chore/upgrade-airflow branch from 4f0219e to 3e36415 Compare August 15, 2025 13:05
vmttn added 21 commits August 18, 2025 13:59
This way downstream models are not explicitely dependent.
And now the tables can be simply updated, without cascades.
Another possible way would have been to load with a tmp table,
truncate and copy.
- add a factory for common dag args
- refactor layout with org namespace
- replace `concurrency` with `max_active_tasks`
- set `max_actives_runs`
- regroup all dags and tasks common utils
- use `chain` operator to prevent typing warnings
@vperron vperron force-pushed the vmttn/chore/upgrade-airflow branch from 3e36415 to 1d34dd1 Compare August 18, 2025 12:02
@vperron
Copy link
Copy Markdown
Contributor

vperron commented Oct 13, 2025

Je me permets de refermer cette PR, on ouvrira si besoin.

@vperron vperron closed this Oct 13, 2025
@vperron vperron deleted the vmttn/chore/upgrade-airflow branch February 23, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants