DAGs with ObjectStoragePath failing to locate credentials with Airflow 3.2 rc

[pankajkoti]

Apache Airflow version

3.2.0rc1

What happened and how to reproduce it?

I am testing 3.2.0rc1 and seeing issues in DAGs using remote ObjectStoragePath  (tried S3 and GCS paths) . The same DAG works fine on 3.1.8. For both these airflow versions, I am using the latest amazon and google providers, so likely not a provider issue, but maybe something related to airflow / sdk not being able to fetch credentials? I am setting up those connections using environment variables for my local environment.

What you think should happen instead?

DAGs should run fine when using remote ObjectStoragePath

Operating System

No response

Versions of Apache Airflow Providers

apache-airflow-providers-amazon | 9.24.0

apache-airflow-providers-google | 21.0.0

Deployment

Astronomer

Deployment details

locally using astro-cli

Anything else?

Attaching failure task logs:

GCS ObjectStoragePath task:
logs_test_object_storage_path_manual__2026-04-02T14_43_24.115545+00_00_check_gcs_-1_1.txt

S3 ObjectStoragePath task:
logs_test_object_storage_path_manual__2026-04-02T14_43_24.115545+00_00_check_s3_-1_1.txt

DAG:
test_object_storage_path.py

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pankajkoti]

cc: @vatsrahul1001

[jason810496]

Just a quick glance on GCS one, it seems like 401 credential error

[2026-04-02T14:43:25.894954Z] WARNING - GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT is set, but failed to import experimental features: No module named 'google.cloud.storage.asyncio'
[2026-04-02T14:43:54.071967Z] ERROR - _request non-retriable exception: Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., 401HttpError: Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., 401
    File /usr/local/lib/python3.13/site-packages/gcsfs/retry.py, line 143 in retry_request
    File /usr/local/lib/python3.13/site-packages/gcsfs/core.py, line 498 in _request
    File /usr/local/lib/python3.13/site-packages/gcsfs/retry.py, line 130 in validate_response

[2026-04-02T14:43:54.072373Z] ERROR - Task failed with exceptionHttpError: Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., 401

[pankajkoti]

@jason810496 yes, the credentials are not fetched/forwarded correctly I guess during task execution. Keeping everything same, it works fine on Airflow 3.1.8

[jason810496]

Got it, thanks for sharing the context.