Asset creation from an ObjectStoragePath doesn't work if a Connection is in use

[baylisscg]

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When creating an Asset using an ObjectStoragePath as the uri parameter if a connection is in use then the _sanitize_uri method used to validate the URI fails with:

"An Asset URI should not contain auth info (e.g. username or password). It has been automatically dropped."

This seems to be due to using a connection_id as here in _sanitize_uri any Asset uri with a userinfo element get's flagged.

What you think should happen instead?

At minimum the message should be clear that it has only removed the userinfo from the Asset uri and it has still been created. As it stands the warning is ambiguous as to what has been "dropped" with the Asset itself being the most likely.

Mangling the user's input, in some cases silently, is just a bad idea. If it's accepted with a warning the implication is it was stored as-is. If userinfo just won't be handled it should be rejected as an error.

Assets should really just accept any valid URI and definitely those generated by other parts of Airflow. As it stands I can't store a URI with an Asset and recover it as s3://conn_1@bucket/data s3://conn_2@bucket/data aren't necessarily the same.

How to reproduce

def test_objectstoragepath_asset():
    path = ObjectStoragePath("s3://example/", conn_id="test")
    asset = Asset(uri=path)
    assert asset.uri == path.as_uri()

's3://example/' != 's3://test@example/'

Expected :'s3://test@example/'
Actual   :'s3://example/'

Operating System

N/A

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

On an OpenStack internal cloud using third-party S3 store.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[astro-anand]

I'm willing to take this on. The sanitize function is implemented with good intentions. I think it is still reasonable to scrub database creds, sftp creds, etc. I'd propose that a fix be implemented specifically for ObjectStoragePath objects. Does that sound ok @baylisscg?

[baylisscg]

@astro-anand I suppose. For schemes that support username:password format userinfo remove remove the password component iff it exists and is not the empty string. So something like:

if scheme.lower() in PASSWORD_AUTH_SCHEMES and ":" in userinfo:
  (username,password) = userinfo.split(":")
  if password and len(password) > 0:
     # Warning / Error / Notice
     userinfo = ""

And add a nice warning notice on the Assets docs that it'll do this and only for passwords in the userinfo segment. Catching pre-signed URLs or other forms of sharing links would be an absolute nightmare.

You would be second guessing the user. In academia this one ftp/http/sftp server with a shared password is terrifyingly common and unavoidable. So a "Hey, if you need to do this define a Connection and ObjectStoragePath"

If Asset is intended as a form of syntactic sugar for defining an event channel for triggering DAGs then sanitising URIs makes sense as uri would be loosely equivalent to task_id and name would be task_display_name

[jgoedeke]

In my use case, I need to work with Azure Data Lake Gen2 URIs of the form:

abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>

Here, the portion before the @ is <file_system>, not a username or credentials, and there is no colon present to indicate a password.

When parsing the netloc, only treat the part before @ as userinfo (and issue a warning or remove it) if it actually matches the username:password pattern (i.e., contains a colon). For example:

parts = parsed.netloc.split("@", 1)
if len(parts) == 2:
    before_at, after_at = parts
    # Only treat as credentials if there is a colon
    if ":" in before_at:
        warnings.warn(
            "An Asset URI should not contain auth info (e.g. username or password). It has been automatically dropped.",
            UserWarning,
            stacklevel=3,
        )
        normalized_netloc = after_at
    else:
        normalized_netloc = parsed.netloc
else:
    normalized_netloc = parsed.netloc

This change would allow legitimate ABFSS URIs to be accepted as-is, while still protecting against accidental inclusion of credentials for schemes that expect them.

For reference, here’s an example of how this logic works for ABFSS:

abfss://filesystem@account.dfs.core.windows.net/path → no colon, nothing stripped, should work fine.