new feature: support AWS Container credential provider

[Samrose-Ahmed]

Feature Description

via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable in ECS/container settings: see https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html

Problem and Solution

load credentials in ecs environment

Additional Context

No response

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

Hi, @Samrose-Ahmed, does this issue related to #4964?

[Samrose-Ahmed]

Yes it's the same.

[Xuanwo]

Thank you! I will redirect the discussion here.

[Xuanwo]

Hi, @Samrose-Ahmed, I expect to add this feature in our next release.

[ChillFish8]

General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.

[Xuanwo]

General question regarding this, would it be a good idea to wrap the AWS SDK with the credentials provider to do this job? This has been my current workaround for this issue and might be a better long-term solution for managing all the various ways you can authenticate with AWS.

Thanks for the suggestion. We previously had a discussion here: #139.

The various methods for authentication are limited and rarely change, so we can ultimately support them all. I believe it isn't worth introducing the AWS SDK for authentication.

[matthias-Q]

I stumbled over this issue today. Here is a workaround with a start script that sets the ENV vars when running inside ECS

#!/bin/bash

CREDS_URI=${AWS_CONTAINER_CREDENTIALS_FULL_URI:-"http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}"}
CREDS_RESPONSE=$(curl -s $CREDS_URI)

export AWS_ACCESS_KEY_ID=$(echo $CREDS_RESPONSE | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS_RESPONSE | jq -r '.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDS_RESPONSE | jq -r '.Token')

exec "$@"

I hope OpenDAL will support the proper Credentials Chain in the future.

[Xuanwo]

Hi, thank you @matthias-Q for this workaround. I'm currently very active in the project https://github.com/Xuanwo/reqsign, which is the crate that opendal relies on for this feature. I plan to integrate these changes in the next major version of opendal.

[michaelrommel]

Hello @Xuanwo ,

Is it possible to update the status here? I try to access S3 buckets from a Fargate Container and cannot get the credentials automatically...

Thanks,

Michael.

[michaelrommel]

Here is a temporary solution:

use aws_config::meta::region::RegionProviderChain;
use aws_credential_types::provider::ProvideCredentials;
use aws_types::SdkConfig;

struct AwsCredentialLoad;

#[async_trait::async_trait]
impl reqsign::AwsCredentialLoad for AwsCredentialLoad {
    async fn load_credential(&self, _client: Client) -> anyhow::Result<Option<AwsCredential>> {
        println!("Loading AWS credentials");
        let region_provider = RegionProviderChain::default_provider().or_else("us-east-1");

        let config: SdkConfig = aws_config::defaults(BehaviorVersion::v2026_01_12())
            .region(region_provider)
            .load()
            .await;

        if let Some(credentials_provider) = config.credentials_provider() {
            let credentials = credentials_provider.provide_credentials().await?;
            let duration = credentials
                .expiry()
                .unwrap()
                .duration_since(UNIX_EPOCH)
                .expect("SystemTime is before UNIX_EPOCH");
            let expiry = Utc
                .timestamp_opt(duration.as_secs() as i64, duration.subsec_nanos())
                .single();
            println!("AWS Token expires at: {:?}", expiry);
            return Ok(Some(AwsCredential {
                access_key_id: credentials.access_key_id().to_string(),
                secret_access_key: credentials.secret_access_key().to_string(),
                session_token: credentials.session_token().map(|s| s.to_string()),
                expires_in: expiry,
            }));
        }
        Ok(None)
    }
}

   let builder = S3::default()
        .customized_credential_load(Box::new(AwsCredentialLoad))
        .endpoint("https://s3.amazonaws.com")
        .region(&region) // String with the region name
        .bucket(&bucket) // String with the bucket name
        .root("/");