bug: UnexpectedError in Python Binding When Accessing S3 on aarch64

[edison-weng]

Bug Description

I encountered an error when using the Python binding within a Docker container. The error occurs while attempting to interact with an S3 bucket. The error message indicates an issue with sending an HTTP request.

Error Message

opendal.UnexpectedError: Unexpected (temporary) at stat, context: { url: https://s3.my-region.amazonaws.com/mybucket/path/to/object.jpg, called: http_util::Client::send, service: s3, path: path/to/object.jpg } => send http request, source: error sending request for url (https://s3.my-region.amazonaws.com/mybucket/path/to/object.jpg)

Reproducible Repository

https://github.com/tacheng9502/opendal-docker-s3-bug

Environment Details

OpenDAL Version: 0.45.13
Python Version: 3.10.16
Docker Base Image: python:3.10-slim, python:3.10, public.ecr.aws/lambda/python:3.10

Additional Context

The issue happens in both the Python slim/full image and AWS Lambda image. However, the same operation works fine outside the Docker container, so this issue seems to be specific to the Docker environment. I’ve verified that the S3 credentials and permissions are correctly set up.

[Xuanwo]

Hi, given the fact that you are using a slim image, have you checked that if ca certs have been installed?

[edison-weng]

Hi, given the fact that you are using a slim image, have you checked that if ca certs have been installed?

Hi Xuanwo,

Thank you for your prompt response.

I’ve verified that the ca-certificates package is installed in the Python slim image, and I also tested with the full base image. Unfortunately, I’m still encountering the same issue. The same issue also happened in the AWS Lambda image - public.ecr.aws/lambda/python:3.10.

Package: ca-certificates
Status: install ok installed
Priority: standard
Section: misc
Installed-Size: 384
Maintainer: Julien Cristau <jcristau@debian.org>
Architecture: all
Multi-Arch: foreign
Version: 20230311
Depends: openssl (>= 1.1.1), debconf (>= 0.5) | debconf-2.0
Breaks: ca-certificates-java (<< 20121112+nmu1)
Enhances: openssl
Description: Common CA certificates
 Contains the certificate authorities shipped with Mozilla's browser to allow
 SSL-based applications to check for the authenticity of SSL connections.
 .
 Please note that Debian can neither confirm nor deny whether the
 certificate authorities whose certificates are included in this package
 have in any way been audited for trustworthiness or RFC 3647 compliance.
 Full responsibility to assess them belongs to the local system
 administrator.

[Xuanwo]

Thank you for the detailed information. I will try reproducing it and figure it out.

[Xuanwo]

Hi @tacheng9502, I set up the environment as described in https://github.com/tacheng9502/opendal-docker-s3-bug but was unable to reproduce the issue.

:) docker run --env-file .env opendal-docker-bug
Error: NotFound (permanent) at stat, context: { uri: https://s3.ap-northeast-1.amazonaws.com/xxxxx/path/to/object.jpg, response: Parts { status: 404, version: HTTP/1.1, headers: {"x-amz-request-id": "K15FAYP2PAKA8H9Y", "x-amz-id-2": "fJ8JVJASq11pV9Htu+BsZpf4+/LqefYqJclEBlwF9g/3JjhBgQL/raTF+yiLWNY3cBBgPmJPuVQ=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Tue, 31 Dec 2024 07:02:51 GMT", "server": "AmazonS3"} }, service: s3, path: path/to/object.jpg }

Could you share the output of?

curl -v https://s3.region.amazonaws.com/xxxxx/path/to/object.jpg

[kikumoto]

@Xuanwo

I believe this issue occurs in the Python bindings for:

• Linux
• aarch64

It does not occur on x86 - Linux.

import opendal

kwargs = {
    'region': 'us-west-2',
    'access_key_id': 'XXXXXXXXXXXXXXXX',
    'secret_access_key': 'ZZZZZZZZZZZZZZZZZZZZZZZZ',
    'endpoint': 'https://s3.amazonaws.com',
    'bucket': 'yyyyyyyyyyyyyyy',
}

op = opendal.Operator(scheme="s3", **kwargs)
res = op.stat(path='requirements.txt').mode.is_file()
print(res)

When running this on Amazon Linux 2023 t4g.medium (Arm64), the same error occurs.
However, it works correctly on t3.medium (x86).

In both cases, opendal was installed via pip.

On the other hand, if I build the Python bindings manually on Arm64 without using pip, it works correctly.
From this, I suspect there might be an issue in the cross-compilation process for aarch64 Linux in the Python bindings.

I would appreciate it if you could look into this.

[Xuanwo]

Cc @Zheaoli & @messense, do you have any ideas?

[Zheaoli]

I will take handle of this

[Zheaoli]

@kikumoto @tacheng9502 would you mind to install https://opendal-infra-s3.s3.ap-northeast-1.amazonaws.com/opendal-0.45.13-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl and test it?

[kikumoto]

@Zheaoli
Thank you for your quick response.
I tried it, and it worked fine.

on Amazon Linux 2023 t4g.medium

$ python -m venv venv2
$ . venv2/bin/activate
$ pip install -U pip

$ pip list
Package Version
------- -------
pip     24.3.1

$ python -V
Python 3.12.4

$ pip install https://opendal-infra-s3.s3.ap-northeast-1.amazonaws.com/opendal-0.45.13-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Collecting opendal==0.45.13
  Downloading https://opendal-infra-s3.s3.ap-northeast-1.amazonaws.com/opendal-0.45.13-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (16.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 13.1 MB/s eta 0:00:00
Installing collected packages: opendal
Successfully installed opendal-0.45.13

$ pip list
Package Version
------- -------
opendal 0.45.13
pip     24.3.1

$ python demo1.py
True

The content of demo1.py is the sample code I provided above.

Thank you very much.

[Zheaoli]

OK fine, here's the result

https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml#L66

I think we set -D__ARM_ARCH=8 before. I think there will be some problems for some algorithm. Actually it's not recommended by the author briansmith/ring#1728 (comment)

I think we can change the environment to

export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
export AR_aarch64_unknown_linux_gnu=aarch64-linux-gnu-ar

It will work(

[Zheaoli]

OK, here's the debug result. I think maybe it's the final result

First, we need ARM C Language Extension for ring lib. https://github.com/briansmith/ring/blob/main/include/ring-core/arm_arch.h#L78-L84

// We require the ARM assembler provide |__ARM_ARCH| from Arm C Language
// Extensions (ACLE). This is supported in GCC 4.8+ and Clang 3.2+. MSVC does
// not implement ACLE, but we require Clang's assembler on Windows.
#if !defined(__ARM_ARCH)
#error "ARM assembler must define __ARM_ARCH"
#endif

OK, let's take a look at the build container now, we use the GCC in the manylinux-cross image. FYI https://github.com/rust-cross/manylinux-cross/blob/main/manylinux2014/x86_64/Dockerfile#L40 and https://github.com/crosstool-ng/crosstool-ng/archive/02d1503f6769be4ad8058b393d4245febced459f.tar.gz

CT_CONFIG_VERSION="3"
CT_OBSOLETE=y
CT_ARCH_ARM=y
CT_ARCH_64=y
CT_TARGET_VENDOR="ol7u9"
CT_KERNEL_LINUX=y
CT_LINUX_USE_ORACLE=y
CT_LINUX_ORACLE_V_4_14=y
CT_LINUX_ORACLE_VERSION="4.14.35-2025.400.8"
CT_BINUTILS_USE_ORACLE=y
CT_BINUTILS_ORACLE_V_2_27_44=y
CT_BINUTILS_LINKER_LD_GOLD=y
CT_BINUTILS_GOLD_THREADS=y
CT_BINUTILS_LD_WRAPPER=y
CT_BINUTILS_PLUGINS=y
CT_GLIBC_USE_ORACLE=y
CT_GLIBC_ORACLE_V_2_17_317_0_3=y
CT_GCC_USE_ORACLE=y
CT_GCC_ORACLE_V_4_8=y
CT_GCC_ORACLE_VERSION="4.8.5-44.0.5"
CT_CC_LANG_CXX=y

I see someone saying that we can use __ARM_ARCH to activate ACLE since GCC 4.8.5. But it's not true. It would be GCC 5.2.0 actually. FYI gcc-mirror/gcc@b1760f7

#ifndef __ARM_ARCH
# if defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
     || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
     || defined(__ARM_ARCH_7EM__)
#  define __ARM_ARCH 7
# elif defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
        || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
        || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) \
	|| defined(__ARM_ARCH_6M__)
#  define __ARM_ARCH 6
# elif defined(__ARM_ARCH_5__) || defined(__ARM_ARCH_5T__) \
	|| defined(__ARM_ARCH_5E__) || defined(__ARM_ARCH_5TE__) \
	|| defined(__ARM_ARCH_5TEJ__)
#  define __ARM_ARCH 5
# else
#  define __ARM_ARCH 4
# endif
#endif

So if we pass an env like CFLAGS_aarch64_unknown_linux_gnu: "-D__ARM_ARCH=8", the ACLE is not working in the final binary result. It will cause an unexcepted error when we use rustls to handle the TLS connection

So, I think we may have two choice

1. Upgrade to manylinux_2_28, the compiler is newer
2. set export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc