Skip to content

Build failing in arm architecture by pymssql #32672

New issue
New issue
Closed
Closed
Build failing in arm architecture by pymssql#32672
Labels
area:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

Description

@starone99
starone99
opened on Jul 18, 2023

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

When we build Airflow, we have build erros when installing pymssql library which contained in constraints on ARM architecture. This issue has been reported to pymssql, but no idea how it will take to fix. I suggest comment out this library temporary to prevent build failure.

Defaulting to user installation because normal site-packages is not writeable
Collecting pymssql
  Using cached pymssql-2.2.7.tar.gz (170 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pymssql
  Building wheel for pymssql (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pymssql (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [119 lines of output]
      setup.py: platform.system() => Linux
      setup.py: platform.architecture() => ('64bit', 'ELF')
      setup.py: platform.libc_ver() => ('glibc', '2.31')
      setup.py: include_dirs => []
      setup.py: library_dirs => []
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-39
      creating build/lib.linux-aarch64-cpython-39/pymssql
      copying src/pymssql/__init__.py -> build/lib.linux-aarch64-cpython-39/pymssql
      running build_ext
      warning: src/pymssql/_mssql.pyx:27:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:28:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:29:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:30:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:31:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:32:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:35:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:36:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:38:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:82:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:230:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:264:4: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:333:4: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:831:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:919:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:1005:12: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:1379:16: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      warning: src/pymssql/_mssql.pyx:1713:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      
      cdef void init_mssql():
          if dbinit() == FAIL:
              raise MSSQLDriverException("dbinit() failed")
      
          dberrhandle(err_handler)
                      ^
      ------------------------------------------------------------
      
      src/pymssql/_mssql.pyx:2173:16: Cannot assign type 'int (DBPROCESS *, int, int, int, char *, char *) except? -1 nogil' to 'EHANDLEFUNC'
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      cdef void init_mssql():
          if dbinit() == FAIL:
              raise MSSQLDriverException("dbinit() failed")
      
          dberrhandle(err_handler)
          dbmsghandle(msg_handler)
                      ^
      ------------------------------------------------------------
      
      src/pymssql/_mssql.pyx:2174:16: Cannot assign type 'int (DBPROCESS *, DBINT, int, int, char *, char *, char *, LINE_T) except? -1 nogil' to 'MHANDLEFUNC'
      Compiling src/pymssql/_mssql.pyx because it changed.
      [1/1] Cythonizing src/pymssql/_mssql.pyx
      Traceback (most recent call last):
        File "/home/vscode/.local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/vscode/.local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/vscode/.local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 416, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 289, in <module>
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 343, in run
          self.run_command("build")
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 181, in build_extensions
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
          new_ext = cythonize(
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
          cythonize_one(*args)
        File "/tmp/pip-build-env-zf2sa7bh/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
          raise CompileError(None, pyx_file)
      Cython.Compiler.Errors.CompileError: src/pymssql/_mssql.pyx
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pymssql
Failed to build pymssql
ERROR: Could not build wheels for pymssql, which is required to install pyproject.toml-based projects

What you think should happen instead

No response

How to reproduce

Build any version of Airflow including pymssql module on ARM architecture.

Operating System

mac os 13.0

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. starone99
    added
    area:core
    kind:bugThis is a clearly a bug
    needs-triagelabel for new issues that we didn't triage yet
    on Jul 18, 2023

  2. boring-cyborg commented on Jul 18, 2023

    @boring-cyborg
    boring-cyborgBot
    on Jul 18, 2023

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

  3. shortcarver commented on Jul 18, 2023

    @shortcarver
    shortcarver
    on Jul 18, 2023

    I ran into this today as well. Appears to be fixed here: https://github.com/pymssql/pymssql/pull/803/files

    You can apply it by updating your requirements.txt to the following until the latest pymssql is released:

    pymssql @ git+https://github.com/matusvalo/pymssql.git@cython3_fix

  4. marcwimmer commented on Jul 19, 2023

    @marcwimmer
    marcwimmer
    on Jul 19, 2023

    this pull requests fixes that; i pinned cython version to 0.29.35 which works
    pymssql/pymssql#827

    can we bring that pull request online?

  5. pierrejeambrun commented on Jul 23, 2023

    @pierrejeambrun
    pierrejeambrun
    on Jul 23, 2023
    Member

    While mssql upstream is working on a new version fixing this, you can follow the recommended way to install the package with no-isolation.

    If you are trying to rebuild from source airflow, you can see here how we fixed our building process for ARM images, the trick is to use the PIP_CONSTRAINT env variable.
    #32748

  6. potiuk commented on Jul 24, 2023

    @potiuk
    potiuk
    on Jul 24, 2023
    Member

    Yes. @pierrejeambrun is right for now we need to wait for pymssql to fix it - this is their (and new Cython) problem and we can't do much about it.

    Hopefully Pymssql developers will provide a fix. but unti then there is a workaround for Mac OS ARM and let me document it here for others who might be looking for solution:

    1) have `brew` installed
2) brew install freetds
3) echo "Cython==0.29.36" >> /tmp/mssql-constraints.txt
4) PIP_CONSTRAINT=/tmp/mssql-constraints.txt pip install pymssql

    Then - in the same venv, installation of Airflow wiht microsoft-mssql extra/provider should correctly install airflow

    I will close the issue as there is not much we can do for now other than the workaround above.

  7. potiuk
    closed this as completedon Jul 24, 2023

  8. potiuk commented on Jul 24, 2023

    @potiuk
    potiuk
    on Jul 24, 2023
    Member

    And I also suggest everyone here to add more comments to the issue pymssql/pymssql#827 - unfortunately there is really not much we can do here - this is low-level installation problem, combination of four factors:

    1. Cython 3 released with breaking changes
    2. Pymssql stating that they need Cython > 0.29.22 (and not stating upper-binding).
    3. Pymssql being not compatible with new Cython 3.0.0
    4. pip using build isolation and downloading and using latest build dependencies when they are building package

    Since - we hope - the issue will be solved soon, this issue with clearly stated steps how to workaround it is best way to document it as users looking for the problem will find it here. If it will last for longer we might need to put it at the provider's page, though hopefully it will not be needed. pymssql is not installable for anyone on ARM MacOS without the workaround so I guess microsoft will fix the problem shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions