Python's ctypes broken on macOS Big Sur

[dhess]

Python's ctypes support is broken on macOS Big Sur. This currently affects nixpkgs. Here's an example with python3.8-regex:

https://gist.github.com/dhess/e1b2662a6732618b252096e0de3a04d4

This has been fixed in CPython trunk and backported to 3.9:

python/cpython#22855
python/cpython#23295

Unfortunately, it's not yet been backported to 3.8, which is the default python3Packages set in nixpkgs.

[FRidh]

That's a pretty big backport! Probably in time it will be backported to 3.8 as well.

[jonringer]

once they do another release, these will be added "automatically"

[rb2k]

Has this actually been resolved in any new release?
I think 3.8.7 as well as 3.9.1 are currently both failing the find_library call for me on Big Sur (fine on Catalina)

# sw_vers
ProductName:	macOS
ProductVersion:	11.2
BuildVersion:	20D64

3.8.7:

nix/store/zlck50w2v1ypwk9yqkfy8mf4kmcs89c9-python3-3.8.7/bin/python3.8
Python 3.8.7 (default, Feb  5 2021, 19:55:34) 
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
>>> 

And 3.9:

nix/store/fk1lwfjvpzzh7d1sj8jyi832kzqczsmi-python3-3.9.1/bin/python3.9
Python 3.9.1 (default, Feb  5 2021, 19:44:28) 
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
>>> 

[i077]

Getting a similar error when starting jupyter lab (which uses ctypes) on Big Sur (please forgive the ugly shell invocation):

% nix shell --expr '(builtins.getFlake "github:NixOS/nixpkgs/nixpkgs-unstable").legacyPackages.x86_64-darwin.python39.withPackages (ps: [ ps.jupyterlab ])' --impure -c jupyter-lab
Traceback (most recent call last):
  File "/nix/store/nzw8mizwzxxadhcwxfr1ldis8wabk50x-python3.9-jupyterlab-2.2.9/bin/.jupyter-lab-wrapped", line 6, in <module>
    from jupyterlab.labapp import main
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab/labapp.py", line 14, in <module>
    from jupyterlab_server import slugify, WORKSPACE_EXTENSION
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/__init__.py", line 4, in <module>
    from .app import LabServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/app.py", line 9, in <module>
    from .server import ServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/server.py", line 20, in <module>
    from notebook.notebookapp import aliases, flags, NotebookApp as ServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/notebook/notebookapp.py", line 86, in <module>
    from .services.contents.filemanager import FileContentsManager
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/notebook/services/contents/filemanager.py", line 17, in <module>
    from send2trash import send2trash
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/send2trash/__init__.py", line 12, in <module>
    from .plat_osx import send2trash
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/send2trash/plat_osx.py", line 17, in <module>
    GetMacOSStatusCommentString = Foundation.GetMacOSStatusCommentString
  File "/nix/store/nvxkg44pcrijvqmv5m6inz36rf7zgshy-python3-3.9.1/lib/python3.9/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/nix/store/nvxkg44pcrijvqmv5m6inz36rf7zgshy-python3-3.9.1/lib/python3.9/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, GetMacOSStatusCommentString): symbol not found

My guess is that it has something to do with the patch to ctypes being applied to all the python derivations.

[jonringer]

Has this actually been resolved in any new release?

Upsteam hasn't cut any new releases yet.

[i077]

But cpython 3.9.1 (which is in nixpkgs-unstable) should work on Big Sur since it has the patch from python/cpython#23295, right?

[i077]

For reference, the latest Python 3.9 from Homebrew works on Big Sur

% which python3
/usr/local/bin/python3
% python3      
Python 3.9.1 (default, Jan 27 2021, 11:07:38) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
'/usr/lib/libc.dylib'
>>> 

[fanzeyi]

It works on Homebrew because they manually applied the patch:

https://github.com/Homebrew/homebrew-core/blob/012fff6f2506fc351fc04b5bb1516669614bcd19/Formula/python%403.9.rb#L78-L86

Nevermind, I was distracted while reading through the Homebrew's commit history, this has been removed in their 3.9.1 update. I'm still not sure why Homebrew's Python 3.9.1 works but not nix

[fanzeyi]

So Python 3.9.1 should have ctypes work on Big Sur. Nix's Python still doesn't work because nixpkg's build doesn't provide mach-o/dyld.h header. Causing this check to fail:

python/cpython@4176193#diff-90d08e583c4c9c6f391b2ae90f819f600a6326928ea9512c9e0c6d98e9f29ac2R12040

I'm checking if there is any way to fix it. (I saw there is nixpkgs.darwin.dyld but it doesn't seem to solve the issue)

[fanzeyi]

Oh well, this is a fun digging.

So apparently dyld in nixpkgs was last updated in macOS 10.12 release to 433.5. The function used in fixing Python's ctypes issue appeared in dyld-832.7.1.tar.gz. So this issue probably is gonna hinge on #101229 (That one is only updating to 10.13, which is still three versions away from having _dyld_shared_cache_contains_path.) Now I can certainly write an overlay locally to update dyld and build Python with it but I don't know if there is a good way to have the Python in nixpkg supports it.

Update: now I realized simply updating dyld will not work since it depends on xnu (because of the Availability.h header), and updating xnu seems to be tricky (heavily patched in Nix and I don't want to rebuild the world). I resorted to patching mach-o/dyld.h with only the addition of the new function (It's just an extern so why not?), and one more patch to Python bypassing a runtime check for reasons I don't exactly understand (possibly related to MACOSX_DEPLOYMENT_TARGET setting).

My messy overlay file and patches (note this is very reckless so probably only use if you know what it means): https://gist.github.com/fanzeyi/3af0396eb5cfaa81ad30af427b1e8069