libpython_link_mode=shared for unix platforms ?

[touilleMan]

Hi @indygreg,

I working on a python binding for the Godot game engine
Godot works as a standalone application that can be extended by loading shared libraries, hence my binding must embed a python interpreter.

So currently I build CPython from the source as part of the binding build, but (as you very well know 😄 ) this is a tedious and error-prone step.
Hence I'll be really interested in using your pre-build releases instead ;-)

However my binding is written in Cython which produces native modules that depend on libpythonXY.so, but your builds only provide libpythonXY.so for Windows :'(

So what the reasons shared library is not available for Unix ? Is there something I can do to help ? ;-)
Does a cheap hack like gcc -shared -o libpythonXY.so build/Object/**.so or recompiling libpythonX.Ym.a into a shared lib could work ?

Another important requirement for godot-python is to allow game developer to be able to use pip to access the full Python ecosystem. Is there some limitations I would face with your builds ?
(From why I've seen in the documentation, the main issues are the shebang in the pip binary which isn't a big deal and the fact that plenty of native lib expect to have a libpython.so available but my project already have this requirement by itself so I have to solve this anyway)

PS: I've been following your Pyoxidizer project since it announcement, it's an incredible work you're doing at addressing one of the biggest weakness of Python. I guess we can confidently say you are building a killer feature for Python 😃

[indygreg]

When I initially wrote this project, the aim was static only builds (to support PyOxidizer's initial vision of single file executables). Only recently have we added support for shared libraries on Windows (out of necessity to support pre-compiled extension modules). That's why we don't distribute a libpythonXY.so outside of Windows.

I agree that shipping a shared library would be useful. I think the Linux and macOS distributions should have shared library distribution variants, if not distribute both the shared and static libraries in the same distribution (it should be possible on these platforms whereas Windows needs separate builds/distributions because of symbol visibility wonkiness).

In the mean time, you should be able to produce your own .so on Linux by linking the .o files in the distribution. You'd need to parse the PYTHON.json file to pick up the exact set of object files you need. You might be able to follow the code at https://github.com/indygreg/PyOxidizer/blob/48a4cab593a53a1c11bf313981f6b29818941ee0/pyoxidizer/src/py_packaging/libpython.rs#L77 to see how PyOxidizer does it.

As for pip, aside from the shebang and potential shared library issues, it should just work. The pip in the distribution archives does have some patches applied to work around pypa/packaging#294, so you'll want to use the distributed pip. There may also be issues compiling extension modules, as things like paths to the Python .h files may reference paths in the build environment. If you run into specific failures, please report them so we can track fixing them!

[indygreg]

I just committed a change to produce a dynamic libpython by default on Linux and macOS. The distribution archives now contain both a shared and static libpython.

There might be some subtle bugs lingering. But I'm pretty confident basic noopt configurations just work. I'm unsure when I will upload new distributions, however, as there are a few other issues I want to get sorted out before doing a new release.

[touilleMan]

That's pretty cool, thanks a lot !

I'm unsure when I will upload new distributions

I'm going to do a custom build from the master, I'll notify you if I encounter any trouble 😄