Policies for packages that use OpenCL

[peastman]

This issue is a followup to conda-forge/openmm-feedstock#7. We are trying to move OpenMM to be distributed through conda-forge, and some policy questions came up that need to be resolved. In particular, we have been told conda-forge has an established policy that any package that uses OpenCL is required to list ocl-icd as a dependency. We do not want to list it as a dependecy: given our particular needs, there doesn't exist any situation where it would benefit our users, and there do exist situations where it would cause problems.

Let me give some background information, since it will help to clarify the issues we're dealing with. OpenMM is a widely used package for molecular dynamics simulation. It can use OpenCL, but it doesn't require it. It comes with multiple computational backends: CUDA, OpenCL, and CPU. It automatically selects which one to use at runtime based on the available hardware and software (whether or CUDA and/or OpenCL is installed). The CUDA and OpenCL backends exist mainly for users with high end GPUs, which can be much faster than CPUs. For users who don't have one of those, it's generally better to use the CPU backend.

Accurate results are critical. This is scientific software, and errors could lead to users having to retract papers and losing months or years of work. We only support a small number of OpenCL implementations. On Linux, we support the ones from AMD and NVIDIA. Neither of those is conda installable. They are included with the video drivers. On macOS, we support Apple's implementation, which is built into the OS. These are the only ones we test. Using any other implementation would have a high risk of producing incorrect results. Our installation instructions tell users how to install appropriate OpenCL implementations. We are very concerned about anything that might lead to a user inadvertently using a different implementation they didn't even know was there.

OpenMM has APIs for C++, C, Fortran, and Python. So although we use conda as a distribution mechanism, it is part of a much larger ecosystem than just conda-forge, conda, or even Python. It needs to work correctly with all other libraries on the OS, regardless of how they are installed or what language they are used from.

So that's the background. Now let me describe our particular concerns.

1. Given our needs, there simply is no benefit to installing ocl-icd. None of the OpenCL implementations we support is conda installable.

2. Installing it creates new opportunities for the user to get incorrect results by silently using a different OpenCL implementation than the supported one we told them to install.

3. ocl-icd loads ICD files from a nonstandard location. Whereas all standard ICD loaders on Linux look for them in /etc/OpenCL/vendors, ocl-icd looks for them in a location inside the conda installation. If users want to use any OpenCL that was installed system-wide (such as the ones from the GPU vendors), they have to add a symlink pointing to the standard location. So including ocl-icd makes life harder for users. (A workaround is to also install ocl-icd-system, which creates the symlink for them, but it appears few packages currently do this.)

4. On macOS there is an additional, very serious problem: ocl-icd is written in a way which is incompatible with Apple's OpenCL framework. If two packages are loaded at the same time, one that links to ocl-icd and the other that links directly to Apple's framework (as nearly all OpenCL-based codes do on macOS), this will result in a crash. Linking to ocl-icd would therefore make OpenMM incompatible with nearly all other OpenCL based software that comes from any source other than conda-forge.

Given these issues, I suggest the existing policy should be reconsidered. Packages should not be required to depend on ocl-icd unless they have a specific reason to depend on it. Even better, I would suggest it should be a dependency for implementations of OpenCL, not for packages that use OpenCL. This is how the ICD mechanism was designed to work in the first place. ICD loaders are normally included with the implementation. They are never included with individual applications that just happen to use OpenCL. Tying the loader to the application rather than the implementation is contrary to the design of the mechanism.

This would make life easier for users of all OpenCL based packages. If they haven't specifically conda installed an OpenCL implementation, ocl-icd would not block the standard implementation that came with their GPU drivers from being found.

I also suggest that any use at all of ocl-icd on macOS should be carefully reconsidered. Until it can be made to coexist properly with the standard framework built into the OS, it just isn't appropriate for any package that is intended to be widely distributed.

cc: @jchodera @jaimergp

[jchodera]

A bit more info on OpenMM, for the curious:

• OpenMM website: http://openmm.org/
• GitHub repo: https://github.com/openmm/openmm

[CJ-Wright]

Thank you for putting this in.
@conda-forge/core thoughts?

I'm not certain how to turn this into generic policy yet.

[scopatz]

If you are worried about reproducibility it really seems like the best option is to have conda manage the ocl-icd and any OpenCL packages all the way down the stack. Then you can preciesly pin against those particular builds that you know you want to use.

[peastman]

Since the only OpenCL implementations we support are ones that aren't conda installable, that isn't really relevant to us. Besides, pinning GPU drivers generally isn't an option.

[scopatz]

Why aren't they conda installable? ocl-idc could be it looks like. And other ones could probably have CDTs for them

[isuruf]

@scopatz, ocl-icd is an ICD loader for OpenCL implementations. These implementations are indicated with a text file <name>.icd in /etc/OpenCL/vendors usually. Some implementations allow linking to the OpenCL implementation directly which means your application can't use other implementations.
Some ICD loaders provide a way to override this vendors directory. ocl-icd packaged in conda looks in $PREFIX/etc/OpenCL/vendors and installing ocl-icd-system will look in both $PREFIX/etc/OpenCL/vendors and /etc/OpenCL/vendors.

Some OpenCL implementations like Apple and NVIDIA are proprietary and we can't package them.

What @peastman is suggesting is to not use ICD loader and allow whatever ICD loader is in the system on Linux and to use only Apple implementation in OSX.

I'm strongly against this idea. Other OpenCL applications in conda-forge use the ICD loader from conda-forge and I'd like to keep the consistency.

You can't block other OpenCL implementations if you use ocl-icd-system because a user might install another implementation in /etc/OpenCL/vendors.

You can however block other conda OpenCL implementations by using the run_constrained trick (but I don't see the point as this won't block implementations installed in /etc/OpenCL/vendors that OpenMM don't support that).

[peastman]

@scopatz let me give a little background on the ICD mechanism. I think that might clarify things.

When Khronos was developing OpenCL, they knew there would be multiple implementations. For example, they figured each GPU manufacturer would create its own implementation. They also figured you might want multiple implementations installed at once, for example one for GPU and one for CPU. So they split the interface from the implementation. libOpenCL is the library that applications link to. It contains the interface but no implementation. When it gets loaded, it looks in a standard location (/etc/OpenCL/vendors on Linux, C:\System\OpenCL\vendors on Windows) for "Installable Client Driver" (ICD) files. Those point to the libraries containing particular implementations.

The intended behavior is that when an OpenCL implementation is installed, the installer will 1) put its libraries in some vendor-specific place, 2) put an ICD file pointing to them in the standard location, and 3) install a copy of libOpenCL in a standard location for system libraries, such that it will always be found by applications.

That's how it works on Linux and Windows, but Apple chose a different approach. They keep tight control over their platform. For example, they don't let GPU vendors install their own drivers. Everything has to go through Apple. And they made OpenCL work the same way. They have their own implementation (OpenCL.framework) which is included as part of the OS. It supports all GPU models that are found in Macs, but doesn't provide any mechanism for users to install other implementations.

(Note: I'm not commenting on whether this is a good system. Obviously there are objections that can be raised to Apple's walled garden, "we control everything" approach. But for better or worse, that's how the platform works, and we need to interoperate with the rest of the ecosystem.)

All of this behavior is described in the Khronos ICD Installation Guidelines. One section is particularly relevant, where it sets out the general guidelines for how and when new ICD loaders should be installed or uninstalled.

Vendor installers MUST install and uninstall their ICD-compliant implementations in such a way that the installer:

1. Installs its own ICD loader library if and only if the existing ICD loader library is older than the one being installed.

2. Does not remove ICD loader library at uninstall if other implementations exist.

3. Does not cause existing installations to become inoperable or unusable in any way. This includes, but is not limited to, WHQL and similar signed package certification check failures.

4. Does not manipulate the vendor enumeration order within the ICD loader library except to add (or remove) the new vendor implementation.

The important thing to notice here is that the conda-forge icd-ocl installer violates three of these four requirements (1, 3, and 4). As long as you view conda-forge as the entire universe, that isn't a problem. But as soon as you want to interoperate with anything else (either OpenCL implementations from other sources, or packages from other sources that use OpenCL), it creates serious problems.

[isuruf]

I don't see how 3 is violated. As long as the ocl-icd library is up-to-date in conda-forge, 1 is not violated either.

[isuruf]

4 is violated by the ICD loader by Khronos. https://github.com/KhronosGroup/OpenCL-ICD-Loader#table-of-debug-environment-variables. See OCL_ICD_VENDORS env variable.

[jchodera]

What @peastman is suggesting is to not use ICD loader and allow whatever ICD loader is in the system on Linux and to use only Apple implementation in OSX.

@isuruf: I believe @peastman suggested reversing the dependency policy:

I would suggest it should be a dependency for implementations of OpenCL, not for packages that use OpenCL.

Could you comment on this part of the proposal?

[isuruf]

What I don't get is, you are totally fine with the user installing an unsupoprted OpenCL implementation in the system, but against installing an unsupported OpenCL with conda.

[peastman]

Since that situation has never come up in 10 years, I'm not too worried about it. Users just don't install OpenCL implementations much, except the ones that come with their GPU drivers. Making them conda installable makes it much easier, and allows it to be invisibly installed without the user ever noticing, just because some unrelated package listed it as a dependency.

But let's get back to the specific proposals. My suggestions were

1. ocl-icd should be a dependency of packages that implement OpenCL, not ones that use OpenCL.
2. Mac packages should use OpenCL.framework unless the developer has a good reason not to.

To be sure, this isn't a complete solution for us. We'll need to take additional steps to protect against using unsupported OpenCL implementations (such as the run_constrained trick). But I'm not suggesting this just to make my own life easier. I believe it's overall a better policy and it will benefit other developers and users too.

[isuruf]

If you use the run_constrained trick, why not install ocl-icd? This has the added benefit of reproducability and also some nice features like OCL_ICD_DEBUG, OCL_ICD_VENDORS. Some ICD loaders on system supports these variables and some don't and using ocl-icd makes this consistent.

[isuruf]

Mac's OpenCL framework is deprecated and has been buggy since OSX 10.14. See inducer/pyopencl#276.