files in ENROOT_CACHE_PATH not group readable

[mathrock74]

the files created by enroot in ENROOT_CACHE_PATH are not readable by the group and therefore can't be used by other users
I solved this for me with adding

cache_path="$(sudo -u "$SLURM_JOB_USER" sh -c 'echo "/var/lib/enroot-cache/group-$(id -g)"')"
find "$cache_path" -type f -exec chmod 0660 {} \;

to /etc/slurm/epilog.d/50-lastuserjob-all-enroot-dirs

[ajdecon]

This is a nice performance optimization to allow users to share cache, but I don't think this is something we want to enable by default. Different users on the cluster may have access to different containers, and we want to maintain stricter permissions by default.

[mathrock74]

Actually this is not an optimization for me, it makes the system work. When I leave it at the default and user foo runs a container, then user bar runs the same container he get's "permission denied" from pyxis:

foo@deepops-login:~$ srun -w deepops-node3 --container-image=centos grep PRETTY /etc/os-release
pyxis: importing docker image ...
PRETTY_NAME="CentOS Linux 8"

bar@deepops-login:~$ srun -w deepops-node3 --container-image=centos grep PRETTY /etc/os-release
pyxis: importing docker image ...
slurmstepd: error: pyxis: child 20158 failed with error code: 1
slurmstepd: error: pyxis: failed to import docker image
slurmstepd: error: pyxis: printing contents of log file ...
slurmstepd: error: pyxis: [INFO] Querying registry for permission grant
slurmstepd: error: pyxis: [INFO] Authenticating with user:
slurmstepd: error: pyxis: [INFO] Authentication succeeded
slurmstepd: error: pyxis: [INFO] Fetching image manifest list
slurmstepd: error: pyxis: [INFO] Fetching image manifest
slurmstepd: error: pyxis: [INFO] Found all layers in cache
slurmstepd: error: pyxis: [INFO] Extracting image layers...
slurmstepd: error: pyxis: tar (child): /var/lib/enroot-cache/group-100/7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621: Cannot open: Permission denied
slurmstepd: error: pyxis: tar (child): Error is not recoverable: exiting now
slurmstepd: error: pyxis: tar: Child returned status 2
slurmstepd: error: pyxis: tar: Error is not recoverable: exiting now
slurmstepd: error: pyxis: couldn't start container
slurmstepd: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd: error: Failed to invoke spank plugin stack
srun: error: deepops-node3: task 0: Exited with exit code 1

Both users have primary group 100. System is Ubuntu 20.04.
Are users expected to have unique primary groups?

[ajdecon]

Oh, that's interesting! In our own usage we do expect users to have unique primary groups, and that's how we test.

But this doesn't seem like it should be a hard requirement for using Enroot/Pyxis with DeepOps, as I know there are plenty of sites that don't have unique primary groups per user.

Re-opening the issue and will take a look at this when I get a chance. I'm still not sure we should make these group-readable by default, but it's possible we can just make the cache directories unique per user.

[ajdecon]

@mathrock74 : My proposed fix for this issue is in #997.

However, it's possible you'll want to keep the group-readable feature for this cache, as that does seem handy when you assume that all users can access the same images 😄

In which case you may want to set up your own custom set of prolog and epilog scripts. You can do this with the prolog_fileglob and epilog_fileglob Ansible vars to point to your own scripts instead of the defaults: https://github.com/NVIDIA/deepops/blob/master/roles/slurm/defaults/main.yml#L80

[ajdecon]

#997 merged, closing issue.