Sporadic EPERM from post start process fusermount unmount in GH actions CI environment.

[adamnovak]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release?
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Version of Singularity

This is with singularity-ce 4.5.0 from https://github.com/sylabs/singularity/releases/download/v4.5.0/singularity-ce_4.5.0-noble_amd64.deb

Describe the bug

When cwltool runs its continuous integration tests (which can involve running several simultaneous Singularity containers against the same SIF files), then on some CI runs, many tests will fail due to apparently not having permission to unmount something.

I reported this as #4044 (comment)

To Reproduce

Steps to reproduce the behavior:

Run the cwltool project's Github Actions continuous integration tests a few times. Commit common-workflow-language/cwltool@b4c828c should work, but commit common-workflow-language/cwltool@c0aab3f will use Singularity 4.5.0.

The tests that fail for this reason are generally the various Python version unit or mypy tests, such as: https://github.com/common-workflow-language/cwltool/actions/runs/30581144108/job/91005564119?pr=2284#step:9:4314

The offending logs look like:

  INFO [job iwdr-entry.cwl] /tmp/250rr2cg$ singularity \
      --quiet \
      run \
      --contain \
      --ipc \
      --cleanenv \
      --no-eval \
      --userns \
      --home \
      /tmp/250rr2cg:/YrcoOu \
      --mount=type=bind,source=/tmp/tp1ggaz1,target=/tmp \
      --pwd \
      /YrcoOu \
      /home/runner/work/cwltool/cwltool/docker.io_s_debian:stable-slim.sif \
      cat \
      example.conf
  WARNING: path /overlay-lowerdir/tmp is already overridden by /tmp/tp1ggaz1, replacing with /var/lib/singularity/mnt/session/tmp
  /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-23770563/root: Operation not permitted
  FATAL:   While running host post start tasks: while unmounting fuse directory: /tmp/rootfs-23770563/root: exit status 1
  /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-23770563/root: Operation not permitted
  CONFIGVAR=hello
  ERROR:   Failed to clean up FUSE mount: while unmounting fuse directory: /tmp/rootfs-23770563/root: exit status 1
  ERROR:   While running host cleanup tasks: encountered errors during CleanupHost: [while unmounting fuse directory: /tmp/rootfs-23770563/root: exit status 1]
  ERROR:   Unprivileged host cleanup failed: host cleanup failed
  FATAL:   host post start process failed: host post start tasks failed

So, under some circumstances (possibly simultaneous execution of other similar commands), when we run that singularity command (with docker.io/debian:stable-slim pulled to that SIF file and file /tmp/250rr2cg/example.conf that exists), Singularity can fail like this at the cleanup stage.

Expected behavior

No test failures should be caused by an Operation not permitted during Singularity's unmount step. If Singularity FUSE-mounted the SIF, it really should be expected to be able to unmount it. If files are somehow still open through the mount when Singularity goes to clean it up and are blocking the unmount (maybe because another simultaneous Singularity run is using it?) then Singularity should wait for them to close before trying to do the unmount.

OS / Linux Distribution

I don't have an interactive shell, but this is happening on the Github Actions flavor of Ubuntu 24.04.

  Image: ubuntu-24.04
  Version: 20260726.254.1
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20260726.254/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20260726.254

Installation Method

We install Singularity from the release .deb files:

          wget --no-verbose https://github.com/sylabs/singularity/releases/download/v4.5.0/singularity-ce_4.5.0-noble_amd64.deb
          sudo apt-get install -y ./singularity-ce_4.5.0-noble_amd64.deb environment-modules

Additional context

I reported this originally at #4044 (comment), but it turns out this issue is different because the underlying FUSE unmount failure is happening because of a different error code (Operation not permitted).

This bug doesn't occur 100% of the time; sometimes the relevant CI tests pass.

When it does occur, it seems to occur several times. Many tests in a single CI test job will fail with similar errors, varying in the temporary directory paths.

This smells a little bit like #2698, which failed at the same point for a different reason.

[adamnovak]

It looks like, drawing on https://www.man7.org/linux/man-pages/man2/umount.2.html and https://docs.kernel.org/filesystems/fuse/fuse.html#aborting-a-filesystem-connection and https://docs.kernel.org/filesystems/fuse/fuse.html#control-filesystem and https://stackoverflow.com/a/68586871, what should happen to completely destroy a FUSE mount is:

1. Try a normal unmount.
2. Try a force unmount.
3. Work out the device number of the mount, which is the FUSE connection number.
4. Abort that FUSE filesystem through the FUSE control filesystem's abort file for that connection. Possibly borrow Go code from bazil's fuse-abort utility.
5. This doesn't actually unmount the filesystem, but once it's aborted it can definitely be unmounted.

[dtrudg]

I'm afraid that this is probably going to need a way of reproducing the issue that is not dependent on running another project's GitHub actions workflows. If you are able to work up a reproducer that is not dependent on GH actions then we will be able to investigate sooner.

What would also be useful to know is when you first started seeing the problem? Did it occur after GH actions updated their Ubuntu image / you updated singularity etc. We routinely run tests of these flows in highly parallel automated testing across distributions, and haven't come across this issue. --debug output from singularity, and anything reported in dmesg, within the GH action runner might be useful, but you would need to modify your workflows to collect this information.

The issue isn't related to #2698 - the FUSE error there was 'invalid argument' due to mount options, and easily repeatable.

Forced un-mounting, and forcing a FUSE abort are not applicable here. The "post start host" process is performing a lazy unmount of the FUSE mounted container image in the host mount namespace. The FUSE mount is still present and used in the container mount namespace at this point. The EPERM error is surprising, and it will likely be difficult to track down the root cause, which is not necessarily in singularity itself.

[adamnovak]

Yeah, I don't see how this could get fixed without divorcing it from Github Actions and cwltool either.

@mr-c Do you know anything more about the history of this kind of test failure in the cwltool CI runs that might be useful here?

One way to approach reproducing the issue might be to just create a mock fusermount3 that exhibits this failure when asked to do this operation. Then Singularity could be adjusted to tolerate that behavior and pass the test. That wouldn't give any insight into why the kernel thinks we don't have permission to do these unmount operations, and might be easy to pass in ways that won't work in the real situation, and wouldn't allow for solutions that set things up in advance so that that the of permissions can't happen. But I don't know that that's possible to ensure this can never happen; I can always write an AppArmor profile that says e.g. "nobody is allowed to do lazy unmounts on this system because there's a CVE in them this week", and I might want Singularity to make an attempt to deal with the result.

In digging around for what a root cause might be, I found containers/oci-umount#48 about lazy unmount for NFS mounts in a container context being able to hang the system, and https://askubuntu.com/questions/1436234/umount2-for-an-nfs-mountpoint-returns-eperm-even-when-called-as-root about a situation in which even root isn't allowed to lazily unmount an NFS mount. Here we're dealing with FUSE mounts and not NFS mounts, but FUSE would have a similar situation where the actual backing storage is unboundedly asynchronous. Maybe someone, somewhere, sometimes thinks that it's not an acceptable idea to combine that with asynchronously dropping the mount?

Do we definitely need to figure out the root cause in the kernel/FUSE, or would it be possible to address the symptoms usefully? Can/should Singularity be made to cope with a situation where it genuinely isn't allowed to do the thing it is trying to do here?

[dtrudg]

One way to approach reproducing the issue might be to just create a mock fusermount3 that exhibits this failure when asked to do this operation. Then Singularity could be adjusted to tolerate that behavior and pass the test. That wouldn't give any insight into why the kernel thinks we don't have permission to do these unmount operations, and might be easy to pass in ways that won't work in the real situation, and wouldn't allow for solutions that set things up in advance so that that the of permissions can't happen. But I don't know that that's possible to ensure this can never happen; I can always write an AppArmor profile that says e.g. "nobody is allowed to do lazy unmounts on this system because there's a CVE in them this week", and I might want Singularity to make an attempt to deal with the result.

The problem here is that a failed unmount is an error state for common use of Singularity. We cannot e.g. allow failed unmounts to be ignored, and over time leave many dangling mounts that pollute an HPC node where perhaps 10s of container runs may happen on a parallel job. A permission error for an unmount, against a mount that singularity itself performed, is definitely indicative of a problem that must be addressed in some way.

While a failed unmount may not be an issue in a GH actions CI environment, this is not the main target usage of Singularity.

In digging around for what a root cause might be, I found containers/oci-umount#48 about lazy unmount for NFS mounts in a container context being able to hang the system, and https://askubuntu.com/questions/1436234/umount2-for-an-nfs-mountpoint-returns-eperm-even-when-called-as-root about a situation in which even root isn't allowed to lazily unmount an NFS mount. Here we're dealing with FUSE mounts and not NFS mounts, but FUSE would have a similar situation where the actual backing storage is unboundedly asynchronous. Maybe someone, somewhere, sometimes thinks that it's not an acceptable idea to combine that with asynchronously dropping the mount?

To be honest, looking across from FUSE to NFS and speculating is unlikely to be useful as there are large differences between these types of filesystems. It doesn't negate the need to collect specific debugging information. The repository you've found there is also for a tool that hasn't been updated for 8 years, and the issue is from 2020, so it's not even clear that the reported NFS situation is still an issue.

Effort needs to be directed to:

1. Getting an easier reproducer than the GH actions / CWL CI environment.
2. Collecting debug output, dmesg output, strace output etc. that can elucidate differences between successful and failing cases.

Do we definitely need to figure out the root cause in the kernel/FUSE, or would it be possible to address the symptoms usefully? Can/should Singularity be made to cope with a situation where it genuinely isn't allowed to do the thing it is trying to do here?

As above - we have to find the underlying cause. We cannot allow mounts to accumulate in the host namespace without an error being reported. To do so is not compatible with the target usage of Singularity.

If you can provide specific debug output, then this is something we can look into. Without that, I'm afraid this will be a low priority given the complexity of replicating, the fact that we haven't seen this ourselves or had a report from anyone else, and that it falls out of the primary HPC use case of Singularity. Apologies I can't promise anything else.