canonical-ubuntu-qa team mailing list archive

[Manuel Diewald <2076611@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Three tests in our overlayfs kernel testsuite mount overlayfs
filesystems in lxd containers with ubuntu:22.04 images in order to
perform their respective checks:

 - ovl-idmapped-mount.sh
 - ovl-mkdir-xattrs.sh
 - ovl-refcnt.sh

All of them started failing for noble in security cycle s2024.07.08 and
SRU cycle 2024.08.05 on all platforms with

mount: <mountpoint>: cannot mount overlay read-only.

and multiple apparmor denials for the mount attempts in the kernel log,
similar to:

audit: type=1400 audit(1723463999.712:603): apparmor="DENIED"
operation="mount" class="mount" info="failed perms check" error=-13
profile="lxd-ovl-test_</var/snap/lxd/common/lxd>"
name="/ovlissue/mountpointtmp/firstunion/" pid=10136 comm="mount"
fstype="overlay" srcname="overlay"

The tests will also fail with older kernels from previous cycles when
running with an up-to-date lxd snap. So this is not a regression of the
latest kernels. I managed to trace this back to an update to the lxd
snap which happened while our regression testing for both security and
SRU cycle were running (instances that ran overlayfs tests before the
snap update were able to successfully complete the testsuite).

The update from

 5.21/stable:      5.21.2-34459c8 2024-07-24 (29568)

to

 latest/stable:    6.1-4cd98ce    2024-08-06 (29727)

broke the tests.

What seems to break the tests is that the new snap now ships
apparmor_parser in the snap itself. This commit seems to add the
apparmor binaries and libraries to the lxd snap
(https://github.com/canonical/lxd-pkg-snap):

commit 8591ed414b4dab7d045ccb344eb914d07f67b93f
Author: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@xxxxxxxxxxxxx>
Date:   Fri Jun 21 13:09:47 2024 +0200

    snapcraft: add apparmor part
    
    Let's ship newer version of AppArmor with support of new features.
    Only needed for core22.
    
    Signed-off-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@xxxxxxxxxxxxx>

My suspicion is that apparmor_parser being available in the snap might
enable additional apparmor policies, causing the mount operation to fail
with insufficient permissions. It seems to be necessary to launch the
lxd instance with security.nesting=true, which makes the tests pass
again. This does not imply security concerns and is considered to be the
new default for unprivileged containers anyway:

https://github.com/canonical/lxd/issues/13631

Some interesting observations I made while trying to debug this:

 - The working lxd snap (v5.21.2 #29568) fails in the exact same way when the snap is modified to contain the apparmor_parser binary. The version of apparmor does not matter it seems, it fails with both apparmor version 4.0.0-beta3 (currently latest in noble) as well as 4.0.2 (which the lxd snap v6.1 #29727 ships).
 - Building lxd from source (either v5.21.2, v6.1 or tip) will make the test also fail (I suspect because the system installation of apparmor/apparmor_parser is available). Removing the apparmor_parser binary will disable apparmor support and let the test succeed.
 - Modifying the lxd snap (v6.1 #29727) to not contain the apparmor_parser binary also makes the test pass.

In conclusion, neither the lxd version, the apparmor version nor the
kernel version seem to matter. The only factor that determines whether
mount operations and hence the tests will succeed is whether
apparmor_parser is included in the snap image or not.

How to reproduce (on noble):
sudo snap refresh lxd --channel latest/stable
git clone https://git.launchpad.net/~canonical-kernel-team/+git/overlay-shiftfs-tests
cd overlay-shiftfs-tests/tests
./ovl-refcnt.sh

** Affects: ubuntu-kernel-tests
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2076611

Title:
  overlayfs tests that use lxd fail with lxd snap version 6.1

Status in ubuntu-kernel-tests:
  New

Bug description:
  Three tests in our overlayfs kernel testsuite mount overlayfs
  filesystems in lxd containers with ubuntu:22.04 images in order to
  perform their respective checks:

   - ovl-idmapped-mount.sh
   - ovl-mkdir-xattrs.sh
   - ovl-refcnt.sh

  All of them started failing for noble in security cycle s2024.07.08
  and SRU cycle 2024.08.05 on all platforms with

  mount: <mountpoint>: cannot mount overlay read-only.

  and multiple apparmor denials for the mount attempts in the kernel
  log, similar to:

  audit: type=1400 audit(1723463999.712:603): apparmor="DENIED"
  operation="mount" class="mount" info="failed perms check" error=-13
  profile="lxd-ovl-test_</var/snap/lxd/common/lxd>"
  name="/ovlissue/mountpointtmp/firstunion/" pid=10136 comm="mount"
  fstype="overlay" srcname="overlay"

  The tests will also fail with older kernels from previous cycles when
  running with an up-to-date lxd snap. So this is not a regression of
  the latest kernels. I managed to trace this back to an update to the
  lxd snap which happened while our regression testing for both security
  and SRU cycle were running (instances that ran overlayfs tests before
  the snap update were able to successfully complete the testsuite).

  The update from

   5.21/stable:      5.21.2-34459c8 2024-07-24 (29568)

  to

   latest/stable:    6.1-4cd98ce    2024-08-06 (29727)

  broke the tests.

  What seems to break the tests is that the new snap now ships
  apparmor_parser in the snap itself. This commit seems to add the
  apparmor binaries and libraries to the lxd snap
  (https://github.com/canonical/lxd-pkg-snap):

  commit 8591ed414b4dab7d045ccb344eb914d07f67b93f
  Author: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@xxxxxxxxxxxxx>
  Date:   Fri Jun 21 13:09:47 2024 +0200

      snapcraft: add apparmor part
      
      Let's ship newer version of AppArmor with support of new features.
      Only needed for core22.
      
      Signed-off-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@xxxxxxxxxxxxx>

  My suspicion is that apparmor_parser being available in the snap might
  enable additional apparmor policies, causing the mount operation to
  fail with insufficient permissions. It seems to be necessary to launch
  the lxd instance with security.nesting=true, which makes the tests
  pass again. This does not imply security concerns and is considered to
  be the new default for unprivileged containers anyway:

  https://github.com/canonical/lxd/issues/13631

  Some interesting observations I made while trying to debug this:

   - The working lxd snap (v5.21.2 #29568) fails in the exact same way when the snap is modified to contain the apparmor_parser binary. The version of apparmor does not matter it seems, it fails with both apparmor version 4.0.0-beta3 (currently latest in noble) as well as 4.0.2 (which the lxd snap v6.1 #29727 ships).
   - Building lxd from source (either v5.21.2, v6.1 or tip) will make the test also fail (I suspect because the system installation of apparmor/apparmor_parser is available). Removing the apparmor_parser binary will disable apparmor support and let the test succeed.
   - Modifying the lxd snap (v6.1 #29727) to not contain the apparmor_parser binary also makes the test pass.

  In conclusion, neither the lxd version, the apparmor version nor the
  kernel version seem to matter. The only factor that determines whether
  mount operations and hence the tests will succeed is whether
  apparmor_parser is included in the snap image or not.

  How to reproduce (on noble):
  sudo snap refresh lxd --channel latest/stable
  git clone https://git.launchpad.net/~canonical-kernel-team/+git/overlay-shiftfs-tests
  cd overlay-shiftfs-tests/tests
  ./ovl-refcnt.sh

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2076611/+subscriptions