touch-packages team mailing list archive

[Jamie Strandboge <jamie@xxxxxxxxxx>]

** Description changed:

  apparmor_parser can take a long time to compile policy especially when there is a lot of policy, so we want to utilize compiled cache profile as much as possible. Cache files will have to be regenerated in the following cases:
   * the kernel .features file is updated (eg, new features are added to
     apparmor in the new kernel)
   * apparmor itself is updated
   * on devices with click packages, apparmor-easyprof-ubuntu and/or
     click-apparmor is updated
  
  As of 2014-10-02, what can be expected is:
  
  - Systems with system-image updates (eg, Ubuntu Touch):
    - First boot will use the precompiled cache files in the rootfs or custom
      tarball and be fast
    - Reboots will use the cache files on the device and be fast
    - First boots after upgrades will use the cache files on the device if the
      above conditions are not met and be fast
    - Production devices will not meet any of those conditions except under
      exceptional and rare circumstances (eg, major OS upgrades like 14.10 to
      15.04) and be fast
    - First boots after upgrades that meet one of the conditions will need to
      regenerate the cache. This can happen on development releases where the
      kernel features file, apparmor, apparmor-easyprof-ubuntu or
      click-apparmor are still under development and getting updates
  - Systems with apt updates (eg, current Ubuntu Desktop and Server):
    - First boot will compile cache files
    - Reboots will use the cache files on the machine and be fast
    - First boots after upgrades will use the cache files on the machine if the
      above conditions are not met and be fast
    - Stable releases of Ubuntu will not meet any of those conditions except
      under exceptional and rare circumstances (eg, major OS upgrades like
      14.10 to 15.04) and be fast
    - First boots after upgrades that meet one of the above conditions will
      need to regenerate the cache. This can happen on development releases
      where the kernel features file, apparmor, apparmor-easyprof-ubuntu or
      click-apparmor are still under development and getting updates
  
  In addition to the above, updates to only apparmor-easyprof-ubuntu will
  regenerate the cache files for only the policy that is affected (eg, if
  there is a change to the location policy group in policy version 1.2,
  only apps using this policy version and this policy group will need to
  be recompiled).
  
  Planned improvements (in order of most likely to be done first):
  1. Finetuning the checks to invalidate the cache (eg, .md5sums could only
     be for /etc/apparmor.d/abstractions, ...) - WONTFIX (will want an
     md5sum on apparmor_parser since it could change the cache and the md5sum
     will always change. Furthermore, apparmor-easyprof-ubuntu is all policy
     so there is no gain there. click-apparmor could possibly benefit, but
     it doesn't change often and when it does, it is typically for policy)
  2. Investigate ways to utilize the custom tarball and rootfs precompiled
     cache files on upgrades when apparmor, apparmor-easyprof-ubuntu and
     click-apparmor are updated
  3. Improve cache handling for app store apps (eg, having the app store
-    server precompile them so that the device can download them when it 
-    needs to rather than having to regenerate them itself): WONTFIX 
-    (doesn't scale)
- 4. For systems with apt upgrades, compile the policy either during 
-    install or on kernel upgrade rather than on boot
- 5. Support cache files per kernel .features file (or kernel version). 
-    This will allow people to boot into a previous kernel without having 
-    to recompile policy
+    server precompile them so that the device can download them when it
+    needs to rather than having to regenerate them itself): WONTFIX
+    (doesn't scale)
+ 4. For systems with apt upgrades, compile the policy either during
+    install or on kernel upgrade rather than on boot
+ 5. Support cache files per kernel .features file (or kernel version).
+    This will allow people to boot into a previous kernel without having
+    to recompile policy
  6. Improve parser compile time
  7. Investigate how to utilize profile composition and profile stacking to
     decrease compile and load times (eg, one idea is that the policy template
     is compiled once and each policy group once such that the parser need
     only stitch the already compiled bits together)
  
  Work is planned for the medium term for '1-3' with '4' and '5' coming as
- needed. '6' and '7' will be handled in the long term.
+ needed. '6' will of course help, but it is already very optimized and
+ compile average ~2 seconds on armhf per profile (note: already faster
+ than Android's 'optimizing apps' per app on a nexus 7) -- if we cut that
+ in half (which is debatable if possible) a typical user with 300 apps
+ would still have to wait 300 seconds so other techniques like '2' should
+ be employed. '6' and '7' will be handled in the long term.
  
  Original description:
  Just updated my Nexus 7 2013 from #160 to #161. It's been sat at the Google logo for 15 minutes now. It looks and feels like it's hung. As a user I'd be rebooting it thinking it had crashed by now. I shell in and find apparmor_parser  using a lot of cpu for a long time.
  
  top - 00:14:01 up 15 min,  2 users,  load average: 5.12, 4.85, 3.21
  Tasks: 202 total,   2 running, 200 sleeping,   0 stopped,   0 zombie
  %Cpu(s): 50.5 us,  0.8 sy,  0.0 ni, 48.5 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
  KiB Mem:   1848024 total,   787400 used,  1060624 free,    54216 buffers
  KiB Swap:    32764 total,        0 used,    32764 free.   579228 cached Mem
  
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   1970 root      20   0    4976   3652    852 R  99.8  0.2  14:31.04 apparmor_parser
   2596 phablet   20   0    5996   1264    824 R   1.3  0.1   0:08.79 top
    914 root       0 -20    7572    552    396 S   0.7  0.0   0:05.02 mpdecision
     21 root      20   0       0      0      0 S   0.3  0.0   0:00.92 kworker/0:1
    229 root      20   0       0      0      0 S   0.3  0.0   0:00.10 jbd2/mmcblk0p30
    982 root      20   0   38856   1164    868 S   0.3  0.1   0:01.77 adbd
   2570 phablet   20   0   10540   1456    692 S   0.3  0.1   0:02.30 sshd
      1 root      20   0    3884   2648   1068 S   0.0  0.1   0:05.98 init
      2 root      -2   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
      3 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/0
  
  ... it eventually finished after 18 minutes.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apparmor in Ubuntu.
https://bugs.launchpad.net/bugs/1350598

Title:
  AppArmor policy compile improvements

Status in AppArmor Linux application security framework:
  Triaged
Status in apparmor package in Ubuntu:
  Triaged

Bug description:
  apparmor_parser can take a long time to compile policy especially when there is a lot of policy, so we want to utilize compiled cache profile as much as possible. Cache files will have to be regenerated in the following cases:
   * the kernel .features file is updated (eg, new features are added to
     apparmor in the new kernel)
   * apparmor itself is updated
   * on devices with click packages, apparmor-easyprof-ubuntu and/or
     click-apparmor is updated

  As of 2014-10-02, what can be expected is:

  - Systems with system-image updates (eg, Ubuntu Touch):
    - First boot will use the precompiled cache files in the rootfs or custom
      tarball and be fast
    - Reboots will use the cache files on the device and be fast
    - First boots after upgrades will use the cache files on the device if the
      above conditions are not met and be fast
    - Production devices will not meet any of those conditions except under
      exceptional and rare circumstances (eg, major OS upgrades like 14.10 to
      15.04) and be fast
    - First boots after upgrades that meet one of the conditions will need to
      regenerate the cache. This can happen on development releases where the
      kernel features file, apparmor, apparmor-easyprof-ubuntu or
      click-apparmor are still under development and getting updates
  - Systems with apt updates (eg, current Ubuntu Desktop and Server):
    - First boot will compile cache files
    - Reboots will use the cache files on the machine and be fast
    - First boots after upgrades will use the cache files on the machine if the
      above conditions are not met and be fast
    - Stable releases of Ubuntu will not meet any of those conditions except
      under exceptional and rare circumstances (eg, major OS upgrades like
      14.10 to 15.04) and be fast
    - First boots after upgrades that meet one of the above conditions will
      need to regenerate the cache. This can happen on development releases
      where the kernel features file, apparmor, apparmor-easyprof-ubuntu or
      click-apparmor are still under development and getting updates

  In addition to the above, updates to only apparmor-easyprof-ubuntu
  will regenerate the cache files for only the policy that is affected
  (eg, if there is a change to the location policy group in policy
  version 1.2, only apps using this policy version and this policy group
  will need to be recompiled).

  Planned improvements (in order of most likely to be done first):
  1. Finetuning the checks to invalidate the cache (eg, .md5sums could only
     be for /etc/apparmor.d/abstractions, ...) - WONTFIX (will want an
     md5sum on apparmor_parser since it could change the cache and the md5sum
     will always change. Furthermore, apparmor-easyprof-ubuntu is all policy
     so there is no gain there. click-apparmor could possibly benefit, but
     it doesn't change often and when it does, it is typically for policy)
  2. Investigate ways to utilize the custom tarball and rootfs precompiled
     cache files on upgrades when apparmor, apparmor-easyprof-ubuntu and
     click-apparmor are updated
  3. Improve cache handling for app store apps (eg, having the app store
     server precompile them so that the device can download them when it
     needs to rather than having to regenerate them itself): WONTFIX
     (doesn't scale)
  4. For systems with apt upgrades, compile the policy either during
     install or on kernel upgrade rather than on boot
  5. Support cache files per kernel .features file (or kernel version).
     This will allow people to boot into a previous kernel without having
     to recompile policy
  6. Improve parser compile time
  7. Investigate how to utilize profile composition and profile stacking to
     decrease compile and load times (eg, one idea is that the policy template
     is compiled once and each policy group once such that the parser need
     only stitch the already compiled bits together)

  Work is planned for the medium term for '1-3' with '4' and '5' coming
  as needed. '6' will of course help, but it is already very optimized
  and compile average ~2 seconds on armhf per profile (note: already
  faster than Android's 'optimizing apps' per app on a nexus 7) -- if we
  cut that in half (which is debatable if possible) a typical user with
  300 apps would still have to wait 300 seconds so other techniques like
  '2' should be employed. '6' and '7' will be handled in the long term.

  Original description:
  Just updated my Nexus 7 2013 from #160 to #161. It's been sat at the Google logo for 15 minutes now. It looks and feels like it's hung. As a user I'd be rebooting it thinking it had crashed by now. I shell in and find apparmor_parser  using a lot of cpu for a long time.

  top - 00:14:01 up 15 min,  2 users,  load average: 5.12, 4.85, 3.21
  Tasks: 202 total,   2 running, 200 sleeping,   0 stopped,   0 zombie
  %Cpu(s): 50.5 us,  0.8 sy,  0.0 ni, 48.5 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
  KiB Mem:   1848024 total,   787400 used,  1060624 free,    54216 buffers
  KiB Swap:    32764 total,        0 used,    32764 free.   579228 cached Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   1970 root      20   0    4976   3652    852 R  99.8  0.2  14:31.04 apparmor_parser
   2596 phablet   20   0    5996   1264    824 R   1.3  0.1   0:08.79 top
    914 root       0 -20    7572    552    396 S   0.7  0.0   0:05.02 mpdecision
     21 root      20   0       0      0      0 S   0.3  0.0   0:00.92 kworker/0:1
    229 root      20   0       0      0      0 S   0.3  0.0   0:00.10 jbd2/mmcblk0p30
    982 root      20   0   38856   1164    868 S   0.3  0.1   0:01.77 adbd
   2570 phablet   20   0   10540   1456    692 S   0.3  0.1   0:02.30 sshd
      1 root      20   0    3884   2648   1068 S   0.0  0.1   0:05.98 init
      2 root      -2   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
      3 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/0

  ... it eventually finished after 18 minutes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/apparmor/+bug/1350598/+subscriptions