yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2045394] Re: Installation of openstack-dashboard fails because of command failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Wesley Hershberger <2045394@xxxxxxxxxxxxxxxxxx>
Date: Thu, 03 Apr 2025 20:55:34 -0000
Reply-to: Bug 2045394 <2045394@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: cloud-archive/epoxy
       Status: Invalid => Triaged

** Changed in: horizon (Ubuntu Oracular)
       Status: Confirmed => In Progress

** Changed in: horizon (Ubuntu Noble)
       Status: Triaged => In Progress

** Changed in: cloud-archive/antelope
       Status: Confirmed => In Progress

** Changed in: cloud-archive/bobcat
       Status: Triaged => In Progress

** Changed in: cloud-archive/caracal
       Status: Triaged => In Progress

** Changed in: cloud-archive/dalmatian
       Status: Triaged => In Progress

** Changed in: cloud-archive/epoxy
       Status: Triaged => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/2045394

Title:
  Installation of openstack-dashboard fails because of command failure

Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive antelope series:
  In Progress
Status in Ubuntu Cloud Archive bobcat series:
  In Progress
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive dalmatian series:
  In Progress
Status in Ubuntu Cloud Archive epoxy series:
  In Progress
Status in OpenStack Dashboard (Horizon):
  Invalid
Status in horizon package in Ubuntu:
  Invalid
Status in horizon source package in Noble:
  In Progress
Status in horizon source package in Oracular:
  In Progress
Status in horizon source package in Plucky:
  Invalid
Status in horizon package in Debian:
  Fix Released

Bug description:
  [ Impact ]
  Starting in OpenStack 2022.2 (Zed), compressing static assets with django-compressor fails around 5-10% of the time (see also the Debian bug [1]). This is done during the Horizon postinst and prevents the package from installing when the race occurs.

  This affects Noble & Oracular, and UCA Zed through Epoxy. Hitting a parse error in pyscss:
  ```
  Looking for 'dashboard/scss/serial_console.scss' in storage
  Traceback (most recent call last):
    File "/usr/share/openstack-dashboard/manage.py", line 25, in <module>
      execute_from_command_line(sys.argv)
    File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
      utility.execute()
    File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute
      self.fetch_command(subcommand).run_from_argv(self.argv)
    File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv
      self.execute(*args, **cmd_options)
    File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
      output = self.handle(*args, **options)
    File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 296, in handle
      self.handle_inner(**options)
    File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 319, in handle_inner
      offline_manifest, block_count, results = self.compress(engine, extensions, verbosity, follow_links, log)
    File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 227, in compress
      raise errors[0]
  django.core.management.base.CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR
  ```

  Debian had little luck in finding the root cause and uploaded a
  workaround that simply retries failed compressions 5 times,
  dramatically reducing the liklihood that the failure breaks
  installation [2].

  django-compressor introduced threaded compression in 3.0 [3], which
  was released in UCA Zed and Ubuntu Noble. I opened [4] in django-
  compressor to troubleshoot. The investigation there points to pyscss
  failing with correct input from django-compressor, indicating that the
  root cause is likely in pyscss.

  Horizon (Flamingo+) has switched from pyscss to libsass [5]. This bug
  is not present in Ubuntu Plucky containing that commit.

  Since the bug is no longer present in devel, I'd like to SRU the
  Debian workaround to the affected Ubuntu & UCA packages without
  knowing the root cause of the race in pyscss.

  [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069097
  [2] https://salsa.debian.org/openstack-team/services/horizon/-/commit/9802ded2ab24cab767d0c91dfcabea117ec5fd9d
  [3] https://github.com/django-compressor/django-compressor/pull/912
  [4] https://github.com/django-compressor/django-compressor/issues/1291
  [5] https://opendev.org/openstack/horizon/commit/283219ab5ef4344e3808cd61b4cc0821886acffb

  [ Test Plan ]

  Install horizon and an affected version of python3-django-compressor
  (>=3.0).

  The Debian bug indicates that this fails between 5-10% of the time.
  Running `dpkg-reconfigure openstack-dashboard` in a loop on ~10
  containers at once usually reproduces it in a couple of minutes, at
  most 30 minutes. The bug can be considered fixed if the loop runs for
  1 hour with no occurances.

  I use the following scripts against my LXD cluster:
  ```
  #!/bin/bash
  #
  # /usr/local/bin/horizon-reconfigure.sh in each test container

  set -ex

  while true; do
      dpkg-reconfigure openstack-dashboard
  done
  ```
  Start and check if running:
  ```
  lxc exec -t "${inst}" -- nohup bash -c "horizon-reconfigure.sh &>> /var/log/dpkg-horizon &"
  lxc exec "${inst}" -- ps -aux | grep horizon-reconfigure
  ```

  For reference, compress can be run with the following to get more output and a traceback on failure:
  ```
  python3 /usr/share/openstack-dashboard/manage.py compress -v 3 --traceback
  ```

  [ Where problems could occur ]

   * The workaround does not fix the race; the package may still fail to
  install if the race occurs 5 times in a row.

   * If the change is wrong/broken, it risks regressions during
  openstack-dashboard install/reconfigure. This is covered by `dpkg-
  reconfigure` in the test plan.

  [ Other info ]

   * As this change affects upgrades between OpenStack versions via the
  UCA, I'm opening MPs against all affected versions, even those that
  are past EOL (Zed+).

   * The patch I've proposed drops the scss change from the Debian
  commit as it is unrelated to the issue described here.

  [ Original Description ]

  We have seen that the puppet integration job with Ubuntu 22.04 and UCA Bobcat consistently fails.
  Looking into the error it seems something is wrong with installation of openstack-dashboard package.

  2023-12-01 07:13:16 +0000 Puppet (err): Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openstack-dashboard' returned 100: Reading package lists...
  Building dependency tree...
  Reading state information...
  The following additional packages will be installed:
    openstack-dashboard-common python3-asgiref python3-csscompressor
    python3-django python3-django-appconf python3-django-compressor
    python3-django-debreach python3-django-horizon python3-django-openstack-auth
    python3-django-pyscss python3-pint python3-pyscss python3-rcssmin
    python3-rjsmin
  Suggested packages:
    bpython3 geoip-database-contrib gettext ipython3 libgdal20
    libsqlite3-mod-spatialite python-django-doc python3-flup python3-mysqldb
    python3-pil python3-selenium python3-sqlite python-django-appconf-doc
    python3-calmjs python-django-debreach-doc
  The following NEW packages will be installed:
    openstack-dashboard openstack-dashboard-common python3-asgiref
    python3-csscompressor python3-django python3-django-appconf
    python3-django-compressor python3-django-debreach python3-django-horizon
    python3-django-openstack-auth python3-django-pyscss python3-pint
    python3-pyscss python3-rcssmin python3-rjsmin
  0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
  Need to get 14.2 MB of archives.
  After this operation, 92.3 MB of additional disk space will be used.
  WARNING: The following packages cannot be authenticated!
    python3-rcssmin python3-rjsmin python3-django-compressor python3-pyscss
    openstack-dashboard-common python3-django-openstack-auth
    python3-django-horizon openstack-dashboard
  Authentication warning overridden.
  Get:1 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rcssmin amd64 1:1.1.0-2~cloud0 [17.5 kB]
  Get:2 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-asgiref all 3.5.0-1 [23.9 kB]
  Get:3 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rjsmin amd64 1.2.0+dfsg1-2build2~cloud0 [16.6 kB]
  Get:4 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-csscompressor all 0.9.5-2 [57.3 kB]
  Get:5 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-compressor all 4.0-2ubuntu1~cloud0 [71.8 kB]
  Get:6 https://mirror.gra1.ovh.opendev.org/ubuntu jammy-updates/main amd64 python3-django all 2:3.2.12-2ubuntu1.9 [3020 kB]
  Get:7 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-pyscss amd64 1.4.0-3~cloud0 [97.9 kB]
  Get:8 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-appconf all 1.0.3-1 [7620 B]
  Get:9 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard-common all 4:23.3.0-0ubuntu1.1~cloud0 [27.0 kB]
  Get:10 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-debreach all 2.0.1-2 [5482 B]
  Get:11 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-openstack-auth all 4:23.3.0-0ubuntu1.1~cloud0 [54.9 kB]
  Get:12 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-pyscss all 2.0.2-10 [9320 B]
  Get:13 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-horizon all 4:23.3.0-0ubuntu1.1~cloud0 [10.6 MB]
  Get:14 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-pint all 0.18-1 [156 kB]
  Get:15 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard all 4:23.3.0-0ubuntu1.1~cloud0 [20.2 kB]
  debconf: delaying package configuration, since apt-utils is not installed
  Fetched 14.2 MB in 7s (2004 kB/s)
  Selecting previously unselected package python3-asgiref.
  (Reading database ...
  (Reading database ... 5%
  (Reading database ... 10%
  (Reading database ... 15%
  (Reading database ... 20%
  (Reading database ... 25%
  (Reading database ... 30%
  (Reading database ... 35%
  (Reading database ... 40%
  (Reading database ... 45%
  (Reading database ... 50%
  (Reading database ... 55%
  (Reading database ... 60%
  (Reading database ... 65%
  (Reading database ... 70%
  (Reading database ... 75%
  (Reading database ... 80%
  (Reading database ... 85%
  (Reading database ... 90%
  (Reading database ... 95%
  (Reading database ... 100%
  (Reading database ... 87302 files and directories currently installed.)
  Preparing to unpack .../00-python3-asgiref_3.5.0-1_all.deb ...
  Unpacking python3-asgiref (3.5.0-1) ...
  Selecting previously unselected package python3-csscompressor.
  Preparing to unpack .../01-python3-csscompressor_0.9.5-2_all.deb ...
  Unpacking python3-csscompressor (0.9.5-2) ...
  Selecting previously unselected package python3-django.
  Preparing to unpack .../02-python3-django_2%3a3.2.12-2ubuntu1.9_all.deb ...
  Unpacking python3-django (2:3.2.12-2ubuntu1.9) ...
  Selecting previously unselected package python3-django-appconf.
  Preparing to unpack .../03-python3-django-appconf_1.0.3-1_all.deb ...
  Unpacking python3-django-appconf (1.0.3-1) ...
  Selecting previously unselected package python3-rcssmin.
  Preparing to unpack .../04-python3-rcssmin_1%3a1.1.0-2~cloud0_amd64.deb ...
  Unpacking python3-rcssmin (1:1.1.0-2~cloud0) ...
  Selecting previously unselected package python3-rjsmin.
  Preparing to unpack .../05-python3-rjsmin_1.2.0+dfsg1-2build2~cloud0_amd64.deb ...
  Unpacking python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ...
  Selecting previously unselected package python3-django-compressor.
  Preparing to unpack .../06-python3-django-compressor_4.0-2ubuntu1~cloud0_all.deb ...
  Unpacking python3-django-compressor (4.0-2ubuntu1~cloud0) ...
  Selecting previously unselected package python3-django-debreach.
  Preparing to unpack .../07-python3-django-debreach_2.0.1-2_all.deb ...
  Unpacking python3-django-debreach (2.0.1-2) ...
  Selecting previously unselected package python3-pyscss.
  Preparing to unpack .../08-python3-pyscss_1.4.0-3~cloud0_amd64.deb ...
  Unpacking python3-pyscss (1.4.0-3~cloud0) ...
  Selecting previously unselected package python3-django-pyscss.
  Preparing to unpack .../09-python3-django-pyscss_2.0.2-10_all.deb ...
  Unpacking python3-django-pyscss (2.0.2-10) ...
  Selecting previously unselected package python3-pint.
  Preparing to unpack .../10-python3-pint_0.18-1_all.deb ...
  Unpacking python3-pint (0.18-1) ...
  Selecting previously unselected package openstack-dashboard-common.
  Preparing to unpack .../11-openstack-dashboard-common_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ...
  Unpacking openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ...
  Selecting previously unselected package python3-django-openstack-auth.
  Preparing to unpack .../12-python3-django-openstack-auth_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ...
  Unpacking python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ...
  Selecting previously unselected package python3-django-horizon.
  Preparing to unpack .../13-python3-django-horizon_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ...
  Unpacking python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ...
  Selecting previously unselected package openstack-dashboard.
  Preparing to unpack .../14-openstack-dashboard_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ...
  Unpacking openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ...
  Setting up python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ...
  Setting up python3-pyscss (1.4.0-3~cloud0) ...
  Setting up python3-rcssmin (1:1.1.0-2~cloud0) ...
  Setting up python3-csscompressor (0.9.5-2) ...
  Setting up python3-pint (0.18-1) ...
  Setting up openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ...
  Setting up python3-asgiref (3.5.0-1) ...
  Setting up python3-django (2:3.2.12-2ubuntu1.9) ...
  Setting up python3-django-appconf (1.0.3-1) ...
  Setting up python3-django-debreach (2.0.1-2) ...
  Setting up python3-django-compressor (4.0-2ubuntu1~cloud0) ...
  Setting up python3-django-pyscss (2.0.2-10) ...
  Setting up python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ...
  Setting up python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ...
  update-alternatives: using /usr/lib/python3/dist-packages/openstack_dashboard to provide /usr/share/openstack-dashboard/openstack_dashboard (openstack_dashboard) in auto mode
  Setting up openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ...
  Adding system user `horizon' (UID 133) ...
  Adding new user `horizon' (UID 133) with group `horizon' ...
  Not creating home directory `/var/lib/openstack-dashboard'.
  Collecting and compressing static assets...
  CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR
  dpkg: error processing package openstack-dashboard (--configure):
   installed openstack-dashboard package post-installation script subprocess returned error exit status 1
  Errors were encountered while processing:
   openstack-dashboard
  E: Sub-process /usr/bin/dpkg returned an error code (1)
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2045394/+subscriptions