← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #87223

[Bug 1944611] [NEW] Apt lock race on Oracle

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When trying to do any kind of apt operation via cloud-init on Oracle,
the operation often fails because snap seeding is holding the apt lock:

cloud-init.log:
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
2021-06-15 21:32:31,809 - util.py[DEBUG]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 885, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 161, in handle
    install_drivers(cfg['drivers'], cloud.distro.install_packages)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 115, in install_drivers
    pkg_install_func(['ubuntu-drivers-common'])
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 111, in install_packages
    self.update_package_sources()
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 206, in update_package_sources
    ["update"], freq=PER_INSTANCE)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 202, in package_command
    args=(cmd,), kwargs={'env': e, 'capture': False})
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
    cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']

cloud-init-output.log:
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
Cloud-init v. 21.1-19-gbad84ad4-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Jun 2021 21:32:29 +0000. Up 36.68 seconds.
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed

apt/history.log:
Start-Date: 2021-06-15  21:32:39
Commandline: apt install -o Dpkg::Options::=--force-confold -y dpkg-sig
Requested-By: snap_daemon (584788)
Install: libconfig-file-perl:amd64 (1.50-3, automatic), dpkg-sig:amd64 (0.13.1+nmu4)
End-Date: 2021-06-15  21:32:40

I've also seen another snap entry in a separate log:
Start-Date: 2021-09-21  21:35:53
Commandline: apt --allow-downgrades install -o Dpkg::Options::=--force-confold -y /var/lib/oracle-cloud-agent/plugins/unifiedmonitoring/temp-unified-monitoring.deb
Requested-By: snap_daemon (584788)
Install: unified-monitoring-agent:amd64 (2.5.8-0)
End-Date: 2021-09-21  21:36:12

This is most easily observed when launching an instance with the following cloud-config:
#cloud-config
drivers:
  nvidia:
    license-accepted: true

** Affects: cloud-init
     Importance: High
         Status: Triaged

** Changed in: cloud-init
       Status: New => Triaged

** Changed in: cloud-init
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1944611

Title:
  Apt lock race on Oracle

Status in cloud-init:
  Triaged

Bug description:
  When trying to do any kind of apt operation via cloud-init on Oracle,
  the operation often fails because snap seeding is holding the apt
  lock:

  cloud-init.log:
  2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
  2021-06-15 21:32:31,809 - util.py[DEBUG]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 885, in _run_modules
      freq=freq)
    File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
      return self._runners.run(name, functor, args, freq, clear_on_fail)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 161, in handle
      install_drivers(cfg['drivers'], cloud.distro.install_packages)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 115, in install_drivers
      pkg_install_func(['ubuntu-drivers-common'])
    File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 111, in install_packages
      self.update_package_sources()
    File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 206, in update_package_sources
      ["update"], freq=PER_INSTANCE)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 202, in package_command
      args=(cmd,), kwargs={'env': e, 'capture': False})
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
      ret = func(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
      cmd=args)
  cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
  Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']

  cloud-init-output.log:
  Reading package lists...
  E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
  E: Unable to lock directory /var/lib/apt/lists/
  Cloud-init v. 21.1-19-gbad84ad4-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Jun 2021 21:32:29 +0000. Up 36.68 seconds.
  2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed

  apt/history.log:
  Start-Date: 2021-06-15  21:32:39
  Commandline: apt install -o Dpkg::Options::=--force-confold -y dpkg-sig
  Requested-By: snap_daemon (584788)
  Install: libconfig-file-perl:amd64 (1.50-3, automatic), dpkg-sig:amd64 (0.13.1+nmu4)
  End-Date: 2021-06-15  21:32:40

  I've also seen another snap entry in a separate log:
  Start-Date: 2021-09-21  21:35:53
  Commandline: apt --allow-downgrades install -o Dpkg::Options::=--force-confold -y /var/lib/oracle-cloud-agent/plugins/unifiedmonitoring/temp-unified-monitoring.deb
  Requested-By: snap_daemon (584788)
  Install: unified-monitoring-agent:amd64 (2.5.8-0)
  End-Date: 2021-09-21  21:36:12

  This is most easily observed when launching an instance with the following cloud-config:
  #cloud-config
  drivers:
    nvidia:
      license-accepted: true

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1944611/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next