yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87223
[Bug 1944611] [NEW] Apt lock race on Oracle
Public bug reported:
When trying to do any kind of apt operation via cloud-init on Oracle,
the operation often fails because snap seeding is holding the apt lock:
cloud-init.log:
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
2021-06-15 21:32:31,809 - util.py[DEBUG]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 885, in _run_modules
freq=freq)
File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 161, in handle
install_drivers(cfg['drivers'], cloud.distro.install_packages)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 115, in install_drivers
pkg_install_func(['ubuntu-drivers-common'])
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 111, in install_packages
self.update_package_sources()
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 206, in update_package_sources
["update"], freq=PER_INSTANCE)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 202, in package_command
args=(cmd,), kwargs={'env': e, 'capture': False})
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']
cloud-init-output.log:
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
Cloud-init v. 21.1-19-gbad84ad4-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Jun 2021 21:32:29 +0000. Up 36.68 seconds.
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
apt/history.log:
Start-Date: 2021-06-15 21:32:39
Commandline: apt install -o Dpkg::Options::=--force-confold -y dpkg-sig
Requested-By: snap_daemon (584788)
Install: libconfig-file-perl:amd64 (1.50-3, automatic), dpkg-sig:amd64 (0.13.1+nmu4)
End-Date: 2021-06-15 21:32:40
I've also seen another snap entry in a separate log:
Start-Date: 2021-09-21 21:35:53
Commandline: apt --allow-downgrades install -o Dpkg::Options::=--force-confold -y /var/lib/oracle-cloud-agent/plugins/unifiedmonitoring/temp-unified-monitoring.deb
Requested-By: snap_daemon (584788)
Install: unified-monitoring-agent:amd64 (2.5.8-0)
End-Date: 2021-09-21 21:36:12
This is most easily observed when launching an instance with the following cloud-config:
#cloud-config
drivers:
nvidia:
license-accepted: true
** Affects: cloud-init
Importance: High
Status: Triaged
** Changed in: cloud-init
Status: New => Triaged
** Changed in: cloud-init
Importance: Undecided => High
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1944611
Title:
Apt lock race on Oracle
Status in cloud-init:
Triaged
Bug description:
When trying to do any kind of apt operation via cloud-init on Oracle,
the operation often fails because snap seeding is holding the apt
lock:
cloud-init.log:
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
2021-06-15 21:32:31,809 - util.py[DEBUG]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 885, in _run_modules
freq=freq)
File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 161, in handle
install_drivers(cfg['drivers'], cloud.distro.install_packages)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 115, in install_drivers
pkg_install_func(['ubuntu-drivers-common'])
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 111, in install_packages
self.update_package_sources()
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 206, in update_package_sources
["update"], freq=PER_INSTANCE)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 202, in package_command
args=(cmd,), kwargs={'env': e, 'capture': False})
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']
cloud-init-output.log:
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
Cloud-init v. 21.1-19-gbad84ad4-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Jun 2021 21:32:29 +0000. Up 36.68 seconds.
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
apt/history.log:
Start-Date: 2021-06-15 21:32:39
Commandline: apt install -o Dpkg::Options::=--force-confold -y dpkg-sig
Requested-By: snap_daemon (584788)
Install: libconfig-file-perl:amd64 (1.50-3, automatic), dpkg-sig:amd64 (0.13.1+nmu4)
End-Date: 2021-06-15 21:32:40
I've also seen another snap entry in a separate log:
Start-Date: 2021-09-21 21:35:53
Commandline: apt --allow-downgrades install -o Dpkg::Options::=--force-confold -y /var/lib/oracle-cloud-agent/plugins/unifiedmonitoring/temp-unified-monitoring.deb
Requested-By: snap_daemon (584788)
Install: unified-monitoring-agent:amd64 (2.5.8-0)
End-Date: 2021-09-21 21:36:12
This is most easily observed when launching an instance with the following cloud-config:
#cloud-config
drivers:
nvidia:
license-accepted: true
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1944611/+subscriptions
Follow ups