yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1986947] [NEW] cc_grub_dpkg: race condition on dpkg-set-selections lock

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Chad Smith <1986947@xxxxxxxxxxxxxxxxxx>
Date: Thu, 18 Aug 2022 13:35:39 -0000
Reply-to: Bug 1986947 <1986947@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Intermittent failure found during jenkins test runs:

We probably want a couple quick retries if lock is held. 
https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_grub_dpkg.py#L155


'2022-08-18 06:54:21,464 - handlers.py[DEBUG]: start: '
 'modules-config/config-grub-dpkg: running config-grub-dpkg with frequency '
 'once-per-instance\n'
 '2022-08-18 06:54:21,464 - util.py[DEBUG]: Writing to '
 '/var/lib/cloud/instances/32493477-976f-924a-b363-1ddcabc724d9/sem/config_grub_dpkg '
 '- wb: [644] 25 bytes\n'
 '2022-08-18 06:54:21,464 - helpers.py[DEBUG]: Running config-grub-dpkg using '
 'lock (<FileLock using file '
 "'/var/lib/cloud/instances/32493477-976f-924a-b363-1ddcabc724d9/sem/config_grub_dpkg'>)\n"
 "2022-08-18 06:54:21,464 - subp.py[DEBUG]: Running command ['grub-probe', "
 "'-t', 'disk', '/boot'] with allowed return codes [0] (shell=False, "
 'capture=True)\n'
 "2022-08-18 06:54:21,710 - subp.py[DEBUG]: Running command ['udevadm', "
 "'info', '--root', '--query=symlink', '/dev/sda'] with allowed return codes "
 '[0] (shell=False, capture=True)\n'
 '2022-08-18 06:54:21,713 - cc_grub_dpkg.py[DEBUG]: considering these device '
 'symlinks: '
 '/dev/disk/azure/root,/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751,/dev/disk/by-id/wwn-0x6002248026fe88a16909c01e11d36751,/dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:0:0,/dev/disk/cloud/azure_root\n'
 '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: filtered to these '
 'disk/by-id symlinks: '
 '/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751,/dev/disk/by-id/wwn-0x6002248026fe88a16909c01e11d36751\n'
 '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: selected '
 '/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751\n'
 '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: Setting grub '
 'debconf-set-selections with '
 "'/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751','false'\n"
 '2022-08-18 06:54:21,714 - subp.py[DEBUG]: Running command '
 "['debconf-set-selections'] with allowed return codes [0] (shell=False, "
 'capture=True)\n'
 '2022-08-18 06:54:21,817 - util.py[WARNING]: Failed to run '
 'debconf-set-selections for grub-dpkg\n'
 '2022-08-18 06:54:21,823 - util.py[DEBUG]: Failed to run '
 'debconf-set-selections for grub-dpkg\n'
 'Traceback (most recent call last):\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_grub_dpkg.py", '
 'line 155, in handle\n'
 '    subp.subp(["debconf-set-selections"], dconf_sel)\n'
 '  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 336, in '
 'subp\n'
 '    stdout=out, stderr=err, exit_code=rc, cmd=args\n'
 'cloudinit.subp.ProcessExecutionError: Unexpected error while running '
 'command.\n'
 "Command: ['debconf-set-selections']\n"
 'Exit code: 1\n'
 'Reason: -\n'
 'Stdout: \n'
 'Stderr: debconf: DbDriver "config": /var/cache/debconf/config.dat is locked '
 'by another process: Resource temporarily unavailable\n'

** Affects: cloud-init
     Importance: Medium
         Status: Triaged


** Tags: bitesize

** Changed in: cloud-init
   Importance: Undecided => Medium

** Tags added: bitesize

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1986947

Title:
  cc_grub_dpkg: race condition on dpkg-set-selections lock

Status in cloud-init:
  Triaged

Bug description:
  Intermittent failure found during jenkins test runs:

  We probably want a couple quick retries if lock is held. 
  https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_grub_dpkg.py#L155

  
  '2022-08-18 06:54:21,464 - handlers.py[DEBUG]: start: '
   'modules-config/config-grub-dpkg: running config-grub-dpkg with frequency '
   'once-per-instance\n'
   '2022-08-18 06:54:21,464 - util.py[DEBUG]: Writing to '
   '/var/lib/cloud/instances/32493477-976f-924a-b363-1ddcabc724d9/sem/config_grub_dpkg '
   '- wb: [644] 25 bytes\n'
   '2022-08-18 06:54:21,464 - helpers.py[DEBUG]: Running config-grub-dpkg using '
   'lock (<FileLock using file '
   "'/var/lib/cloud/instances/32493477-976f-924a-b363-1ddcabc724d9/sem/config_grub_dpkg'>)\n"
   "2022-08-18 06:54:21,464 - subp.py[DEBUG]: Running command ['grub-probe', "
   "'-t', 'disk', '/boot'] with allowed return codes [0] (shell=False, "
   'capture=True)\n'
   "2022-08-18 06:54:21,710 - subp.py[DEBUG]: Running command ['udevadm', "
   "'info', '--root', '--query=symlink', '/dev/sda'] with allowed return codes "
   '[0] (shell=False, capture=True)\n'
   '2022-08-18 06:54:21,713 - cc_grub_dpkg.py[DEBUG]: considering these device '
   'symlinks: '
   '/dev/disk/azure/root,/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751,/dev/disk/by-id/wwn-0x6002248026fe88a16909c01e11d36751,/dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:0:0,/dev/disk/cloud/azure_root\n'
   '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: filtered to these '
   'disk/by-id symlinks: '
   '/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751,/dev/disk/by-id/wwn-0x6002248026fe88a16909c01e11d36751\n'
   '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: selected '
   '/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751\n'
   '2022-08-18 06:54:21,714 - cc_grub_dpkg.py[DEBUG]: Setting grub '
   'debconf-set-selections with '
   "'/dev/disk/by-id/scsi-36002248026fe88a16909c01e11d36751','false'\n"
   '2022-08-18 06:54:21,714 - subp.py[DEBUG]: Running command '
   "['debconf-set-selections'] with allowed return codes [0] (shell=False, "
   'capture=True)\n'
   '2022-08-18 06:54:21,817 - util.py[WARNING]: Failed to run '
   'debconf-set-selections for grub-dpkg\n'
   '2022-08-18 06:54:21,823 - util.py[DEBUG]: Failed to run '
   'debconf-set-selections for grub-dpkg\n'
   'Traceback (most recent call last):\n'
   '  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_grub_dpkg.py", '
   'line 155, in handle\n'
   '    subp.subp(["debconf-set-selections"], dconf_sel)\n'
   '  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 336, in '
   'subp\n'
   '    stdout=out, stderr=err, exit_code=rc, cmd=args\n'
   'cloudinit.subp.ProcessExecutionError: Unexpected error while running '
   'command.\n'
   "Command: ['debconf-set-selections']\n"
   'Exit code: 1\n'
   'Reason: -\n'
   'Stdout: \n'
   'Stderr: debconf: DbDriver "config": /var/cache/debconf/config.dat is locked '
   'by another process: Resource temporarily unavailable\n'

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1986947/+subscriptions
Follow ups

[Bug 1986947] Re: cc_grub_dpkg: race condition on dpkg-set-selections lock
From: James Falcon, 2023-05-12