yahoo-eng-team team mailing list archive

[Macmillan <2117122@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I deployed openstack in aarch64 machine, and `/usr/bin/unattended-
upgrade` trigered containerd restarting then nova_libvirt container
restarted. but it failed in "Deleting /etc/libvirt/secrets" :

```
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /etc/kolla/defaults/etc/libvirt/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Copying permissions from /etc/kolla/defaults/etc/libvirt/libvirtd.conf onto /etc/libvirt/libvirtd.conf
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /etc/kolla/defaults/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Copying permissions from /etc/kolla/defaults/etc/libvirt/qemu.conf onto /etc/libvirt/qemu.conf
INFO:__main__:Deleting /etc/libvirt/secrets
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 575, in main
    execute_config_strategy(config)
  File "/usr/local/bin/kolla_set_configs", line 539, in execute_config_strategy
    handle_defaults(config)
  File "/usr/local/bin/kolla_set_configs", line 530, in handle_defaults
    remove_or_restore_configs(state)
  File "/usr/local/bin/kolla_set_configs", line 474, in remove_or_restore_configs
    config_file.copy()
  File "/usr/local/bin/kolla_set_configs", line 153, in copy
    self._delete_path(dest)
  File "/usr/local/bin/kolla_set_configs", line 110, in _delete_path
    shutil.rmtree(path)
  File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 16] Device or resource busy: '/etc/libvirt/secrets'
```

'/etc/libvirt/secrets' in host is
'/var/lib/docker/volumes/libvirtd/_data', here is lsof result:


```
COMMAND       PID     USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
qemu-syst 3775877    42436   43u   REG    8,2 67108864 4849771 /var/lib/docker/volumes/libvirtd/_data/qemu/nvram/instance-00003722_VARS.fd
```

my temporary method is trigger `kolla-ansible reconfigure`, everything
is ok.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2117122

Title:
  nova_libvirt start failed because deleting inused nvram: Device or
  resource busy /etc/libvirt/secrets

Status in OpenStack Compute (nova):
  New

Bug description:
  I deployed openstack in aarch64 machine, and `/usr/bin/unattended-
  upgrade` trigered containerd restarting then nova_libvirt container
  restarted. but it failed in "Deleting /etc/libvirt/secrets" :

  ```
  + sudo -E kolla_set_configs
  INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
  INFO:__main__:Validating config file
  INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
  INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
  INFO:__main__:Copying /etc/kolla/defaults/etc/libvirt/libvirtd.conf to /etc/libvirt/libvirtd.conf
  INFO:__main__:Copying permissions from /etc/kolla/defaults/etc/libvirt/libvirtd.conf onto /etc/libvirt/libvirtd.conf
  INFO:__main__:Deleting /etc/libvirt/qemu.conf
  INFO:__main__:Copying /etc/kolla/defaults/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
  INFO:__main__:Copying permissions from /etc/kolla/defaults/etc/libvirt/qemu.conf onto /etc/libvirt/qemu.conf
  INFO:__main__:Deleting /etc/libvirt/secrets
  ERROR:__main__:Unexpected error:
  Traceback (most recent call last):
    File "/usr/local/bin/kolla_set_configs", line 575, in main
      execute_config_strategy(config)
    File "/usr/local/bin/kolla_set_configs", line 539, in execute_config_strategy
      handle_defaults(config)
    File "/usr/local/bin/kolla_set_configs", line 530, in handle_defaults
      remove_or_restore_configs(state)
    File "/usr/local/bin/kolla_set_configs", line 474, in remove_or_restore_configs
      config_file.copy()
    File "/usr/local/bin/kolla_set_configs", line 153, in copy
      self._delete_path(dest)
    File "/usr/local/bin/kolla_set_configs", line 110, in _delete_path
      shutil.rmtree(path)
    File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
      onerror(os.rmdir, path, sys.exc_info())
    File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
      os.rmdir(path, dir_fd=dir_fd)
  OSError: [Errno 16] Device or resource busy: '/etc/libvirt/secrets'
  ```

  '/etc/libvirt/secrets' in host is
  '/var/lib/docker/volumes/libvirtd/_data', here is lsof result:

  
  ```
  COMMAND       PID     USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
  qemu-syst 3775877    42436   43u   REG    8,2 67108864 4849771 /var/lib/docker/volumes/libvirtd/_data/qemu/nvram/instance-00003722_VARS.fd
  ```

  my temporary method is trigger `kolla-ansible reconfigure`, everything
  is ok.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2117122/+subscriptions