yahoo-eng-team team mailing list archive

[Tobias Urdin <1960230@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

This bug is related to resize with the libvirt driver

If you are performing a resize and it fails the
_cleanup_remote_migration() [1] function in the libvirt driver will try
to cleanup the /var/lib/nova/instances/<uuid>_resize directory on the
remote side [2] - if this fails the <uuid>_resize directory will be left
behind and block any future resize attempts.

2021-12-14 14:40:12.535 175177 INFO nova.virt.libvirt.driver
[req-9d3477d4-3bb2-456f-9be6-dce9893b0e95
23d6aa8884ab44ef9f214ad195d273c0 050c556faa5944a8953126c867313770 -
default default] [instance: 99287438-c37b-44b0-834e-55685b6e83eb]
Deletion of
/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize
failed

Then on next resize attempt a long time later

2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10429, in migrate_disk_and_power_off
2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server     os.rename(inst_base, inst_base_resize)
2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server FileExistsError: [Errno 17] File exists: '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb' -> '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize'

This is happens here [3] because os.rename tries to rename the
/var/lib/nova/instances/<uuid> dir to <uuid>_resize that already exists
and fails with FileExistsError.

We should check if the directory exists before trying to rename and
delete it before.

[1] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10773
[2] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10965
[3] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10915

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1960230

Title:
  resize fails with FileExistsError if earlier resize attempt failed to
  cleanup

Status in OpenStack Compute (nova):
  New

Bug description:
  This bug is related to resize with the libvirt driver

  If you are performing a resize and it fails the
  _cleanup_remote_migration() [1] function in the libvirt driver will
  try to cleanup the /var/lib/nova/instances/<uuid>_resize directory on
  the remote side [2] - if this fails the <uuid>_resize directory will
  be left behind and block any future resize attempts.

  2021-12-14 14:40:12.535 175177 INFO nova.virt.libvirt.driver
  [req-9d3477d4-3bb2-456f-9be6-dce9893b0e95
  23d6aa8884ab44ef9f214ad195d273c0 050c556faa5944a8953126c867313770 -
  default default] [instance: 99287438-c37b-44b0-834e-55685b6e83eb]
  Deletion of
  /var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize
  failed

  Then on next resize attempt a long time later

  2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10429, in migrate_disk_and_power_off
  2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server     os.rename(inst_base, inst_base_resize)
  2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server FileExistsError: [Errno 17] File exists: '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb' -> '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize'

  This is happens here [3] because os.rename tries to rename the
  /var/lib/nova/instances/<uuid> dir to <uuid>_resize that already
  exists and fails with FileExistsError.

  We should check if the directory exists before trying to rename and
  delete it before.

  [1] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10773
  [2] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10965
  [3] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10915

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1960230/+subscriptions