yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2063345] [NEW] Instance state is error when resizing or migrating

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Khoi <2063345@xxxxxxxxxxxxxxxxxx>
Date: Wed, 24 Apr 2024 14:33:55 -0000
Reply-to: Bug 2063345 <2063345@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Openstack Details:

Base OS - Ubuntu 20.04 LTS

HW - 3 Controller and multi computes

Openstack Version - Yoga(Kolla-Ansible)

Cinder Backend: Powerstore 5000T using ISCSI

Hello, I have this error when I resize or migrate instances

2024-04-12 09:49:35.284 7 WARNING os_brick.exception [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] Flushing 36000d3100578ec0000000000000005ba failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -f 36000d3100578ec0000000000000005ba
Exit code: 1
Stdout: ''
Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Traceback (most recent call last):
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 10415, in _error_out_instance_on_exception
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] yield
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 5712, in _resize_instance
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] disk_info = self.driver.migrate_disk_and_power_off(
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 11252, in migrate_disk_and_power_off
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._disconnect_volume(context, connection_info, instance)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1971, in _disconnect_volume
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] vol_driver.disconnect_volume(
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.connector.disconnect_volume(
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 382, in change_encrypted2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] res = func(**call_args)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 855, in disconnect_volume
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self._cleanup_connection(connection_properties, force=force,
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 904, in _cleanup_connection
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] multipath_name = self._linuxscsi.remove_connection(
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 310, in remove_connection
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.flush_multipath_device(multipath_name)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 378, in flush_multipath_device
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._execute('multipath', '-f', device_map_name, run_as_root=True,
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/executor.py", line 53, in _execute
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] result = self.__execute(*args, **kwargs)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return execute_root(*cmd, **kwargs)
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self.channel.remote_call(name, args, kwargs,
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] raise exc_type(*result[2])
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Command: multipath -f 36000d3100578ec0000000000000005ba
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Exit code: 1
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stdout: ''
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380]

This happen randomly, so I cannot procedure it.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2063345

Title:
  Instance state is error  when resizing or  migrating

Status in OpenStack Compute (nova):
  New

Bug description:
  Openstack Details:

  Base OS - Ubuntu 20.04 LTS

  HW - 3 Controller and multi computes

  Openstack Version - Yoga(Kolla-Ansible)

  Cinder Backend: Powerstore 5000T using ISCSI

  Hello, I have this error when I resize or migrate instances

  2024-04-12 09:49:35.284 7 WARNING os_brick.exception [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] Flushing 36000d3100578ec0000000000000005ba failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Command: multipath -f 36000d3100578ec0000000000000005ba
  Exit code: 1
  Stdout: ''
  Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Traceback (most recent call last):
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 10415, in _error_out_instance_on_exception
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] yield
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 5712, in _resize_instance
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] disk_info = self.driver.migrate_disk_and_power_off(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 11252, in migrate_disk_and_power_off
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._disconnect_volume(context, connection_info, instance)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1971, in _disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] vol_driver.disconnect_volume(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.connector.disconnect_volume(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 382, in change_encrypted2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] res = func(**call_args)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 855, in disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self._cleanup_connection(connection_properties, force=force,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 904, in _cleanup_connection
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] multipath_name = self._linuxscsi.remove_connection(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 310, in remove_connection
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.flush_multipath_device(multipath_name)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 378, in flush_multipath_device
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._execute('multipath', '-f', device_map_name, run_as_root=True,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/executor.py", line 53, in _execute
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] result = self.__execute(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return execute_root(*cmd, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self.channel.remote_call(name, args, kwargs,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] raise exc_type(*result[2])
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Command: multipath -f 36000d3100578ec0000000000000005ba
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Exit code: 1
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stdout: ''
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380]

  This happen randomly, so I cannot procedure it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2063345/+subscriptions
Follow ups

[Bug 2063345] Re: Instance state is error when resizing or migrating
From: Khoi, 2024-04-26