yahoo-eng-team team mailing list archive

[Khoi <2063345@xxxxxxxxxxxxxxxxxx>]

** Project changed: nova => os-brick

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2063345

Title:
  Instance state is error  when resizing or  migrating

Status in os-brick:
  New

Bug description:
  Openstack Details:

  Base OS - Ubuntu 20.04 LTS

  HW - 3 Controller and multi computes

  Openstack Version - Yoga(Kolla-Ansible)

  Cinder Backend: Powerstore 5000T using ISCSI

  Hello, I have this error when I resize or migrate instances

  2024-04-12 09:49:35.284 7 WARNING os_brick.exception [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] Flushing 36000d3100578ec0000000000000005ba failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [req-ea4c2c71-d1f1-45ab-82b5-9b384bc93101 5875e342bc6c39052c9b291df7da1a4f4960cb5578defbadb0b432f5b5dbd087 a14e68c02f6344cb8acdd279dce5aab1 - default default] [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Command: multipath -f 36000d3100578ec0000000000000005ba
  Exit code: 1
  Stdout: ''
  Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Traceback (most recent call last):
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 10415, in _error_out_instance_on_exception
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] yield
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 5712, in _resize_instance
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] disk_info = self.driver.migrate_disk_and_power_off(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 11252, in migrate_disk_and_power_off
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._disconnect_volume(context, connection_info, instance)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1971, in _disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] vol_driver.disconnect_volume(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.connector.disconnect_volume(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 169, in trace_logging_wrapper
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return f(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 382, in change_encrypted2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] res = func(**call_args)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 855, in disconnect_volume
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self._cleanup_connection(connection_properties, force=force,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 904, in _cleanup_connection
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] multipath_name = self._linuxscsi.remove_connection(
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 310, in remove_connection
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self.flush_multipath_device(multipath_name)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 378, in flush_multipath_device
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] self._execute('multipath', '-f', device_map_name, run_as_root=True,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/executor.py", line 53, in _execute
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] result = self.__execute(*args, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return execute_root(*cmd, **kwargs)
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] return self.channel.remote_call(name, args, kwargs,
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] raise exc_type(*result[2])
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Command: multipath -f 36000d3100578ec0000000000000005ba
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Exit code: 1
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stdout: ''
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380] Stderr: 'Apr 12 09:48:55 | 36000d3100578ec0000000000000005ba: map in use\n'
  2024-04-12 09:49:35.309 7 ERROR nova.compute.manager [instance: 17a71500-ebb8-42cf-82b8-0d380929d380]

  This happen randomly, so I cannot procedure it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/os-brick/+bug/2063345/+subscriptions