yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2062459] [NEW] VM's Won't Start Again After CLI/SDK Driven Shutdown - Exception during message handling: os_brick.exception.ExceptionChainer

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Noel Ashford <2062459@xxxxxxxxxxxxxxxxxx>
Date: Fri, 19 Apr 2024 04:23:59 -0000
Reply-to: Bug 2062459 <2062459@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

This was a brand new healthy VM. On Reboot, I noticed it and all other
VM's would not come up and as i run kolla ansible in containers, i
assumed it to be a race condition corrupting it. I created all new VM's
(that is fine) they started up and even shut down a few times ok, i
tried via SDK and also CLI and then then wont start again.... (no
shutdown involved) so this must be a nova or iscsi issue. Thoughts?

==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.073 8 INFO nova.virt.libvirt.driver [-] [instance: 712454ec-3ab3-4648-97c5-ede2edce5c9c] Instance destroyed successfully.
2024-04-19 00:12:23.087 8 INFO os_vif [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:cc:30:e0,bridge_name='qbrc93e0089-32',has_traffic_filtering=True,id=c93e0089-3236-4271-bd61-90f2cd47afdd,network=Network(53acdf7a-617f-4ab5-a62e-0cb96936d619),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc93e0089-32')
2024-04-19 00:12:23.090 8 WARNING os_brick.initiator.connectors.base [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Service needs to call os_brick.setup() before connecting volumes, if it doesn't it will break on the next release
2024-04-19 00:12:23.140 8 WARNING os_brick.exception [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Disconnect from 192.168.5.1:3260 iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-04-19 00:12:23.140 8 WARNING os_brick.initiator.connectors.iscsi [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] There were errors removing set(), leftovers may remain in the system

==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.352 8 INFO nova.compute.manager [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] [instance: 712454ec-3ab3-4648-97c5-ede2edce5c9c] Successfully reverted task state from powering-on on failure for instance.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Exception during message handling: os_brick.exception.ExceptionChainer:
Chained Exception #1
        Traceback (most recent call last):
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1177, in _disconnect_connection
            self._disconnect_from_iscsi_portal(props)
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1159, in _disconnect_from_iscsi_portal
            self._run_iscsiadm(connection_properties, ("--logout",),
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1025, in _run_iscsiadm
            (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line 55, in _execute
            result = self.__execute(*args, **kwargs)
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
            return execute_root(*cmd, **kwargs)
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
            return self.channel.remote_call(name, args, kwargs,
          File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
            raise exc_type(*result[2])
        oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
        Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p 192.168.5.1:3260 --logout
        Exit code: 32
        Stdout: 'Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260]\n'
        Stderr: 'iscsiadm: Could not logout of [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 157, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 3424, in start_instance
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 3392, in _power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.driver.power_on(context, instance,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 4136, in power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3955, in _hard_reboot
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.destroy(context, instance, network_info, destroy_disks=False,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1565, in destroy
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.cleanup(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1635, in cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._cleanup(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1690, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception() as ctxt:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1686, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2017, in _disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     vol_driver.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.connector.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 173, in trace_logging_wrapper
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py", line 78, in inner
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 405, in change_encrypted
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     res = func(**call_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 891, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._cleanup_connection(connection_properties, force=force,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 969, in _cleanup_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise exc  # type: ignore
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server os_brick.exception.ExceptionChainer:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Chained Exception #1
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Traceback (most recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1177, in _disconnect_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           self._disconnect_from_iscsi_portal(props)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1159, in _disconnect_from_iscsi_portal
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           self._run_iscsiadm(connection_properties, ("--logout",),
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1025, in _run_iscsiadm
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line 55, in _execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           result = self.__execute(*args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           return execute_root(*cmd, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           return self.channel.remote_call(name, args, kwargs,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           raise exc_type(*result[2])
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p 192.168.5.1:3260 --logout
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Exit code: 32
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Stdout: 'Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260]\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Stderr: 'iscsiadm: Could not logout of [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2062459

Title:
  VM's Won't Start Again After CLI/SDK Driven Shutdown - Exception
  during message handling: os_brick.exception.ExceptionChainer

Status in OpenStack Compute (nova):
  New

Bug description:
  This was a brand new healthy VM. On Reboot, I noticed it and all other
  VM's would not come up and as i run kolla ansible in containers, i
  assumed it to be a race condition corrupting it. I created all new
  VM's (that is fine) they started up and even shut down a few times ok,
  i tried via SDK and also CLI and then then wont start again.... (no
  shutdown involved) so this must be a nova or iscsi issue. Thoughts?

  ==> /var/log/kolla/nova/nova-compute.log <==
  2024-04-19 00:12:23.073 8 INFO nova.virt.libvirt.driver [-] [instance: 712454ec-3ab3-4648-97c5-ede2edce5c9c] Instance destroyed successfully.
  2024-04-19 00:12:23.087 8 INFO os_vif [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:cc:30:e0,bridge_name='qbrc93e0089-32',has_traffic_filtering=True,id=c93e0089-3236-4271-bd61-90f2cd47afdd,network=Network(53acdf7a-617f-4ab5-a62e-0cb96936d619),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc93e0089-32')
  2024-04-19 00:12:23.090 8 WARNING os_brick.initiator.connectors.base [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Service needs to call os_brick.setup() before connecting volumes, if it doesn't it will break on the next release
  2024-04-19 00:12:23.140 8 WARNING os_brick.exception [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Disconnect from 192.168.5.1:3260 iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 failed: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-19 00:12:23.140 8 WARNING os_brick.initiator.connectors.iscsi [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] There were errors removing set(), leftovers may remain in the system

  ==> /var/log/kolla/nova/nova-compute.log <==
  2024-04-19 00:12:23.352 8 INFO nova.compute.manager [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] [instance: 712454ec-3ab3-4648-97c5-ede2edce5c9c] Successfully reverted task state from powering-on on failure for instance.
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server [None req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98 e95c8c84b62e416daf03b91a9648962b - - default default] Exception during message handling: os_brick.exception.ExceptionChainer:
  Chained Exception #1
          Traceback (most recent call last):
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1177, in _disconnect_connection
              self._disconnect_from_iscsi_portal(props)
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1159, in _disconnect_from_iscsi_portal
              self._run_iscsiadm(connection_properties, ("--logout",),
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1025, in _run_iscsiadm
              (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line 55, in _execute
              result = self.__execute(*args, **kwargs)
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
              return execute_root(*cmd, **kwargs)
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
              return self.channel.remote_call(name, args, kwargs,
            File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
              raise exc_type(*result[2])
          oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
          Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p 192.168.5.1:3260 --logout
          Exit code: 32
          Stdout: 'Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260]\n'
          Stderr: 'iscsiadm: Could not logout of [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not connected)\niscsiadm: Could not logout of all requested sessions\n'
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 3424, in start_instance
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 3392, in _power_on
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.driver.power_on(context, instance,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 4136, in power_on
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3955, in _hard_reboot
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.destroy(context, instance, network_info, destroy_disks=False,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1565, in destroy
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.cleanup(context, instance, network_info, block_device_info,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1635, in cleanup
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._cleanup(
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1690, in _cleanup
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception() as ctxt:
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 1686, in _cleanup
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self._disconnect_volume(
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2017, in _disconnect_volume
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     vol_driver.disconnect_volume(
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     self.connector.disconnect_volume(
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 173, in trace_logging_wrapper
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py", line 78, in inner
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 405, in change_encrypted
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     res = func(**call_args)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 891, in disconnect_volume
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     return self._cleanup_connection(connection_properties, force=force,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 969, in _cleanup_connection
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server     raise exc  # type: ignore
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server os_brick.exception.ExceptionChainer:
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Chained Exception #1
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Traceback (most recent call last):
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1177, in _disconnect_connection
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           self._disconnect_from_iscsi_portal(props)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1159, in _disconnect_from_iscsi_portal
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           self._run_iscsiadm(connection_properties, ("--logout",),
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 1025, in _run_iscsiadm
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           (out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line 55, in _execute
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           result = self.__execute(*args, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           return execute_root(*cmd, **kwargs)
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           return self.channel.remote_call(name, args, kwargs,
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server         File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server           raise exc_type(*result[2])
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p 192.168.5.1:3260 --logout
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Exit code: 32
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Stdout: 'Logging out of session [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260]\n'
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server       Stderr: 'iscsiadm: Could not logout of [sid: 1, target: iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal: 192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not connected)\niscsiadm: Could not logout of all requested sessions\n'
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
  2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2062459/+subscriptions