yahoo-eng-team team mailing list archive

[Vasyl Saienko <2064826@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

After switching to openstacksdk hard reboot stop working for ironic.

2024-05-04 17:43:20.213 1 DEBUG nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338
a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Checking state _get_power_state /var/lib/openstack/lib/python3.10/site-packages/nova/com
pute/manager.py:1784
2024-05-04 17:43:20.403 1 DEBUG nova.virt.ironic.driver [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Reboot(type HARD) called for instance reboot /var/lib/openstack/lib/python3.10/site-packages/nova/virt/ironic/driver.py:1401
2024-05-04 17:43:20.485 1 DEBUG nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Checking state _get_power_state /var/lib/openstack/lib/python3.10/site-packages/nova/compute/manager.py:1784
2024-05-04 17:43:20.530 1 WARNING nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Reboot failed but instance is running: openstack.exceptions.BadRequestException: Failed to set power state for bare metal node b9fbad5f-c0c9-40be-9521-ca777c85740a to reboot: Client Error for url: http://ironic-api.openstack.svc.cluster.local:6385/v1/nodes/b9fbad5f-c0c9-40be-9521-ca777c85740a/states/power, The requested action "reboot" can not be performed on node "b9fbad5f-c0c9-40be-9521-ca777c85740a" while it is in state "power on".
2024-05-04 17:43:20.555 1 DEBUG nova.compute.utils [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] BadRequestException notify_about_instance_usage /var/lib/openstack/lib/python3.10/site-packages/nova/compute/utils.py:430

With manual run got same

>>> ocm.baremetal.set_node_power_state('a38a8613-f961-4633-b972-6be00f0f9339', 'reboot')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/openstack/lib/python3.10/site-packages/openstack/baremetal/v1/_proxy.py", line 617, in set_node_power_state
    self._get_resource(_node.Node, node).set_power_state(
  File "/var/lib/openstack/lib/python3.10/site-packages/openstack/baremetal/v1/node.py", line 782, in set_power_state
    exceptions.raise_from_response(response, error_message=msg)
  File "/var/lib/openstack/lib/python3.10/site-packages/openstack/exceptions.py", line 247, in raise_from_response
    raise cls(
openstack.exceptions.BadRequestException: Failed to set power state for bare metal node a38a8613-f961-4633-b972-6be00f0f9339 to reboot: Client Error for url: http://ironic-api.openstack.svc.cluster.local:6385/v1/nodes/a38a8613-f961-4633-b972-6be00f0f9339/states/power, The requested action "reboot" can not be performed on node "a38a8613-f961-4633-b972-6be00f0f9339" while it is in state "power on".
>>> ocm.baremetal.set_node_power_state('a38a8613-f961-4633-b972-6be00f0f9339', 'rebooting')
>>> 

To fix the problem we need to use correct target state `rebooting` not
`reboot`

https://github.com/openstack/openstacksdk/blob/master/openstack/baremetal/v1/node.py#L44

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2064826

Title:
  Can not hard reboot VM with ironic driver

Status in OpenStack Compute (nova):
  New

Bug description:
  After switching to openstacksdk hard reboot stop working for ironic.

  2024-05-04 17:43:20.213 1 DEBUG nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338
  a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Checking state _get_power_state /var/lib/openstack/lib/python3.10/site-packages/nova/com
  pute/manager.py:1784
  2024-05-04 17:43:20.403 1 DEBUG nova.virt.ironic.driver [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Reboot(type HARD) called for instance reboot /var/lib/openstack/lib/python3.10/site-packages/nova/virt/ironic/driver.py:1401
  2024-05-04 17:43:20.485 1 DEBUG nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Checking state _get_power_state /var/lib/openstack/lib/python3.10/site-packages/nova/compute/manager.py:1784
  2024-05-04 17:43:20.530 1 WARNING nova.compute.manager [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] Reboot failed but instance is running: openstack.exceptions.BadRequestException: Failed to set power state for bare metal node b9fbad5f-c0c9-40be-9521-ca777c85740a to reboot: Client Error for url: http://ironic-api.openstack.svc.cluster.local:6385/v1/nodes/b9fbad5f-c0c9-40be-9521-ca777c85740a/states/power, The requested action "reboot" can not be performed on node "b9fbad5f-c0c9-40be-9521-ca777c85740a" while it is in state "power on".
  2024-05-04 17:43:20.555 1 DEBUG nova.compute.utils [None req-dfe5f95e-afa2-433d-bc34-ca9186936928 462cb36b3999412393b826b8d8bcc323 a9267fb61f0a428882eafc45b42bc489 - - f0b81946e4eb4f338a13f241726662ea f0b81946e4eb4f338a13f241726662ea] [instance: 5987800b-77f6-451b-ad83-38af30eac1b1] BadRequestException notify_about_instance_usage /var/lib/openstack/lib/python3.10/site-packages/nova/compute/utils.py:430

  With manual run got same

  >>> ocm.baremetal.set_node_power_state('a38a8613-f961-4633-b972-6be00f0f9339', 'reboot')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/var/lib/openstack/lib/python3.10/site-packages/openstack/baremetal/v1/_proxy.py", line 617, in set_node_power_state
      self._get_resource(_node.Node, node).set_power_state(
    File "/var/lib/openstack/lib/python3.10/site-packages/openstack/baremetal/v1/node.py", line 782, in set_power_state
      exceptions.raise_from_response(response, error_message=msg)
    File "/var/lib/openstack/lib/python3.10/site-packages/openstack/exceptions.py", line 247, in raise_from_response
      raise cls(
  openstack.exceptions.BadRequestException: Failed to set power state for bare metal node a38a8613-f961-4633-b972-6be00f0f9339 to reboot: Client Error for url: http://ironic-api.openstack.svc.cluster.local:6385/v1/nodes/a38a8613-f961-4633-b972-6be00f0f9339/states/power, The requested action "reboot" can not be performed on node "a38a8613-f961-4633-b972-6be00f0f9339" while it is in state "power on".
  >>> ocm.baremetal.set_node_power_state('a38a8613-f961-4633-b972-6be00f0f9339', 'rebooting')
  >>> 

  To fix the problem we need to use correct target state `rebooting` not
  `reboot`

  https://github.com/openstack/openstacksdk/blob/master/openstack/baremetal/v1/node.py#L44

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2064826/+subscriptions