yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Changed in: nova
   Importance: Low => Medium

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Changed in: nova/stein
       Status: New => In Progress

** Changed in: nova/rocky
       Status: New => In Progress

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova/stein
   Importance: Undecided => Medium

** Changed in: nova/rocky
     Assignee: (unassigned) => Julia Kreger (juliaashleykreger)

** Changed in: nova/stein
     Assignee: (unassigned) => Julia Kreger (juliaashleykreger)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815799

Title:
  Attempting to delete a baremetal server places server into ERROR state

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) rocky series:
  In Progress
Status in OpenStack Compute (nova) stein series:
  In Progress

Bug description:
  When deleting a baremetal server, we see:

    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a] Traceback (most recent call last):
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2698, in do_terminate_instance
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self._delete_instance(context, instance, bdms)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     rv = f(*args, **kwargs)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2635, in _delete_instance
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self._shutdown_instance(context, instance, bdms)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2527, in _shutdown_instance
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     requested_networks)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self.force_reraise()
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     six.reraise(self.type_, self.value, self.tb)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2514, in _shutdown_instance
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     block_device_info)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1245, in destroy
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self._cleanup_deploy(node, instance, network_info)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 461, in _cleanup_deploy
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self._remove_instance_info_from_node(node, instance)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 396, in _remove_instance_info_from_node
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     self.ironicclient.call('node.update', node.uuid, patch)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py", line 170, in call
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     return self._multi_getattr(client, method)(*args, **kwargs)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/ironicclient/v1/node.py", line 360, in update
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     params=params)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 232, in _update
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     resp, body = self.api.json_request(method, url, **kwargs)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 678, in json_request
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     resp = self._http_request(url, method, **kwargs)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 287, in wrapper
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     return func(self, url, method, **kwargs)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]   File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 660, in _http_request
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a]     error_json.get('debuginfo'), method, url)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a] Conflict: Node a00696d5-32ba-475e-9528-59bf11cffea6 can not be updated while a state transition is in progress. (HTTP 409)
    2019-02-13 12:58:16.856 7 ERROR nova.compute.manager [instance: dcb4f055-cda4-4d61-ba8f-976645c4e92a] 

  This places the server into an ERROR state:

    $ openstack server list
    +--------------------------------------+------------+--------+----------------------------+---------+------------+
    | ID                                   | Name       | Status | Networks                   | Image   | Flavor     |
    +--------------------------------------+------------+--------+----------------------------+---------+------------+
    | dcb4f055-cda4-4d61-ba8f-976645c4e92a | larstest2  | ERROR  |                            |         | baremetal  |
    +--------------------------------------+------------+--------+----------------------------+---------+------------+

  A subsequent "openstack server delete" will successfully delete the
  server.

  On the ironic side, we see the following message logged several times
  before Nova reports the conflict:

    2019-02-13 12:58:15.138 21 DEBUG wsme.api [req-b6f0a1a3-e2cc-
  4b13-a979-3676757b0972 7a9b1ac45e084e7cbeb9cb740ffe8d08
  41ea8af8d00e46438c7be3b182bbb53f - default default] Client-side error:
  Node a00696d5-32ba-475e-9528-59bf11cffea6 can not be updated while a
  state transition is in progress. format_exception /usr/lib/python2.7
  /site-packages/wsme/api.py:222

  This is with nova @ ad842aa and ironic @ 4404292.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815799/+subscriptions