yahoo-eng-team team mailing list archive

[OpenStack Infra <1765942@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/563418
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c6dda9f8064e0fcdf7ac85d08ed68554083be74b
Submitter: Zuul
Branch:    master

commit c6dda9f8064e0fcdf7ac85d08ed68554083be74b
Author: MultipleCrashes <inimitableharish@xxxxxxxxx>
Date:   Wed Jul 4 16:55:22 2018 +0530

    Retry decorator fix for instances which go into ERROR state during bulk delete
    
    During bulk delete, some of the machines go into ERROR state
    rather than being deleted. This happens once in a while during
    deletion of huge number of machines concurrently. The failure occurs
    during deallocation of network. At a later point in time the ERROR
    state gets cleared if the user deletes the instance manually later
    on. This fix proposes for retries for certain number of
    time with variable delay to allow proper deallocation of network.
    
    Change-Id: I32212b4d8180e947fdc958449aebd822f50e97fd
    Closes-Bug: #1765942


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1765942

Title:
  Delete issued to building server leaves the instance in error state
  due to " not able to deallocate network" error .

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Delete issued to building server leaves the instance in error state
  due to " not able to deallocate network" error.During bulk delete of
  instances, some of the machines go into ERROR state rather than being
  deleted.This happens intermittently during deletion of huge number of
  instances at once.Log line has this entry "Setting instance vm_state
  to ERROR".Failure in do_terminate_instance.The cause of the failure
  seems to be the ConnectFailure as the traceback suggests that the
  caller is not able to establish connection to one of the endpoints.

  Here is the traceback from the failure:

  Error Log

  => Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.332s inner nova/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  => [instance: instance_id] Failed to deallocate network for instance.
  => Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks nova/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
  => CAST unique_id: exchange 'nova' topic 'cells' _send nova/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
  => CAST unique_id: NOTIFY exchange 'nova' topic 'notifications.info' _send nova/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:438
  => CAST unique_id: NOTIFY exchange 'nova' topic 'monitor.info' _send nova/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:438
  => [instance: instance_id] Setting instance vm_state to ERROR Traceback (most recent call last): File "nova/
      self._delete_instance(context, instance, bdms, quotas)
    File "nova/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
      rv = f(*args, **kwargs)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2520, in _delete_instance
  6 quotas.rollback()
    File "nova/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "nova/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2496, in _delete_instance
      self._shutdown_instance(context, instance, bdms)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2393, in _shutdown_instance
      self._try_deallocate_network(context, instance, requested_networks)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2317, in _try_deallocate_network
      self._set_instance_obj_error_state(context, instance)
    File "nova/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "nova/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2312, in _try_deallocate_network
      self._deallocate_network(context, instance, requested_networks)
    File "nova/lib/python2.7/site-packages/nova/compute/manager.py", line 1858, in _deallocate_network
      context, instance, requested_networks=requested_networks)
    File "nova/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 819, in deallocate_for_instance
      data = neutron.list_ports(**search_opts)
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 618, in list_ports
      **_params)
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 357, in list
      for r in self._pagination(collection, path, **params):
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 372, in _pagination
      res = self.get(path, params=params)
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 342, in get
      headers=headers, params=params)
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 319, in retry_request
      headers=headers, params=params)
    File "nova/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in do_request
      resp, replybody = self.httpclient.do_request(action, method, body=body)
    File "nova/lib/python2.7/site-packages/neutronclient/client.py", line 322, in do_request
      return self.request(url, method, **kwargs)
    File "nova/lib/python2.7/site-packages/neutronclient/client.py", line 310, in request
      resp = super(SessionClient, self).request(*args, **kwargs)
    File "nova/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 106, in request
      return self.session.request(url, method, **kwargs)
    File "nova/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
      return wrapped(*args, **kwargs)
    File "nova/lib/python2.7/site-packages/keystoneauth1/session.py", line 452, in request
      resp = send(**kwargs)
    File "nova/lib/python2.7/site-packages/keystoneauth1/session.py", line 501, in _send_request
      raise exceptions.ConnectFailure(msg)
  ConnectFailure: Unable to establish connection to http:/somehost:someport/v2.0/ports.json?device_id=someid
  6854139:2018-02-21 16:43:19.551 26180 ERROR nova.compute.manager [instance: instance_id]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1765942/+subscriptions