← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1288392] Re: instances get stuck in ERROR/deleting when Neutron is unavailable

 

** Changed in: nova/icehouse
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1288392

Title:
  instances get stuck in ERROR/deleting when Neutron is unavailable

Status in OpenStack Compute (Nova):
  Fix Committed
Status in OpenStack Compute (nova) icehouse series:
  Fix Released

Bug description:
  We had a temporary outage of Neutron, and many instances got stuck in
  this state. 'nova delete' on them does not work until nova-compute is
  forcibly restarted.

  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                                                                                                      |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                                                                     |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                                                       |
  | OS-EXT-SRV-ATTR:host                 | ci-overcloud-novacompute1-4q2dbhdklrkq                                                                                                                                                                                     |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | ci-overcloud-novacompute1-4q2dbhdklrkq.novalocal                                                                                                                                                                           |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00003f80                                                                                                                                                                                                          |
  | OS-EXT-STS:power_state               | 1                                                                                                                                                                                                                          |
  | OS-EXT-STS:task_state                | deleting                                                                                                                                                                                                                   |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                                                                                                      |
  | OS-SRV-USG:launched_at               | 2014-03-05T03:54:49.000000                                                                                                                                                                                                 |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                                          |
  | accessIPv4                           |                                                                                                                                                                                                                            |
  | accessIPv6                           |                                                                                                                                                                                                                            |
  | config_drive                         |                                                                                                                                                                                                                            |
  | created                              | 2014-03-05T03:46:25Z                                                                                                                                                                                                       |
  | default-net network                  | 10.0.58.225                                                                                                                                                                                                                |
  | fault                                | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "details": "  File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 253, in decorated_function |
  |                                      |     return function(self, context, *args, **kwargs)                                                                                                                                                                        |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2038, in terminate_instance                                                                                               |
  |                                      |     do_terminate_instance(instance, bdms)                                                                                                                                                                                  |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\", line 249, in inner                                                                                                  |
  |                                      |     return f(*args, **kwargs)                                                                                                                                                                                              |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2036, in do_terminate_instance                                                                                            |
  |                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                                                                                                              |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                                                                                 |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                                                                                                           |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2026, in do_terminate_instance                                                                                            |
  |                                      |     reservations=reservations)                                                                                                                                                                                             |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py\", line 103, in inner                                                                                                                       |
  |                                      |     rv = f(*args, **kwargs)                                                                                                                                                                                                |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2005, in _delete_instance                                                                                                 |
  |                                      |     user_id=user_id)                                                                                                                                                                                                       |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                                                                                 |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                                                                                                           |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1975, in _delete_instance                                                                                                 |
  |                                      |     self._shutdown_instance(context, db_inst, bdms)                                                                                                                                                                        |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1884, in _shutdown_instance                                                                                               |
  |                                      |     network_info = self._get_instance_nw_info(context, instance)                                                                                                                                                           |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 902, in _get_instance_nw_info                                                                                             |
  |                                      |     instance)                                                                                                                                                                                                              |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py\", line 48, in wrapper                                                                                                                |
  |                                      |     res = f(self, context, *args, **kwargs)                                                                                                                                                                                |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 445, in get_instance_nw_info                                                                                        |
  |                                      |     result = self._get_instance_nw_info(context, instance, networks)                                                                                                                                                       |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 452, in _get_instance_nw_info                                                                                       |
  |                                      |     nw_info = self._build_network_info_model(context, instance, networks)                                                                                                                                                  |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 1010, in _build_network_info_model                                                                                  |
  |                                      |     data = client.list_ports(**search_opts)                                                                                                                                                                                |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 112, in with_params                                                                                                  |
  |                                      |     ret = self.function(instance, *args, **kwargs)                                                                                                                                                                         |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 307, in list_ports                                                                                                   |
  |                                      |     **_params)                                                                                                                                                                                                             |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1251, in list                                                                                                        |
  |                                      |     for r in self._pagination(collection, path, **params):                                                                                                                                                                 |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1264, in _pagination                                                                                                 |
  |                                      |     res = self.get(path, params=params)                                                                                                                                                                                    |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1237, in get                                                                                                         |
  |                                      |     headers=headers, params=params)                                                                                                                                                                                        |
  |                                      |   File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1229, in retry_request                                                                                               |
  |                                      |     raise exceptions.ConnectionFailed(reason=_(\"Maximum attempts reached\"))                                                                                                                                              |
  |                                      | ", "created": "2014-03-05T04:22:04Z"}                                                                                                                                                                                      |
  | flavor                               | h1.large (872d8f61-c45a-45c3-87da-466d9f0f241b)                                                                                                                                                                            |
  | hostId                               | 0bab209cc6f26a8d5c4bc76e3da39d4fa68e5fefc6e5c0eada7a90d2                                                                                                                                                                   |
  | id                                   | ae9c75e3-51d2-43a3-8b20-34375b4c72d3                                                                                                                                                                                       |
  | image                                | tripleo-precise-1393812840.template.openstack.org (114e4b92-567e-4348-9ed8-e88281104208)                                                                                                                                   |
  | key_name                             | -                                                                                                                                                                                                                          |
  | metadata                             | {}                                                                                                                                                                                                                         |
  | name                                 | tripleo-precise-tripleo-test-cloud-2125650.slave.openstack.org                                                                                                                                                             |
  | os-extended-volumes:volumes_attached | []                                                                                                                                                                                                                         |
  | security_groups                      | default, default                                                                                                                                                                                                           |
  | status                               | ERROR                                                                                                                                                                                                                      |
  | tenant_id                            | 64d2d3bc07084ef1accd4e3502909c77                                                                                                                                                                                           |
  | tripleo-bm-test network              | 192.168.1.78                                                                                                                                                                                                               |
  | updated                              | 2014-03-05T04:22:04Z                                                                                                                                                                                                       |
  | user_id                              | 35ef3ce265cb4a25b5303f3daa143f4e                                                                                                                                                                                           |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1288392/+subscriptions


References