| Thread Previous • Date Previous • Date Next • Thread Next |
Public bug reported:
We had a temporary outage of Neutron, and many instances got stuck in
this state. 'nova delete' on them does not work until nova-compute is
forcibly restarted.
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ci-overcloud-novacompute1-4q2dbhdklrkq |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ci-overcloud-novacompute1-4q2dbhdklrkq.novalocal |
| OS-EXT-SRV-ATTR:instance_name | instance-00003f80 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-03-05T03:54:49.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-03-05T03:46:25Z |
| default-net network | 10.0.58.225 |
| fault | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "details": " File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 253, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2038, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\", line 249, in inner |
| | return f(*args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2036, in do_terminate_instance |
| | self._set_instance_error_state(context, instance['uuid']) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2026, in do_terminate_instance |
| | reservations=reservations) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py\", line 103, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2005, in _delete_instance |
| | user_id=user_id) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1975, in _delete_instance |
| | self._shutdown_instance(context, db_inst, bdms) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1884, in _shutdown_instance |
| | network_info = self._get_instance_nw_info(context, instance) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 902, in _get_instance_nw_info |
| | instance) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py\", line 48, in wrapper |
| | res = f(self, context, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 445, in get_instance_nw_info |
| | result = self._get_instance_nw_info(context, instance, networks) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 452, in _get_instance_nw_info |
| | nw_info = self._build_network_info_model(context, instance, networks) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 1010, in _build_network_info_model |
| | data = client.list_ports(**search_opts) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 112, in with_params |
| | ret = self.function(instance, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 307, in list_ports |
| | **_params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1251, in list |
| | for r in self._pagination(collection, path, **params): |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1264, in _pagination |
| | res = self.get(path, params=params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1237, in get |
| | headers=headers, params=params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1229, in retry_request |
| | raise exceptions.ConnectionFailed(reason=_(\"Maximum attempts reached\")) |
| | ", "created": "2014-03-05T04:22:04Z"} |
| flavor | h1.large (872d8f61-c45a-45c3-87da-466d9f0f241b) |
| hostId | 0bab209cc6f26a8d5c4bc76e3da39d4fa68e5fefc6e5c0eada7a90d2 |
| id | ae9c75e3-51d2-43a3-8b20-34375b4c72d3 |
| image | tripleo-precise-1393812840.template.openstack.org (114e4b92-567e-4348-9ed8-e88281104208) |
| key_name | - |
| metadata | {} |
| name | tripleo-precise-tripleo-test-cloud-2125650.slave.openstack.org |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default, default |
| status | ERROR |
| tenant_id | 64d2d3bc07084ef1accd4e3502909c77 |
| tripleo-bm-test network | 192.168.1.78 |
| updated | 2014-03-05T04:22:04Z |
| user_id | 35ef3ce265cb4a25b5303f3daa143f4e |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1288392
Title:
instances get stuck in ERROR/deleting when Neutron is unavailable
Status in OpenStack Compute (Nova):
New
Bug description:
We had a temporary outage of Neutron, and many instances got stuck in
this state. 'nova delete' on them does not work until nova-compute is
forcibly restarted.
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ci-overcloud-novacompute1-4q2dbhdklrkq |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ci-overcloud-novacompute1-4q2dbhdklrkq.novalocal |
| OS-EXT-SRV-ATTR:instance_name | instance-00003f80 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2014-03-05T03:54:49.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-03-05T03:46:25Z |
| default-net network | 10.0.58.225 |
| fault | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "details": " File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 253, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2038, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\", line 249, in inner |
| | return f(*args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2036, in do_terminate_instance |
| | self._set_instance_error_state(context, instance['uuid']) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2026, in do_terminate_instance |
| | reservations=reservations) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py\", line 103, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 2005, in _delete_instance |
| | user_id=user_id) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1975, in _delete_instance |
| | self._shutdown_instance(context, db_inst, bdms) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 1884, in _shutdown_instance |
| | network_info = self._get_instance_nw_info(context, instance) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py\", line 902, in _get_instance_nw_info |
| | instance) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py\", line 48, in wrapper |
| | res = f(self, context, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 445, in get_instance_nw_info |
| | result = self._get_instance_nw_info(context, instance, networks) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 452, in _get_instance_nw_info |
| | nw_info = self._build_network_info_model(context, instance, networks) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 1010, in _build_network_info_model |
| | data = client.list_ports(**search_opts) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 112, in with_params |
| | ret = self.function(instance, *args, **kwargs) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 307, in list_ports |
| | **_params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1251, in list |
| | for r in self._pagination(collection, path, **params): |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1264, in _pagination |
| | res = self.get(path, params=params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1237, in get |
| | headers=headers, params=params) |
| | File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1229, in retry_request |
| | raise exceptions.ConnectionFailed(reason=_(\"Maximum attempts reached\")) |
| | ", "created": "2014-03-05T04:22:04Z"} |
| flavor | h1.large (872d8f61-c45a-45c3-87da-466d9f0f241b) |
| hostId | 0bab209cc6f26a8d5c4bc76e3da39d4fa68e5fefc6e5c0eada7a90d2 |
| id | ae9c75e3-51d2-43a3-8b20-34375b4c72d3 |
| image | tripleo-precise-1393812840.template.openstack.org (114e4b92-567e-4348-9ed8-e88281104208) |
| key_name | - |
| metadata | {} |
| name | tripleo-precise-tripleo-test-cloud-2125650.slave.openstack.org |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default, default |
| status | ERROR |
| tenant_id | 64d2d3bc07084ef1accd4e3502909c77 |
| tripleo-bm-test network | 192.168.1.78 |
| updated | 2014-03-05T04:22:04Z |
| user_id | 35ef3ce265cb4a25b5303f3daa143f4e |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1288392/+subscriptions
| Thread Previous • Date Previous • Date Next • Thread Next |
