← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1428182] [NEW] Removing network bridge cause ERROR state of instance during deletion

 

Public bug reported:

Instance moves to ERROR state after deleting this instance
Configuration:
  1. network_manager = nova.network.manager.VlanManager
  2. teardown_unused_network_gateway = true
Steps to reproduce:
  1. launch instance
  2. remove instance
Expected result:
instance is removed without error
Actual result:
instance is not removed, instance' state become ERROR:
root@node-1:~# nova list
+--------------------------------------+-----------------+--------+------------+-------------+----------------------+
| ID                                   | Name            | Status | Task State | Power State | Networks             |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------+
| fd581745-8ecd-4fa2-af80-82493d083b97 | test_del_bridge | ERROR  | deleting   | Running     | novanetwork=10.0.0.3 |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------+
root@node-1:~# nova show fd581745-8ecd-4fa2-af80-82493d083b97
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                  |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                                                                 |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                   |
| OS-EXT-SRV-ATTR:host                 | node-2                                                                                                                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node-2                                                                                                                                 |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                                                                                                      |
| OS-EXT-STS:power_state               | 1                                                                                                                                      |
| OS-EXT-STS:task_state                | deleting                                                                                                                               |
| OS-EXT-STS:vm_state                  | error                                                                                                                                  |
| OS-SRV-USG:launched_at               | 2015-03-04T15:14:44.000000                                                                                                             |
| OS-SRV-USG:terminated_at             | -                                                                                                                                      |
| accessIPv4                           |                                                                                                                                        |
| accessIPv6                           |                                                                                                                                        |
| config_drive                         |                                                                                                                                        |
| created                              | 2015-03-04T15:14:40Z                                                                                                                   |
| fault                                | {"message": "Remote error: ProcessExecutionError Unexpected error while running command.                                               |
|                                      | Command: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br103 10.0.0.3 fa:16:3e:d2:1c:9a                                      |
|                                      | Exit code: 1                                                                                                                           |
|                                      | Stdout: ''                                                                                                                             |
|                                      | Stderr: 'cannot setup interface: No such device\                                                                                       |
|                                      | '                                                                                                                                      |
|                                      | [u'Tra", "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 296, in decorated_function |
|                                      |     return function(self, context, *args, **kwargs)                                                                                    |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2262, in terminate_instance                                  |
|                                      |     do_terminate_instance(instance, bdms)                                                                                              |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 249, in inner                                     |
|                                      |     return f(*args, **kwargs)                                                                                                          |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2260, in do_terminate_instance                               |
|                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                          |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
|                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2250, in do_terminate_instance                               |
|                                      |     reservations=reservations)                                                                                                         |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 103, in inner                                                          |
|                                      |     rv = f(*args, **kwargs)                                                                                                            |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2220, in _delete_instance                                    |
|                                      |     user_id=user_id)                                                                                                                   |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
|                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2190, in _delete_instance                                    |
|                                      |     self._shutdown_instance(context, db_inst, bdms)                                                                                    |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2127, in _shutdown_instance                                  |
|                                      |     self._try_deallocate_network(context, instance, requested_networks)                                                                |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2086, in _try_deallocate_network                             |
|                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                          |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
|                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2081, in _try_deallocate_network                             |
|                                      |     self._deallocate_network(context, instance, requested_networks)                                                                    |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1775, in _deallocate_network                                 |
|                                      |     context, instance, requested_networks=requested_networks)                                                                          |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/api.py\", line 94, in wrapped                                                   |
|                                      |     return func(self, context, *args, **kwargs)                                                                                        |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/api.py\", line 318, in deallocate_for_instance                                  |
|                                      |     requested_networks=requested_networks)                                                                                             |
|                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py\", line 190, in deallocate_for_instance                               |
|                                      |     return cctxt.call(ctxt, 'deallocate_for_instance', **kwargs)                                                                       |
|                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py\", line 150, in call                                            |
|                                      |     wait_for_reply=True, timeout=timeout)                                                                                              |
|                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py\", line 90, in _send                                             |
|                                      |     timeout=timeout)                                                                                                                   |
|                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py\", line 409, in send                                   |
|                                      |     return self._send(target, ctxt, message, wait_for_reply, timeout)                                                                  |
|                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py\", line 402, in _send                                  |
|                                      |     raise result                                                                                                                       |
|                                      | ", "created": "2015-03-04T15:14:48Z"}                                                                                                  |
| flavor                               | m1.tiny (1)                                                                                                                            |
| hostId                               | 8a17e6dfb8afa7a82fa5964edc54bb5355cc46fb73e000bc2b310623                                                                               |
| id                                   | fd581745-8ecd-4fa2-af80-82493d083b97                                                                                                   |
| image                                | TestVM (a671fa04-d19c-49d1-888d-9a43412e13ac)                                                                                          |
| key_name                             | KEY_PAIR                                                                                                                               |
| metadata                             | {}                                                                                                                                     |
| name                                 | test_del_bridge                                                                                                                        |
| novanetwork network                  | 10.0.0.3                                                                                                                               |
| os-extended-volumes:volumes_attached | []                                                                                                                                     |
| security_groups                      | default                                                                                                                                |
| status                               | ERROR                                                                                                                                  |
| tenant_id                            | dc669bf633ae4bd7b995c885e0428fec                                                                                                       |
| updated                              | 2015-03-04T15:14:48Z                                                                                                                   |
| user_id                              | bd2a2c6487484a24937c39740a76e501                                                                                                       |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+

** Affects: nova
     Importance: Undecided
     Assignee: Marian Horban (mhorban)
         Status: In Progress

** Changed in: nova
       Status: New => In Progress

** Changed in: nova
     Assignee: (unassigned) => Marian Horban (mhorban)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1428182

Title:
  Removing network bridge cause ERROR state of instance during deletion

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Instance moves to ERROR state after deleting this instance
  Configuration:
    1. network_manager = nova.network.manager.VlanManager
    2. teardown_unused_network_gateway = true
  Steps to reproduce:
    1. launch instance
    2. remove instance
  Expected result:
  instance is removed without error
  Actual result:
  instance is not removed, instance' state become ERROR:
  root@node-1:~# nova list
  +--------------------------------------+-----------------+--------+------------+-------------+----------------------+
  | ID                                   | Name            | Status | Task State | Power State | Networks             |
  +--------------------------------------+-----------------+--------+------------+-------------+----------------------+
  | fd581745-8ecd-4fa2-af80-82493d083b97 | test_del_bridge | ERROR  | deleting   | Running     | novanetwork=10.0.0.3 |
  +--------------------------------------+-----------------+--------+------------+-------------+----------------------+
  root@node-1:~# nova show fd581745-8ecd-4fa2-af80-82493d083b97
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                  |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                 |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                   |
  | OS-EXT-SRV-ATTR:host                 | node-2                                                                                                                                 |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | node-2                                                                                                                                 |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                                                                                                      |
  | OS-EXT-STS:power_state               | 1                                                                                                                                      |
  | OS-EXT-STS:task_state                | deleting                                                                                                                               |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                  |
  | OS-SRV-USG:launched_at               | 2015-03-04T15:14:44.000000                                                                                                             |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                      |
  | accessIPv4                           |                                                                                                                                        |
  | accessIPv6                           |                                                                                                                                        |
  | config_drive                         |                                                                                                                                        |
  | created                              | 2015-03-04T15:14:40Z                                                                                                                   |
  | fault                                | {"message": "Remote error: ProcessExecutionError Unexpected error while running command.                                               |
  |                                      | Command: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br103 10.0.0.3 fa:16:3e:d2:1c:9a                                      |
  |                                      | Exit code: 1                                                                                                                           |
  |                                      | Stdout: ''                                                                                                                             |
  |                                      | Stderr: 'cannot setup interface: No such device\                                                                                       |
  |                                      | '                                                                                                                                      |
  |                                      | [u'Tra", "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 296, in decorated_function |
  |                                      |     return function(self, context, *args, **kwargs)                                                                                    |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2262, in terminate_instance                                  |
  |                                      |     do_terminate_instance(instance, bdms)                                                                                              |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 249, in inner                                     |
  |                                      |     return f(*args, **kwargs)                                                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2260, in do_terminate_instance                               |
  |                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2250, in do_terminate_instance                               |
  |                                      |     reservations=reservations)                                                                                                         |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 103, in inner                                                          |
  |                                      |     rv = f(*args, **kwargs)                                                                                                            |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2220, in _delete_instance                                    |
  |                                      |     user_id=user_id)                                                                                                                   |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2190, in _delete_instance                                    |
  |                                      |     self._shutdown_instance(context, db_inst, bdms)                                                                                    |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2127, in _shutdown_instance                                  |
  |                                      |     self._try_deallocate_network(context, instance, requested_networks)                                                                |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2086, in _try_deallocate_network                             |
  |                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                    |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                       |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2081, in _try_deallocate_network                             |
  |                                      |     self._deallocate_network(context, instance, requested_networks)                                                                    |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1775, in _deallocate_network                                 |
  |                                      |     context, instance, requested_networks=requested_networks)                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/api.py\", line 94, in wrapped                                                   |
  |                                      |     return func(self, context, *args, **kwargs)                                                                                        |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/api.py\", line 318, in deallocate_for_instance                                  |
  |                                      |     requested_networks=requested_networks)                                                                                             |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py\", line 190, in deallocate_for_instance                               |
  |                                      |     return cctxt.call(ctxt, 'deallocate_for_instance', **kwargs)                                                                       |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py\", line 150, in call                                            |
  |                                      |     wait_for_reply=True, timeout=timeout)                                                                                              |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py\", line 90, in _send                                             |
  |                                      |     timeout=timeout)                                                                                                                   |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py\", line 409, in send                                   |
  |                                      |     return self._send(target, ctxt, message, wait_for_reply, timeout)                                                                  |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py\", line 402, in _send                                  |
  |                                      |     raise result                                                                                                                       |
  |                                      | ", "created": "2015-03-04T15:14:48Z"}                                                                                                  |
  | flavor                               | m1.tiny (1)                                                                                                                            |
  | hostId                               | 8a17e6dfb8afa7a82fa5964edc54bb5355cc46fb73e000bc2b310623                                                                               |
  | id                                   | fd581745-8ecd-4fa2-af80-82493d083b97                                                                                                   |
  | image                                | TestVM (a671fa04-d19c-49d1-888d-9a43412e13ac)                                                                                          |
  | key_name                             | KEY_PAIR                                                                                                                               |
  | metadata                             | {}                                                                                                                                     |
  | name                                 | test_del_bridge                                                                                                                        |
  | novanetwork network                  | 10.0.0.3                                                                                                                               |
  | os-extended-volumes:volumes_attached | []                                                                                                                                     |
  | security_groups                      | default                                                                                                                                |
  | status                               | ERROR                                                                                                                                  |
  | tenant_id                            | dc669bf633ae4bd7b995c885e0428fec                                                                                                       |
  | updated                              | 2015-03-04T15:14:48Z                                                                                                                   |
  | user_id                              | bd2a2c6487484a24937c39740a76e501                                                                                                       |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1428182/+subscriptions


Follow ups

References