yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #15341
[Bug 1327124] [NEW] Instance stuck at spawning status after rescheduling
Public bug reported:
After fix this bug https://bugs.launchpad.net/nova/+bug/1326207 , The
reschedule works.
But found instance stuck at spawning status after rescheduling.
Envoriment is:
os3 is running ovs agent
os4 is running linuxbridge agent
When boot new instance, instance schedule to os3, but failed.
nova begin to reschedule instance to os4.
But the instance is stuck at spawning status.
os@os3:~/devstack$ nova show vm8
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | os4 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | os4 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000034 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | spawning |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-06-06T09:18:38Z |
| flavor | m1.tiny (1) |
| hostId | bf5039d22c8737b688f12f58afef5cde768f38732c47e48403932b44 |
| id | b6f6d043-1cfc-4c6d-914d-df4a48134589 |
| image | cirros-0.3.2-x86_64-uec (855f5370-8c60-4840-990d-f89b87bd3d19) |
| key_name | - |
| metadata | {} |
| name | vm8 |
| net1 network | 13.0.0.33 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 5c76520922254aa0a1459dc687bcbc1d |
| updated | 2014-06-06T09:18:40Z |
| user_id | 9be4442569e54c15ac6bab7d30af7d8f |
+--------------------------------------+----------------------------------------------------------------+
Get error from os4 linuxbridge agent:
2014-06-06 17:18:46.570 ERROR neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [-] Error in agent loop. Devices info: {'current': set(['tap15c2b6bc-51', 'tap2723ce44-87', 'tap484a667a-b1', 'tap8641bd9a-64', 'tap78fef3dd-1f', 'tap66d5a70e-0d']), 'removed': set([]), 'added': set(['tap15c2b6bc-51', 'tap2723ce44-87', 'tap484a667a-b1', 'tap8641bd9a-64', 'tap78fef3dd-1f', 'tap66d5a70e-0d'])}
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1011, in daemon_loop
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 908, in process_network_devices
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added(device_info['added'])
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 946, in treat_devices_added
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent details['port_id']):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 424, in add_interface
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 406, in add_tap_interface
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent root_helper=self.root_helper):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 76, in execute
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent raise RuntimeError(m)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError:
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'brctl', 'addif', 'brq4103cd57-70', 'tap8641bd9a-64']
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Exit code: 1
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdout: ''
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stderr: "device tap8641bd9a-64 is already a member of a bridge; can't enslave it to bridge brq4103cd57-70.\n"
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent
Check the port status:
os@os3:~/devstack$ neutron port-show 8641bd9a-64b2-439f-a9bd-8644787d25b6
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | os3 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | b6f6d043-1cfc-4c6d-914d-df4a48134589 |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "c26683fc-122f-4c30-bd05-f27f0499c6af", "ip_address": "13.0.0.33"} |
| id | 8641bd9a-64b2-439f-a9bd-8644787d25b6 |
| mac_address | fa:16:3e:4e:69:3b |
| name | |
| network_id | 4103cd57-70d9-4aa4-9501-5441585278f5 |
| security_groups | 8a836d5b-27fc-499f-9b7d-7d279afaca3d |
| status | BUILD |
| tenant_id | 5c76520922254aa0a1459dc687bcbc1d |
+-----------------------+----------------------------------------------------------------------------------+
It still binding at os3 host.
** Affects: nova
Importance: Undecided
Assignee: Alex Xu (xuhj)
Status: In Progress
** Changed in: nova
Assignee: (unassigned) => Alex Xu (xuhj)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1327124
Title:
Instance stuck at spawning status after rescheduling
Status in OpenStack Compute (Nova):
In Progress
Bug description:
After fix this bug https://bugs.launchpad.net/nova/+bug/1326207 , The
reschedule works.
But found instance stuck at spawning status after rescheduling.
Envoriment is:
os3 is running ovs agent
os4 is running linuxbridge agent
When boot new instance, instance schedule to os3, but failed.
nova begin to reschedule instance to os4.
But the instance is stuck at spawning status.
os@os3:~/devstack$ nova show vm8
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | os4 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | os4 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000034 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | spawning |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-06-06T09:18:38Z |
| flavor | m1.tiny (1) |
| hostId | bf5039d22c8737b688f12f58afef5cde768f38732c47e48403932b44 |
| id | b6f6d043-1cfc-4c6d-914d-df4a48134589 |
| image | cirros-0.3.2-x86_64-uec (855f5370-8c60-4840-990d-f89b87bd3d19) |
| key_name | - |
| metadata | {} |
| name | vm8 |
| net1 network | 13.0.0.33 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 5c76520922254aa0a1459dc687bcbc1d |
| updated | 2014-06-06T09:18:40Z |
| user_id | 9be4442569e54c15ac6bab7d30af7d8f |
+--------------------------------------+----------------------------------------------------------------+
Get error from os4 linuxbridge agent:
2014-06-06 17:18:46.570 ERROR neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [-] Error in agent loop. Devices info: {'current': set(['tap15c2b6bc-51', 'tap2723ce44-87', 'tap484a667a-b1', 'tap8641bd9a-64', 'tap78fef3dd-1f', 'tap66d5a70e-0d']), 'removed': set([]), 'added': set(['tap15c2b6bc-51', 'tap2723ce44-87', 'tap484a667a-b1', 'tap8641bd9a-64', 'tap78fef3dd-1f', 'tap66d5a70e-0d'])}
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1011, in daemon_loop
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 908, in process_network_devices
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added(device_info['added'])
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 946, in treat_devices_added
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent details['port_id']):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 424, in add_interface
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 406, in add_tap_interface
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent root_helper=self.root_helper):
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 76, in execute
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent raise RuntimeError(m)
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError:
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'brctl', 'addif', 'brq4103cd57-70', 'tap8641bd9a-64']
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Exit code: 1
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdout: ''
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stderr: "device tap8641bd9a-64 is already a member of a bridge; can't enslave it to bridge brq4103cd57-70.\n"
2014-06-06 17:18:46.570 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent
Check the port status:
os@os3:~/devstack$ neutron port-show 8641bd9a-64b2-439f-a9bd-8644787d25b6
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | os3 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | b6f6d043-1cfc-4c6d-914d-df4a48134589 |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "c26683fc-122f-4c30-bd05-f27f0499c6af", "ip_address": "13.0.0.33"} |
| id | 8641bd9a-64b2-439f-a9bd-8644787d25b6 |
| mac_address | fa:16:3e:4e:69:3b |
| name | |
| network_id | 4103cd57-70d9-4aa4-9501-5441585278f5 |
| security_groups | 8a836d5b-27fc-499f-9b7d-7d279afaca3d |
| status | BUILD |
| tenant_id | 5c76520922254aa0a1459dc687bcbc1d |
+-----------------------+----------------------------------------------------------------------------------+
It still binding at os3 host.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1327124/+subscriptions
Follow ups
References
