yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #39083
[Bug 1499312] [NEW] [kilo]Some instances are in error status after 20 instances launched concurrently due to neutron scheduling.
Public bug reported:
[Summary]
Launch 20 instances concurrently, found 5 instances are in ERROR status.
[Topo]
Both controller and compute nodes are running on centos7.0 img: 3.10.0-229.11.1.el7.x86_64
ovs_version: "2.3.1"
[Reproduceable or not]
Can be reproduced easily.
[Recreate Steps]
1. Launch 20 instances concurrently:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 boot --flavor 1 --image dbeaeaf5-b996-48ee-8272-dada151cf34e --nic net-id=d5be47c2-f54e-49b9-9feb-d89cc6b86c56 --availability-zone scaling:quasarin-1 in1-20 --max 20
2. check instances state, 5 instances are in error status:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning | NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning | NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning | NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning | NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning | NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[logs]
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning | NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning | NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning | NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning | NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning | NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[root@quasarin-1 ~(keystone_admin)]$ nova show bf41557b-466c-423f-a36f-f25dd1453981
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | scaling |
| OS-EXT-SRV-ATTR:host | quasarin-1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | quasarin-1.node.os |
| OS-EXT-SRV-ATTR:instance_name | instance-000002da |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | spawning |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-09-07T02:46:19Z |
| fault | {"message": "Build of instance bf41557b-466c-423f-a36f-f25dd1453981 aborted: Failed to allocate the network(s), not rescheduling.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2235, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2368, in _build_and_run_instance |
| | reason=msg) |
| | ", "created": "2015-09-07T02:52:08Z"} |
| flavor | m1.tiny (1) |
| hostId | 47e713ab57ba5e9d11ddbec87bfe89f48987bf3761592656d5149dfb |
| id | bf41557b-466c-423f-a36f-f25dd1453981 |
| image | cirros image (dbeaeaf5-b996-48ee-8272-dada151cf34e) |
| key_name | - |
| metadata | {} |
| name | in1-20-13 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 3729e963b79d421ab2274f32a50b7918 |
| updated | 2015-09-07T02:52:09Z |
| user_id | 97f21887fa8e400cac95208acfb00c24 |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@quasarin-1 ~(keystone_admin)]$
Compute nova log:
2015-09-06 20:04:39.504 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Instance failed to spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] yield resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] raise exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.789 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Failed to allocate network(s)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] raise exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.815 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] filter_properties)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] reason=msg)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] BuildAbortException: Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:40.325 10862 ERROR nova.network.neutronv2.api [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] Unable to clear device ID for port 'None'
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api port_client.update_port(port_id, port_req_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api ret = self.function(instance, *args, **kwargs)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api return self.put(self.port_path % (port), body=body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api self._handle_fault_response(status_code, replybody)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api exception_handler_v20(status_code, des_error_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api message=message)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api The resource could not be found.
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:13:55.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:23:58.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 20:34:14.082 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:44:19.109 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:44:19.191 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.101 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.150 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 21:16:00.291 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 21:16:00.292 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 21:18:53.025 10862 WARNING nova.virt.libvirt.driver [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Timeout waiting for vif plugging callback for instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a
2015-09-06 21:18:53.667 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Instance failed to spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] yield resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.970 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Failed to allocate network(s)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.993 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] filter_properties)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] reason=msg)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] BuildAbortException: Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:55.114 10862 ERROR nova.network.neutronv2.api [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Unable to clear device ID for port 'None'
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api port_client.update_port(port_id, port_req_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api ret = self.function(instance, *args, **kwargs)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api return self.put(self.port_path % (port), body=body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api self._handle_fault_response(status_code, replybody)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api exception_handler_v20(status_code, des_error_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api message=message)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api The resource could not be found.
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1499312
Title:
[kilo]Some instances are in error status after 20 instances launched
concurrently due to neutron scheduling.
Status in OpenStack Compute (nova):
New
Bug description:
[Summary]
Launch 20 instances concurrently, found 5 instances are in ERROR status.
[Topo]
Both controller and compute nodes are running on centos7.0 img: 3.10.0-229.11.1.el7.x86_64
ovs_version: "2.3.1"
[Reproduceable or not]
Can be reproduced easily.
[Recreate Steps]
1. Launch 20 instances concurrently:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 boot --flavor 1 --image dbeaeaf5-b996-48ee-8272-dada151cf34e --nic net-id=d5be47c2-f54e-49b9-9feb-d89cc6b86c56 --availability-zone scaling:quasarin-1 in1-20 --max 20
2. check instances state, 5 instances are in error status:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning | NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning | NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning | NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning | NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning | NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[logs]
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning | NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning | NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning | NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning | NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning | NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - | Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[root@quasarin-1 ~(keystone_admin)]$ nova show bf41557b-466c-423f-a36f-f25dd1453981
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | scaling |
| OS-EXT-SRV-ATTR:host | quasarin-1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | quasarin-1.node.os |
| OS-EXT-SRV-ATTR:instance_name | instance-000002da |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | spawning |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-09-07T02:46:19Z |
| fault | {"message": "Build of instance bf41557b-466c-423f-a36f-f25dd1453981 aborted: Failed to allocate the network(s), not rescheduling.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2235, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2368, in _build_and_run_instance |
| | reason=msg) |
| | ", "created": "2015-09-07T02:52:08Z"} |
| flavor | m1.tiny (1) |
| hostId | 47e713ab57ba5e9d11ddbec87bfe89f48987bf3761592656d5149dfb |
| id | bf41557b-466c-423f-a36f-f25dd1453981 |
| image | cirros image (dbeaeaf5-b996-48ee-8272-dada151cf34e) |
| key_name | - |
| metadata | {} |
| name | in1-20-13 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 3729e963b79d421ab2274f32a50b7918 |
| updated | 2015-09-07T02:52:09Z |
| user_id | 97f21887fa8e400cac95208acfb00c24 |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@quasarin-1 ~(keystone_admin)]$
Compute nova log:
2015-09-06 20:04:39.504 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Instance failed to spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] yield resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] raise exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.789 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Failed to allocate network(s)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] raise exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.815 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] filter_properties)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] reason=msg)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] BuildAbortException: Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:40.325 10862 ERROR nova.network.neutronv2.api [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] Unable to clear device ID for port 'None'
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api port_client.update_port(port_id, port_req_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api ret = self.function(instance, *args, **kwargs)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api return self.put(self.port_path % (port), body=body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api self._handle_fault_response(status_code, replybody)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api exception_handler_v20(status_code, des_error_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api message=message)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api The resource could not be found.
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:13:55.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:23:58.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 20:34:14.082 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
2015-09-06 20:44:19.109 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:44:19.191 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.101 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.150 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 21:16:00.291 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 21:16:00.292 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 21:18:53.025 10862 WARNING nova.virt.libvirt.driver [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Timeout waiting for vif plugging callback for instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a
2015-09-06 21:18:53.667 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Instance failed to spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] yield resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.970 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Failed to allocate network(s)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.993 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] filter_properties)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] reason=msg)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] BuildAbortException: Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:55.114 10862 ERROR nova.network.neutronv2.api [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Unable to clear device ID for port 'None'
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api port_client.update_port(port_id, port_req_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api ret = self.function(instance, *args, **kwargs)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api return self.put(self.port_path % (port), body=body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api self._handle_fault_response(status_code, replybody)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api exception_handler_v20(status_code, des_error_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api message=message)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api The resource could not be found.
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1499312/+subscriptions
Follow ups
