← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1499312] Re: [kilo]Some instances are in error status after 20 instances launched concurrently due to neutron scheduling.

 

The nova-docker virt driver is out of tree for nova and as far as I know
no longer maintained, so please don't report issues against nova, report
them against the nova-docker project and then after triage if they are
nova issues add nova to the bug report.

** Also affects: nova-docker
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1499312

Title:
  [kilo]Some instances are in error status after 20 instances launched
  concurrently due to neutron scheduling.

Status in OpenStack Compute (nova):
  Invalid
Status in nova-docker:
  New

Bug description:
  [Summary]
  Launch 20 instances concurrently, found 5 instances are in ERROR status.

  [Topo]
  Both controller and compute nodes are running on centos7.0 img: 3.10.0-229.11.1.el7.x86_64
  ovs_version: "2.3.1"

  [Reproduceable or not]
  Can be reproduced easily.

  [Recreate Steps]
  1. Launch 20 instances concurrently:
  [root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 boot --flavor 1 --image dbeaeaf5-b996-48ee-8272-dada151cf34e --nic net-id=d5be47c2-f54e-49b9-9feb-d89cc6b86c56 --availability-zone scaling:quasarin-1 in1-20 --max 20

  2. check instances state, 5 instances are in error status:
  [root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
    InsecurePlatformWarning
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
    SecurityWarning
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
  | ID                                   | Name      | Status | Task State | Power State | Networks                         |
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
  | 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.2  |
  | 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.9  |
  | eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.12 |
  | 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.13 |
  | bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR  | spawning   | NOSTATE     |                                  |
  | a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.15 |
  | 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.16 |
  | a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR  | spawning   | NOSTATE     |                                  |
  | ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR  | spawning   | NOSTATE     |                                  |
  | f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.19 |
  | 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.20 |
  | 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2  | ERROR  | spawning   | NOSTATE     |                                  |
  | a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.21 |
  | a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.4  |
  | e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.11 |
  | e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5  | ERROR  | spawning   | NOSTATE     |                                  |
  | e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.6  |
  | 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.8  |
  | 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.7  |
  | 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.10 |
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+

  
  [logs]
  [root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1 list
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
    InsecurePlatformWarning
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
    SecurityWarning
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
  | ID                                   | Name      | Status | Task State | Power State | Networks                         |
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
  | 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.2  |
  | 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.9  |
  | eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.12 |
  | 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.13 |
  | bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR  | spawning   | NOSTATE     |                                  |
  | a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.15 |
  | 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.16 |
  | a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR  | spawning   | NOSTATE     |                                  |
  | ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR  | spawning   | NOSTATE     |                                  |
  | f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.19 |
  | 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.20 |
  | 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2  | ERROR  | spawning   | NOSTATE     |                                  |
  | a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.21 |
  | a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.4  |
  | e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.11 |
  | e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5  | ERROR  | spawning   | NOSTATE     |                                  |
  | e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.6  |
  | 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.8  |
  | 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.7  |
  | 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9  | ACTIVE | -          | Running     | herman-tenant-1-net-1=100.1.1.10 |
  +--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
  [root@quasarin-1 ~(keystone_admin)]$ nova show bf41557b-466c-423f-a36f-f25dd1453981
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
    InsecurePlatformWarning
  /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
    SecurityWarning
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                                                                                                                                                      |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                                                                                                                     |
  | OS-EXT-AZ:availability_zone          | scaling                                                                                                                                                                                                                                                                    |
  | OS-EXT-SRV-ATTR:host                 | quasarin-1                                                                                                                                                                                                                                                                 |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | quasarin-1.node.os                                                                                                                                                                                                                                                         |
  | OS-EXT-SRV-ATTR:instance_name        | instance-000002da                                                                                                                                                                                                                                                          |
  | OS-EXT-STS:power_state               | 0                                                                                                                                                                                                                                                                          |
  | OS-EXT-STS:task_state                | spawning                                                                                                                                                                                                                                                                   |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                                                                                                                                                      |
  | OS-SRV-USG:launched_at               | -                                                                                                                                                                                                                                                                          |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                                                                                          |
  | accessIPv4                           |                                                                                                                                                                                                                                                                            |
  | accessIPv6                           |                                                                                                                                                                                                                                                                            |
  | config_drive                         |                                                                                                                                                                                                                                                                            |
  | created                              | 2015-09-07T02:46:19Z                                                                                                                                                                                                                                                       |
  | fault                                | {"message": "Build of instance bf41557b-466c-423f-a36f-f25dd1453981 aborted: Failed to allocate the network(s), not rescheduling.", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2235, in _do_build_and_run_instance |
  |                                      |     filter_properties)                                                                                                                                                                                                                                                     |
  |                                      |   File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2368, in _build_and_run_instance                                                                                                                                                                 |
  |                                      |     reason=msg)                                                                                                                                                                                                                                                            |
  |                                      | ", "created": "2015-09-07T02:52:08Z"}                                                                                                                                                                                                                                      |
  | flavor                               | m1.tiny (1)                                                                                                                                                                                                                                                                |
  | hostId                               | 47e713ab57ba5e9d11ddbec87bfe89f48987bf3761592656d5149dfb                                                                                                                                                                                                                   |
  | id                                   | bf41557b-466c-423f-a36f-f25dd1453981                                                                                                                                                                                                                                       |
  | image                                | cirros image (dbeaeaf5-b996-48ee-8272-dada151cf34e)                                                                                                                                                                                                                        |
  | key_name                             | -                                                                                                                                                                                                                                                                          |
  | metadata                             | {}                                                                                                                                                                                                                                                                         |
  | name                                 | in1-20-13                                                                                                                                                                                                                                                                  |
  | os-extended-volumes:volumes_attached | []                                                                                                                                                                                                                                                                         |
  | status                               | ERROR                                                                                                                                                                                                                                                                      |
  | tenant_id                            | 3729e963b79d421ab2274f32a50b7918                                                                                                                                                                                                                                           |
  | updated                              | 2015-09-07T02:52:09Z                                                                                                                                                                                                                                                       |
  | user_id                              | 97f21887fa8e400cac95208acfb00c24                                                                                                                                                                                                                                           |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  [root@quasarin-1 ~(keystone_admin)]$ 

  
  Compute nova log:
  2015-09-06 20:04:39.504 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Instance failed to spawn
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     yield resources
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     block_device_info=block_device_info)
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     block_device_info=block_device_info)
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     raise exception.VirtualInterfaceCreateException()
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
  2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] 
  2015-09-06 20:04:39.789 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Failed to allocate network(s)
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     block_device_info=block_device_info)
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     block_device_info=block_device_info)
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     raise exception.VirtualInterfaceCreateException()
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual Interface creation failed
  2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] 
  2015-09-06 20:04:39.815 10862 ERROR nova.compute.manager [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     filter_properties)
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f]     reason=msg)
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] BuildAbortException: Build of instance 2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the network(s), not rescheduling.
  2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance: 2151425f-c0bb-4226-8dc0-fef4285d176f] 
  2015-09-06 20:04:40.325 10862 ERROR nova.network.neutronv2.api [req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24 89480271287c49a49c1d26f467289497 - - -] Unable to clear device ID for port 'None'
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     port_client.update_port(port_id, port_req_body)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     ret = self.function(instance, *args, **kwargs)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     return self.put(self.port_path % (port), body=body)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     headers=headers, params=params)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     headers=headers, params=params)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     self._handle_fault_response(status_code, replybody)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     exception_handler_v20(status_code, des_error_body)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api     message=message)
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api 
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api The resource could not be found.
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api 
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api    
  2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api 
  2015-09-06 20:13:55.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
  2015-09-06 20:23:58.084 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
  2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
  2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
  2015-09-06 20:34:14.082 10862 WARNING nova.compute.manager [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
  2015-09-06 20:44:19.109 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
  2015-09-06 20:44:19.191 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
  2015-09-06 20:54:28.101 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connecting to AMQP server on openstack-rabbitmq.service.os:5672
  2015-09-06 20:54:28.150 10862 INFO oslo_messaging._drivers.impl_rabbit [-] Connected to AMQP server on openstack-rabbitmq.service.os:5672
  2015-09-06 21:16:00.291 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
  2015-09-06 21:16:00.292 10862 WARNING nova.virt.libvirt.imagecache [req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file: /var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
  2015-09-06 21:18:53.025 10862 WARNING nova.virt.libvirt.driver [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Timeout waiting for vif plugging callback for instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a
  2015-09-06 21:18:53.667 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Instance failed to spawn
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in _build_resources
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     yield resources
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     block_device_info=block_device_info)
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     block_device_info=block_device_info)
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     raise exception.VirtualInterfaceCreateException()
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
  2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] 
  2015-09-06 21:18:53.970 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Failed to allocate network(s)
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in _build_and_run_instance
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     block_device_info=block_device_info)
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     block_device_info=block_device_info)
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     raise exception.VirtualInterfaceCreateException()
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual Interface creation failed
  2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] 
  2015-09-06 21:18:53.993 10862 ERROR nova.compute.manager [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in _do_build_and_run_instance
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     filter_properties)
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in _build_and_run_instance
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a]     reason=msg)
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] BuildAbortException: Build of instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the network(s), not rescheduling.
  2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance: 66a809c3-4cbd-4d30-bfc4-41316b82cd2a] 
  2015-09-06 21:18:55.114 10862 ERROR nova.network.neutronv2.api [req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24 99784719d0a04f4fa1f2333d984e7c24 - - -] Unable to clear device ID for port 'None'
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api Traceback (most recent call last):
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in _unbind_ports
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     port_client.update_port(port_id, port_req_body)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     ret = self.function(instance, *args, **kwargs)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in update_port
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     return self.put(self.port_path % (port), body=body)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in put
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     headers=headers, params=params)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     headers=headers, params=params)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     self._handle_fault_response(status_code, replybody)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     exception_handler_v20(status_code, des_error_body)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api     message=message)
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api NeutronClientException: 404 Not Found
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api 
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api The resource could not be found.
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api 
  2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1499312/+subscriptions


References