yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1367243] [NEW] Bad error report when trying to connect VM to an overcrouded Network

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yair Fried <yfried@xxxxxxxxxx>
Date: Tue, 09 Sep 2014 11:36:53 -0000
Reply-to: Bug 1367243 <1367243@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

trying to create a Port when the network has no more addresses, Neutron returns this error:
No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886.

However, when trying to create a VM in a network that has no more addresses, the VM is created with Error details:
"No valid host was found"

That's because the the compute agent is seeing the PortLimitExceeded error from neutron and reports the the scheduler.
The scheduler mistakes that for Hypervisor limit and tries another compute (which fails for the same reason) and finally reports the error as "No host found", while the error has to be "No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886"


Neutron however, doesn't register any errors for this flow.

Recreate:
1. create a subnet with 31 mask bits (which means no ports available for VMs)
2. Boot a VM. errors will be seen in nova-compute, and nova-scheduler

nova compute:
2014-09-09 14:30:11.517 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Starting instance...
2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Attempting claim: memory 64 MB, disk 0 GB, VCPUs 1
2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total memory: 31952 MB, used: 1152.00 MB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] memory limit: 47928.00 MB, free: 46776.00 MB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total disk: 442 GB, used: 0.00 GB
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] disk limit not specified, defaulting to unlimited
2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total CPUs: 24 VCPUs, used: 10.00 VCPUs
2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] CPUs limit: 384.00 VCPUs, free: 374.00 VCPUs
2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Claim successful
2014-09-09 14:30:12.141 123433 WARNING nova.network.neutronv2.api [-] Neutron error: quota exceeded
2014-09-09 14:30:12.142 123433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager Traceback (most recent call last):
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     LOG.exception(msg, port_id)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     raise exception.PortLimitExceeded()
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager
2014-09-09 14:30:12.158 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Creating image
2014-09-09 14:30:12.261 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Instance failed to spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
2014-09-09 14:30:12.326 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Terminating instance
2014-09-09 14:30:12.334 123433 ERROR nova.virt.libvirt.driver [-] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] During wait destroy, instance disappeared.
2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deleting instance files /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb
2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deletion of /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb complete
2014-09-09 14:30:12.558 123433 INFO nova.network.neutronv2.api [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Unable to reset device ID for port None
2014-09-09 14:30:12.644 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error: Maximum number of ports exceeded
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     set_access_ip=set_access_ip)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return function(self, context, *args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
2014-09-09 14:30:33.435 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
2014-09-09 14:30:34.137 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com
2014-09-09 14:30:51.734 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
2014-09-09 14:30:52.406 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
2014-09-09 14:30:52.429 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com

nova-scheduler
2014-09-09 14:30:11.382 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
2014-09-09 14:30:11.387 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Choosing host WeighedHost [host: puma48.scl.lab.tlv.redhat.com, weight: 1.0] for instance 8f50312c-f47e-4793-aff7-a890f20ee2bb
2014-09-09 14:30:12.665 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
2014-09-09 14:30:12.665 123181 ERROR nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error from last host: puma48.scl.lab.tlv.redhat.com (node puma48.scl.lab.tlv.redhat.com): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn\n    admin_pass=admin_password)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image\n    instance, network_info, admin_pass, files, suffix)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data\n    net = netutils.get_injected_network_template(network_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template\n    if not (network_info and template):\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait\n    self[:] = self._gt.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait\n    current.throw(*self._exc)\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance\n    LOG.exception(msg, port_id)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance\n    security_group_ids, available_macs, dhcp_opts)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port\n    raise exception.PortLimitExceeded()\n', u'PortLimitExceeded: Maximum number of ports exceeded\n']
2014-09-09 14:30:12.669 123181 INFO nova.filters [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Filter RetryFilter returned 0 hosts
2014-09-09 14:30:12.669 123181 WARNING nova.scheduler.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Setting instance to ERROR state.

[root@puma48 ~(keystone_admin)]# nova show 8f50312c-f47e-4793-aff7-a890f20ee2bb
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                                                                |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                               |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                 |
| OS-EXT-SRV-ATTR:host                 | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
| OS-EXT-SRV-ATTR:instance_name        | instance-000003d7                                                                                                                                                                    |
| OS-EXT-STS:power_state               | 0                                                                                                                                                                                    |
| OS-EXT-STS:task_state                | -                                                                                                                                                                                    |
| OS-EXT-STS:vm_state                  | error                                                                                                                                                                                |
| OS-SRV-USG:launched_at               | -                                                                                                                                                                                    |
| OS-SRV-USG:terminated_at             | -                                                                                                                                                                                    |
| accessIPv4                           |                                                                                                                                                                                      |
| accessIPv6                           |                                                                                                                                                                                      |
| config_drive                         |                                                                                                                                                                                      |
| created                              | 2014-09-09T11:30:11Z                                                                                                                                                                 |
| fault                                | {"message": "No valid host was found. ", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\", line 108, in schedule_run_instance |
|                                      |     raise exception.NoValidHost(reason=\"\")                                                                                                                                         |
|                                      | ", "created": "2014-09-09T11:30:12Z"}                                                                                                                                                |
| flavor                               | m1.nano (42)                                                                                                                                                                         |
| hostId                               | 624ae58ea64f1cfb0ea62f3b0eced7799b21a3e55414343a6352ca67                                                                                                                             |
| id                                   | 8f50312c-f47e-4793-aff7-a890f20ee2bb                                                                                                                                                 |
| image                                | cirros (49948054-e169-4e1b-a5b4-6441959ef75c)                                                                                                                                        |
| key_name                             | -                                                                                                                                                                                    |
| metadata                             | {}                                                                                                                                                                                   |
| name                                 | test_VM                                                                                                                                                                              |
| os-extended-volumes:volumes_attached | []                                                                                                                                                                                   |
| status                               | ERROR                                                                                                                                                                                |
| tenant_id                            | a0a2f1afe57d422887b48c204d536df0                                                                                                                                                     |
| updated                              | 2014-09-09T11:30:12Z                                                                                                                                                                 |
| user_id                              | e9bb9ce3fbe344e9b49182a13dcfb9c3                                                                                                                                                     |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: network

** Description changed:

  trying to create a Port when the network has no more addresses, Neutron returns this error:
  No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886.
  
  However, when trying to create a VM in a network that has no more addresses, the VM is created with Error details:
  "No valid host was found"
  
  That's because the the compute agent is seeing the PortLimitExceeded error from neutron and reports the the scheduler.
  The scheduler mistakes that for Hypervisor limit and tries another compute (which fails for the same reason) and finally reports the error as "No host found", while the error has to be "No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886"
  
+ 
+ Neutron however, doesn't register any errors for this flow.
+ 
+ Recreate:
+ 1. create a subnet with 31 mask bits (which means no ports available for VMs)
+ 2. Boot a VM. errors will be seen in nova-compute, and nova-scheduler
  
  nova compute:
  2014-09-09 14:30:11.517 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Starting instance...
  2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Attempting claim: memory 64 MB, disk 0 GB, VCPUs 1
  2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total memory: 31952 MB, used: 1152.00 MB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] memory limit: 47928.00 MB, free: 46776.00 MB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total disk: 442 GB, used: 0.00 GB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] disk limit not specified, defaulting to unlimited
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total CPUs: 24 VCPUs, used: 10.00 VCPUs
  2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] CPUs limit: 384.00 VCPUs, free: 374.00 VCPUs
  2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Claim successful
  2014-09-09 14:30:12.141 123433 WARNING nova.network.neutronv2.api [-] Neutron error: quota exceeded
  2014-09-09 14:30:12.142 123433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager Traceback (most recent call last):
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager PortLimitExceeded: Maximum number of ports exceeded
- 2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager 
+ 2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager
  2014-09-09 14:30:12.158 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Creating image
  2014-09-09 14:30:12.261 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Instance failed to spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
- 2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] 
+ 2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
  2014-09-09 14:30:12.326 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Terminating instance
  2014-09-09 14:30:12.334 123433 ERROR nova.virt.libvirt.driver [-] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] During wait destroy, instance disappeared.
  2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deleting instance files /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb
  2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deletion of /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb complete
  2014-09-09 14:30:12.558 123433 INFO nova.network.neutronv2.api [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Unable to reset device ID for port None
  2014-09-09 14:30:12.644 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error: Maximum number of ports exceeded
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     set_access_ip=set_access_ip)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return function(self, context, *args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
- 2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] 
+ 2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
  2014-09-09 14:30:33.435 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
  2014-09-09 14:30:34.137 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com
  2014-09-09 14:30:51.734 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
  2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
  2014-09-09 14:30:52.406 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
  2014-09-09 14:30:52.429 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com
  
- 
- 
  nova-scheduler
  2014-09-09 14:30:11.382 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
  2014-09-09 14:30:11.387 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Choosing host WeighedHost [host: puma48.scl.lab.tlv.redhat.com, weight: 1.0] for instance 8f50312c-f47e-4793-aff7-a890f20ee2bb
  2014-09-09 14:30:12.665 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
  2014-09-09 14:30:12.665 123181 ERROR nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error from last host: puma48.scl.lab.tlv.redhat.com (node puma48.scl.lab.tlv.redhat.com): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn\n    admin_pass=admin_password)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image\n    instance, network_info, admin_pass, files, suffix)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data\n    net = netutils.get_injected_network_template(network_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template\n    if not (network_info and template):\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait\n    self[:] = self._gt.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait\n    current.throw(*self._exc)\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance\n    LOG.exception(msg, port_id)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance\n    security_group_ids, available_macs, dhcp_opts)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port\n    raise exception.PortLimitExceeded()\n', u'PortLimitExceeded: Maximum number of ports exceeded\n']
  2014-09-09 14:30:12.669 123181 INFO nova.filters [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Filter RetryFilter returned 0 hosts
  2014-09-09 14:30:12.669 123181 WARNING nova.scheduler.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Setting instance to ERROR state.
- 
  
  [root@puma48 ~(keystone_admin)]# nova show 8f50312c-f47e-4793-aff7-a890f20ee2bb
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                                                                |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                               |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                 |
  | OS-EXT-SRV-ATTR:host                 | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
  | OS-EXT-SRV-ATTR:instance_name        | instance-000003d7                                                                                                                                                                    |
  | OS-EXT-STS:power_state               | 0                                                                                                                                                                                    |
  | OS-EXT-STS:task_state                | -                                                                                                                                                                                    |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                                                                |
  | OS-SRV-USG:launched_at               | -                                                                                                                                                                                    |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                                                                    |
  | accessIPv4                           |                                                                                                                                                                                      |
  | accessIPv6                           |                                                                                                                                                                                      |
  | config_drive                         |                                                                                                                                                                                      |
  | created                              | 2014-09-09T11:30:11Z                                                                                                                                                                 |
  | fault                                | {"message": "No valid host was found. ", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\", line 108, in schedule_run_instance |
  |                                      |     raise exception.NoValidHost(reason=\"\")                                                                                                                                         |
  |                                      | ", "created": "2014-09-09T11:30:12Z"}                                                                                                                                                |
  | flavor                               | m1.nano (42)                                                                                                                                                                         |
  | hostId                               | 624ae58ea64f1cfb0ea62f3b0eced7799b21a3e55414343a6352ca67                                                                                                                             |
  | id                                   | 8f50312c-f47e-4793-aff7-a890f20ee2bb                                                                                                                                                 |
  | image                                | cirros (49948054-e169-4e1b-a5b4-6441959ef75c)                                                                                                                                        |
  | key_name                             | -                                                                                                                                                                                    |
  | metadata                             | {}                                                                                                                                                                                   |
  | name                                 | test_VM                                                                                                                                                                              |
  | os-extended-volumes:volumes_attached | []                                                                                                                                                                                   |
  | status                               | ERROR                                                                                                                                                                                |
  | tenant_id                            | a0a2f1afe57d422887b48c204d536df0                                                                                                                                                     |
  | updated                              | 2014-09-09T11:30:12Z                                                                                                                                                                 |
  | user_id                              | e9bb9ce3fbe344e9b49182a13dcfb9c3                                                                                                                                                     |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1367243

Title:
  Bad error report when trying to connect VM to an overcrouded Network

Status in OpenStack Compute (Nova):
  New

Bug description:
  trying to create a Port when the network has no more addresses, Neutron returns this error:
  No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886.

  However, when trying to create a VM in a network that has no more addresses, the VM is created with Error details:
  "No valid host was found"

  That's because the the compute agent is seeing the PortLimitExceeded error from neutron and reports the the scheduler.
  The scheduler mistakes that for Hypervisor limit and tries another compute (which fails for the same reason) and finally reports the error as "No host found", while the error has to be "No more IP addresses available on network a4e997dc-ba2e-4444-9394-cfd89f670886"

  
  Neutron however, doesn't register any errors for this flow.

  Recreate:
  1. create a subnet with 31 mask bits (which means no ports available for VMs)
  2. Boot a VM. errors will be seen in nova-compute, and nova-scheduler

  nova compute:
  2014-09-09 14:30:11.517 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Starting instance...
  2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Attempting claim: memory 64 MB, disk 0 GB, VCPUs 1
  2014-09-09 14:30:11.608 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total memory: 31952 MB, used: 1152.00 MB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] memory limit: 47928.00 MB, free: 46776.00 MB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total disk: 442 GB, used: 0.00 GB
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] disk limit not specified, defaulting to unlimited
  2014-09-09 14:30:11.609 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Total CPUs: 24 VCPUs, used: 10.00 VCPUs
  2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] CPUs limit: 384.00 VCPUs, free: 374.00 VCPUs
  2014-09-09 14:30:11.610 123433 AUDIT nova.compute.claims [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Claim successful
  2014-09-09 14:30:12.141 123433 WARNING nova.network.neutronv2.api [-] Neutron error: quota exceeded
  2014-09-09 14:30:12.142 123433 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager Traceback (most recent call last):
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager PortLimitExceeded: Maximum number of ports exceeded
  2014-09-09 14:30:12.142 123433 TRACE nova.compute.manager
  2014-09-09 14:30:12.158 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Creating image
  2014-09-09 14:30:12.261 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Instance failed to spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
  2014-09-09 14:30:12.261 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
  2014-09-09 14:30:12.326 123433 AUDIT nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Terminating instance
  2014-09-09 14:30:12.334 123433 ERROR nova.virt.libvirt.driver [-] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] During wait destroy, instance disappeared.
  2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deleting instance files /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb
  2014-09-09 14:30:12.425 123433 INFO nova.virt.libvirt.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Deletion of /var/lib/nova/instances/8f50312c-f47e-4793-aff7-a890f20ee2bb complete
  2014-09-09 14:30:12.558 123433 INFO nova.network.neutronv2.api [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Unable to reset device ID for port None
  2014-09-09 14:30:12.644 123433 ERROR nova.compute.manager [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error: Maximum number of ports exceeded
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Traceback (most recent call last):
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     set_access_ip=set_access_ip)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return function(self, context, *args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     block_device_info)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     admin_pass=admin_password)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     instance, network_info, admin_pass, files, suffix)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     net = netutils.get_injected_network_template(network_info)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     if not (network_info and template):
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._sync_wrapper(fn, *args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     self[:] = self._gt.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     return self._exit_event.wait()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     current.throw(*self._exc)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     result = function(*args, **kwargs)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     dhcp_options=dhcp_options)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     LOG.exception(msg, port_id)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     six.reraise(self.type_, self.value, self.tb)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     security_group_ids, available_macs, dhcp_opts)
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]     raise exception.PortLimitExceeded()
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] PortLimitExceeded: Maximum number of ports exceeded
  2014-09-09 14:30:12.644 123433 TRACE nova.compute.manager [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb]
  2014-09-09 14:30:33.435 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
  2014-09-09 14:30:34.112 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
  2014-09-09 14:30:34.137 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com
  2014-09-09 14:30:51.734 123433 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 30736
  2014-09-09 14:30:52.405 123433 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 442
  2014-09-09 14:30:52.406 123433 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 13
  2014-09-09 14:30:52.429 123433 INFO nova.compute.resource_tracker [-] Compute_service record updated for puma48.scl.lab.tlv.redhat.com:puma48.scl.lab.tlv.redhat.com

  nova-scheduler
  2014-09-09 14:30:11.382 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
  2014-09-09 14:30:11.387 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Choosing host WeighedHost [host: puma48.scl.lab.tlv.redhat.com, weight: 1.0] for instance 8f50312c-f47e-4793-aff7-a890f20ee2bb
  2014-09-09 14:30:12.665 123181 INFO nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Attempting to build 1 instance(s) uuids: [u'8f50312c-f47e-4793-aff7-a890f20ee2bb']
  2014-09-09 14:30:12.665 123181 ERROR nova.scheduler.filter_scheduler [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Error from last host: puma48.scl.lab.tlv.redhat.com (node puma48.scl.lab.tlv.redhat.com): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2282, in spawn\n    admin_pass=admin_password)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2733, in _create_image\n    instance, network_info, admin_pass, files, suffix)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2551, in _inject_data\n    net = netutils.get_injected_network_template(network_info)\n', u'  File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template\n    if not (network_info and template):\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 420, in __len__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 407, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 439, in wait\n    self[:] = self._gt.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 120, in wait\n    current.throw(*self._exc)\n', u'  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1528, in _allocate_network_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance\n    LOG.exception(msg, port_id)\n', u'  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance\n    security_group_ids, available_macs, dhcp_opts)\n', u'  File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port\n    raise exception.PortLimitExceeded()\n', u'PortLimitExceeded: Maximum number of ports exceeded\n']
  2014-09-09 14:30:12.669 123181 INFO nova.filters [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] Filter RetryFilter returned 0 hosts
  2014-09-09 14:30:12.669 123181 WARNING nova.scheduler.driver [req-19bf0ebe-4f8d-47bb-9506-617ae54cd6b4 e9bb9ce3fbe344e9b49182a13dcfb9c3 a0a2f1afe57d422887b48c204d536df0] [instance: 8f50312c-f47e-4793-aff7-a890f20ee2bb] Setting instance to ERROR state.

  [root@puma48 ~(keystone_admin)]# nova show 8f50312c-f47e-4793-aff7-a890f20ee2bb
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                                                                |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                               |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                 |
  | OS-EXT-SRV-ATTR:host                 | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | puma48.scl.lab.tlv.redhat.com                                                                                                                                                        |
  | OS-EXT-SRV-ATTR:instance_name        | instance-000003d7                                                                                                                                                                    |
  | OS-EXT-STS:power_state               | 0                                                                                                                                                                                    |
  | OS-EXT-STS:task_state                | -                                                                                                                                                                                    |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                                                                |
  | OS-SRV-USG:launched_at               | -                                                                                                                                                                                    |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                                                                    |
  | accessIPv4                           |                                                                                                                                                                                      |
  | accessIPv6                           |                                                                                                                                                                                      |
  | config_drive                         |                                                                                                                                                                                      |
  | created                              | 2014-09-09T11:30:11Z                                                                                                                                                                 |
  | fault                                | {"message": "No valid host was found. ", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\", line 108, in schedule_run_instance |
  |                                      |     raise exception.NoValidHost(reason=\"\")                                                                                                                                         |
  |                                      | ", "created": "2014-09-09T11:30:12Z"}                                                                                                                                                |
  | flavor                               | m1.nano (42)                                                                                                                                                                         |
  | hostId                               | 624ae58ea64f1cfb0ea62f3b0eced7799b21a3e55414343a6352ca67                                                                                                                             |
  | id                                   | 8f50312c-f47e-4793-aff7-a890f20ee2bb                                                                                                                                                 |
  | image                                | cirros (49948054-e169-4e1b-a5b4-6441959ef75c)                                                                                                                                        |
  | key_name                             | -                                                                                                                                                                                    |
  | metadata                             | {}                                                                                                                                                                                   |
  | name                                 | test_VM                                                                                                                                                                              |
  | os-extended-volumes:volumes_attached | []                                                                                                                                                                                   |
  | status                               | ERROR                                                                                                                                                                                |
  | tenant_id                            | a0a2f1afe57d422887b48c204d536df0                                                                                                                                                     |
  | updated                              | 2014-09-09T11:30:12Z                                                                                                                                                                 |
  | user_id                              | e9bb9ce3fbe344e9b49182a13dcfb9c3                                                                                                                                                     |
  +--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1367243/+subscriptions
Follow ups

[Bug 1367243] Re: Bad error report when trying to connect VM to an overcrouded Network
From: Rodolfo Alonso, 2022-10-19
[Bug 1367243] Re: Bad error report when trying to connect VM to an overcrouded Network
From: Markus Zoeller (markus_z), 2016-07-05
[Bug 1367243] Re: Bad error report when trying to connect VM to an overcrouded Network
From: Sean Dague, 2014-09-09
[Bug 1367243] [NEW] Bad error report when trying to connect VM to an overcrouded Network
From: Yair Fried, 2014-09-09
References

[Bug 1367243] [NEW] Bad error report when trying to connect VM to an overcrouded Network
From: Yair Fried, 2014-09-09