yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #33683
[Bug 1464554] [NEW] instance failed to spawn with external network
Public bug reported:
I'm trying to launch an instance with external network but result in
failed status. But instances with internal network are fine.
F ollowing is the nova-compute.log from compute node
2015-06-12 15:22:50.899 3121 INFO nova.compute.manager [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Starting instance...
2015-06-12 15:22:50.997 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Attempting claim: memory 2048 MB, disk 50 GB
2015-06-12 15:22:50.997 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Total memory: 515884 MB, used: 2560.00 MB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] memory limit: 773826.00 MB, free: 771266.00 MB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Total disk: 1144 GB, used: 50.00 GB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] disk limit not specified, defaulting to unlimited
2015-06-12 15:22:51.023 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Claim successful
2015-06-12 15:22:51.134 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.270 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.470 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Creating image
2015-06-12 15:22:51.760 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.993 3121 ERROR nova.compute.manager [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Instance failed to spawn
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Traceback (most recent call last):
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] yield resources
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] block_device_info=block_device_info)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2351, in spawn
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] write_to_disk=True)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4172, in _get_guest_xml
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] context)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4043, in _get_guest_config
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] flavor, virt_type)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 374, in get_config
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] _("Unexpected vif_type=%s") % vif_type)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] NovaException: Unexpected vif_type=binding_failed
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772]
2015-06-12 15:22:51.995 3121 INFO nova.compute.manager [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Terminating instance
2015-06-12 15:22:52.002 3121 INFO nova.virt.libvirt.driver [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] During wait destroy, instance disappeared.
2015-06-12 15:22:52.015 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Deleting instance files /var/lib/nova/instances/8da458b4-c064-47c8-a1bb-aad4e4400772_del
2015-06-12 15:22:52.017 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Deletion of /var/lib/nova/instances/8da458b4-c064-47c8-a1bb-aad4e4400772_del complete
neuton-server.log from neutron-server node
2015-06-12 15:22:51.437 6583 INFO neutron.wsgi [req-bd04266c-e4a1-4c5b-be8d-52a30e32b46a ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "GET /v2.0/security-groups.json?tenant_id=700e680640e0415faf591e950cdb42d0 HTTP/1.1" 200 1765 0.017019
2015-06-12 15:22:51.440 6583 INFO neutron.wsgi [req-835c2aab-e3f5-44bb-8586-8c37b6e6b228 ] 11.11.176.35 - - [12/Jun/2015 15:22:51] "GET /v2.0/extensions.json HTTP/1.1" 200 4806 0.002097
2015-06-12 15:22:51.518 6583 INFO neutron.callbacks.manager [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Notify callbacks for port, after_create
2015-06-12 15:22:51.518 6583 INFO neutron.callbacks.manager [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Calling callback neutron.db.l3_dvrscheduler_db._notify_l3_agent_new_port
2015-06-12 15:22:51.528 6583 ERROR neutron.plugins.ml2.managers [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Failed to bind port 68a23b41-a9dd-4d4b-88b0-359834b75f97 on host openstack-kvm1
2015-06-12 15:22:51.529 6583 ERROR neutron.plugins.ml2.managers [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Failed to bind port 68a23b41-a9dd-4d4b-88b0-359834b75f97 on host openstack-kvm1
2015-06-12 15:22:51.571 6583 INFO neutron.wsgi [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "POST /v2.0/ports.json HTTP/1.1" 201 928 0.130556
2015-06-12 15:22:51.594 6583 INFO neutron.wsgi [req-c88f52ef-e3fc-4245-9225-980ad67c8718 ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "GET /v2.0/ports.json?tenant_id=700e680640e0415faf591e950cdb42d0&device_id=8da458b4-c064-47c8-a1bb-aad4e4400772 HTTP/1.1" 200 926 0.019260
nova-scheduler.log from nova-controller node
2015-06-12 15:22:52.271 2673 INFO nova.filters [req-6b9424ce-2eda-469e-
9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66
700e680640e0415faf591e950cdb42d0 - - -] Filter RetryFilter returned 0
hosts
nova-conductor.log from nova-controller node
2015-06-12 15:22:52.252 1191 ERROR nova.scheduler.utils [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Error from last host: openstack-kvm1 (node openstack-kvm1): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2362, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 8da458b4-c064-47c8-a1bb-aad4e4400772 was re-scheduled: Unexpected vif_type=binding_failed\n']
2015-06-12 15:22:52.275 1191 WARNING nova.scheduler.utils [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 86, in select_destinations
filter_properties)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 80, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts
available.
2015-06-12 15:22:52.276 1191 WARNING nova.scheduler.utils [req-6b9424ce-
2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66
700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8
-a1bb-aad4e4400772] Setting instance to ERROR state.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1464554
Title:
instance failed to spawn with external network
Status in OpenStack Neutron (virtual network service):
New
Bug description:
I'm trying to launch an instance with external network but result in
failed status. But instances with internal network are fine.
F ollowing is the nova-compute.log from compute node
2015-06-12 15:22:50.899 3121 INFO nova.compute.manager [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Starting instance...
2015-06-12 15:22:50.997 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Attempting claim: memory 2048 MB, disk 50 GB
2015-06-12 15:22:50.997 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Total memory: 515884 MB, used: 2560.00 MB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] memory limit: 773826.00 MB, free: 771266.00 MB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Total disk: 1144 GB, used: 50.00 GB
2015-06-12 15:22:50.998 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] disk limit not specified, defaulting to unlimited
2015-06-12 15:22:51.023 3121 INFO nova.compute.claims [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Claim successful
2015-06-12 15:22:51.134 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.270 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.470 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Creating image
2015-06-12 15:22:51.760 3121 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack-kvm1', 'openstack-kvm1')
2015-06-12 15:22:51.993 3121 ERROR nova.compute.manager [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Instance failed to spawn
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Traceback (most recent call last):
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] yield resources
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] block_device_info=block_device_info)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2351, in spawn
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] write_to_disk=True)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4172, in _get_guest_xml
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] context)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4043, in _get_guest_config
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] flavor, virt_type)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 374, in get_config
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] _("Unexpected vif_type=%s") % vif_type)
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] NovaException: Unexpected vif_type=binding_failed
2015-06-12 15:22:51.993 3121 TRACE nova.compute.manager [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772]
2015-06-12 15:22:51.995 3121 INFO nova.compute.manager [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Terminating instance
2015-06-12 15:22:52.002 3121 INFO nova.virt.libvirt.driver [-] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] During wait destroy, instance disappeared.
2015-06-12 15:22:52.015 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Deleting instance files /var/lib/nova/instances/8da458b4-c064-47c8-a1bb-aad4e4400772_del
2015-06-12 15:22:52.017 3121 INFO nova.virt.libvirt.driver [req-02283432-2fd3-4835-a548-8c5bd74f4340 - - - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Deletion of /var/lib/nova/instances/8da458b4-c064-47c8-a1bb-aad4e4400772_del complete
neuton-server.log from neutron-server node
2015-06-12 15:22:51.437 6583 INFO neutron.wsgi [req-bd04266c-e4a1-4c5b-be8d-52a30e32b46a ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "GET /v2.0/security-groups.json?tenant_id=700e680640e0415faf591e950cdb42d0 HTTP/1.1" 200 1765 0.017019
2015-06-12 15:22:51.440 6583 INFO neutron.wsgi [req-835c2aab-e3f5-44bb-8586-8c37b6e6b228 ] 11.11.176.35 - - [12/Jun/2015 15:22:51] "GET /v2.0/extensions.json HTTP/1.1" 200 4806 0.002097
2015-06-12 15:22:51.518 6583 INFO neutron.callbacks.manager [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Notify callbacks for port, after_create
2015-06-12 15:22:51.518 6583 INFO neutron.callbacks.manager [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Calling callback neutron.db.l3_dvrscheduler_db._notify_l3_agent_new_port
2015-06-12 15:22:51.528 6583 ERROR neutron.plugins.ml2.managers [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Failed to bind port 68a23b41-a9dd-4d4b-88b0-359834b75f97 on host openstack-kvm1
2015-06-12 15:22:51.529 6583 ERROR neutron.plugins.ml2.managers [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] Failed to bind port 68a23b41-a9dd-4d4b-88b0-359834b75f97 on host openstack-kvm1
2015-06-12 15:22:51.571 6583 INFO neutron.wsgi [req-523b2d9c-8026-4c9c-9ba0-cfd30833e3b8 ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "POST /v2.0/ports.json HTTP/1.1" 201 928 0.130556
2015-06-12 15:22:51.594 6583 INFO neutron.wsgi [req-c88f52ef-e3fc-4245-9225-980ad67c8718 ] 11.11.176.41 - - [12/Jun/2015 15:22:51] "GET /v2.0/ports.json?tenant_id=700e680640e0415faf591e950cdb42d0&device_id=8da458b4-c064-47c8-a1bb-aad4e4400772 HTTP/1.1" 200 926 0.019260
nova-scheduler.log from nova-controller node
2015-06-12 15:22:52.271 2673 INFO nova.filters [req-6b9424ce-2eda-
469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66
700e680640e0415faf591e950cdb42d0 - - -] Filter RetryFilter returned 0
hosts
nova-conductor.log from nova-controller node
2015-06-12 15:22:52.252 1191 ERROR nova.scheduler.utils [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8-a1bb-aad4e4400772] Error from last host: openstack-kvm1 (node openstack-kvm1): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2362, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 8da458b4-c064-47c8-a1bb-aad4e4400772 was re-scheduled: Unexpected vif_type=binding_failed\n']
2015-06-12 15:22:52.275 1191 WARNING nova.scheduler.utils [req-6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66 700e680640e0415faf591e950cdb42d0 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 86, in select_destinations
filter_properties)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 80, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts
available.
2015-06-12 15:22:52.276 1191 WARNING nova.scheduler.utils [req-
6b9424ce-2eda-469e-9cba-63807a8643c9 28740e72adf04dde88a2b2a1aa701e66
700e680640e0415faf591e950cdb42d0 - - -] [instance: 8da458b4-c064-47c8
-a1bb-aad4e4400772] Setting instance to ERROR state.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1464554/+subscriptions
Follow ups
References