yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #71996
[Bug 1746674] Re: emulator_threads_policy=isolate doesn't work with multi numa node
Reviewed: https://review.openstack.org/539865
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=24d9e06ec68dfd0ead9bc56ad186e7b4e0acd8fb
Submitter: Zuul
Branch: master
commit 24d9e06ec68dfd0ead9bc56ad186e7b4e0acd8fb
Author: Tetsuro Nakamura <nakamura.tetsuro@xxxxxxxxxxxxx>
Date: Thu Feb 1 04:32:41 2018 +0900
add check before adding cpus to cpuset_reserved
There are some cases where None value is set to cpuset_reserved in
InstanceNUMATopology at _numa_fit_instance_cell() function in
hardware.py. However, libvirt driver treat cpuset_reserved value
as an iterate object when it constructs xml configuration.
To avoid a risk to get an error in libvirt driver, this patch adds
a check to see if the value is not None before adding the cpus
for emulator threads.
Change-Id: Iab3d950c4f4138118ac6a9fd98407eaadcb24d9e
Closes-Bug: #1746674
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746674
Title:
emulator_threads_policy=isolate doesn't work with multi numa node
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Description
===========
As described in test_multi_nodes_isolate() in https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/test_hardware.py#L3006-L3024,
numa_fit_instance_to_host() function returns None for cpuset_reserved for cells with id > 0.
---------
def test_multi_nodes_isolate(self):
host_topo = self._host_topology()
inst_topo = objects.InstanceNUMATopology(
emulator_threads_policy=(
fields.CPUEmulatorThreadsPolicy.ISOLATE),
cells=[objects.InstanceNUMACell(
id=0,
cpuset=set([0]), memory=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED),
objects.InstanceNUMACell(
id=1,
cpuset=set([1]), memory=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
self.assertEqual({0: 0}, inst_topo.cells[0].cpu_pinning)
self.assertEqual(set([1]), inst_topo.cells[0].cpuset_reserved)
self.assertEqual({1: 2}, inst_topo.cells[1].cpu_pinning)
self.assertIsNone(inst_topo.cells[1].cpuset_reserved)
--------
However, we are testing libvirt driver with non-None cpuset_reserved value in
https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/test_driver.py#L3052.
--------
def test_get_guest_config_numa_host_instance_isolated_emulator_threads(
self):
instance_topology = objects.InstanceNUMATopology(
emulator_threads_policy=(
fields.CPUEmulatorThreadsPolicy.ISOLATE),
cells=[
objects.InstanceNUMACell(
id=0, cpuset=set([0, 1]),
memory=1024, pagesize=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
cpu_pinning={0: 4, 1: 5},
cpuset_reserved=set([6])),
objects.InstanceNUMACell(
id=1, cpuset=set([2, 3]),
memory=1024, pagesize=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
cpu_pinning={2: 7, 3: 8},
cpuset_reserved=set([]))]) # <- this should be None!!
...(snip)...
--------
Actually, this causes errors when deploying VMs with multi numa nodes
with `emulator_threads_policy=isolate`.
Environment & Steps to reproduce
==================
1. Use devstack to build all-in-one OpenStack with libvirt/KVM driver
on a VM whose lscpu looks like this.
--------
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 2
NUMA node0 CPU(s): 0-3
NUMA node1 CPU(s): 4-7
--------
2. Try to Build a VM with multi numa nodes with
emulator_threads_policy=isolate.
--------
$ openstack flavor create c2r1024d1 --id 6 --ram 1024 --disk 1 --vcpu 2
$ openstack flavor set c2r1024d1 --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=isolate --property hw:numa_nodes=2
$ openstack server create test1 --image cirros-0.3.5-x86_64-disk --flavor c2r1024d1 --network private
--------
Expected & Actual result
===============
Expected: VM is built without an error
Actual: VM goes into an ERROR state with the following message.
--------
$ openstack server show test1
--------
...(snip)...
| fault | {u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac.', u'code': 500, u'details': u' File "/opt/stack/nova/nova/conductor/manager.py", line 578, in build_instances\\n raise exception.MaxRetriesExceeded(reason=msg)\\n', u'created': u'2018-02-07T09:48:43Z'} |
...(snip)...
Logs & Configs
==============
--------
$ sudo journalctl -a --unit devstack@n-* | tail -1000 | grep -e ERROR -e INFO
--------
...(snip)...
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3]], vCPUs
mapping: [(0, 0)]
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology reserved pCPUs: usable pCPUs: [[], [1], [2], [3]],
reserved pCPUs: set([1])
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs
mapping: [(1, 4)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Attempting claim on node
ubuntu01: memory 1024 MB, disk 1 GB, vcpus 2 CPU
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total memory: 5824 MB, used:
512.00 MB
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] memory limit not specified,
defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total disk: 90 GB, used: 0.00 GB
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] disk limit not specified,
defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total vcpu: 8 VCPU, used: 0.00
VCPU
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] vcpu limit not specified,
defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3]], vCPUs
mapping: [(0, 0)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology reserved pCPUs: usable pCPUs: [[], [1], [2], [3]],
reserved pCPUs: set([1])
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed
NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs
mapping: [(1, 4)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims
[None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Claim successful on node
ubuntu01
Feb 07 04:48:40 ubuntu01 nova-compute[9951]: INFO
nova.virt.libvirt.driver [None req-
ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance:
e8d7537f-9140-4ae6-bb3f-b98458a862ac] Creating image
Feb 07 04:48:41 ubuntu01 devstack@n-api.service[28408]: INFO
nova.api.openstack.compute.server_external_events [None req-a9935589
-09fd-45aa-b172-0e51ef79ca88 service nova] Creating event network-
changed:a67cd6cd-1553-4f36-bbcc-eea893f55941 for instance
e8d7537f-9140-4ae6-bb3f-b98458a862ac on ubuntu01
Feb 07 04:48:41 ubuntu01 devstack@n-api.service[28408]: INFO
nova.api.openstack.requestlog [None req-a9935589-09fd-
45aa-b172-0e51ef79ca88 service nova] 192.168.122.101 "POST
/compute/v2.1/os-server-external-events" status: 200 len: 179
microversion: 2.1 time: 0.080556
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [None req-a9935589-09fd-45aa-b172-0e51ef79ca88
service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
Instance failed to spawn: TypeError: 'NoneType' object is not iterable
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
Traceback (most recent call last):
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/compute/manager.py", line 2249, in
_build_resources
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
yield resources
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/compute/manager.py", line 2033, in
_build_and_run_instance
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
block_device_info=block_device_info)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3005, in
spawn
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
mdevs=mdevs)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5253, in
_get_guest_xml
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
context, mdevs)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4987, in
_get_guest_config
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
instance.numa_topology, flavor, allowed_cpus, image_meta)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4326, in
_get_guest_numa_config
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
object_numa_cell.cpuset_reserved)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
TypeError: 'NoneType' object is not iterable
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR
nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.compute.manager
[None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova]
[instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Terminating instance
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO
nova.virt.libvirt.driver [None req-a9935589-09fd-
45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6
-bb3f-b98458a862ac] Instance destroyed successfully.
Feb 07 04:48:42 ubuntu01 ovs-vsctl[18497]: ovs|00001|vsctl|INFO|Called
as ovs-vsctl --timeout=120 -- --if-exists del-port br-int
qvoa67cd6cd-15
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO os_vif [None
req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] Successfully
unplugged vif
VIFBridge(active=False,address=fa:16:3e:77:cf:80,bridge_name='qbra67cd6cd-15',has_traffic_filtering=True,id=a67cd6cd-1553-4f36
-bbcc-eea893f55941,network=Network(f17bbb1a-c562-4722-a95a-
5c117af6ee19),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapa67cd6cd-15')
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO
nova.virt.libvirt.driver [None req-a9935589-09fd-
45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6
-bb3f-b98458a862ac] Deleting instance files
/opt/stack/data/nova/instances/e8d7537f-9140-4ae6-bb3f-
b98458a862ac_del
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO
nova.virt.libvirt.driver [None req-a9935589-09fd-
45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6
-bb3f-b98458a862ac] Deletion of
/opt/stack/data/nova/instances/e8d7537f-9140-4ae6-bb3f-
b98458a862ac_del complete
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.compute.manager
[None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova]
[instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Took 0.19 seconds to
destroy the instance on the hypervisor.
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO
nova.scheduler.client.report [None req-a9935589-09fd-
45aa-b172-0e51ef79ca88 service nova] Deleted allocation for instance
e8d7537f-9140-4ae6-bb3f-b98458a862ac
Feb 07 04:48:42 ubuntu01 nova-conductor[9025]: ERROR
nova.scheduler.utils [None req-ec367a39-e9b3-4403-ab28-00519b5760ce
demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Error
from last host: ubuntu01 (node ubuntu01): [u'Traceback (most recent
call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py",
line 1851, in _do_build_and_run_instance\n filter_properties,
request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py",
line 2119, in _build_and_run_instance\n
instance_uuid=instance.uuid, reason=six.text_type(e))\n',
u"RescheduledException: Build of instance e8d7537f-9140-4ae6-bb3f-
b98458a862ac was re-scheduled: 'NoneType' object is not iterable\n"]
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: WARNING
nova.scheduler.utils [None req-ec367a39-e9b3-4403-ab28-00519b5760ce
demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Setting
instance to ERROR state.: MaxRetriesExceeded: Exceeded maximum number
of retries. Exhausted all hosts available for retrying build failures
for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac.
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [None req-ec367a39-e9b3-4403-ab28-00519b5760ce
demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Failed to
deallocate networks: EndpointNotFound: ['internal', 'public'] endpoint
for network service not found
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] Traceback (most recent call last):
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/conductor/manager.py", line
364, in _cleanup_allocated_networks
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] context, instance,
requested_networks=requested_networks)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 1253, in deallocate_for_instance
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] data = neutron.list_ports(**search_opts)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 786, in list_ports
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] **_params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 369, in list
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] for r in self._pagination(collection, path,
**params):
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 384, in _pagination
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] res = self.get(path, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 354, in get
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] headers=headers, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 331, in retry_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] headers=headers, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py",
line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/v2_0/client.py", line 282, in do_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] headers=headers)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/client.py", line 342, in do_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] self._check_uri_length(url)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/client.py", line 335, in _check_uri_length
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] uri_len = len(self.endpoint_url) + len(url)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/neutronclient/client.py", line 349, in endpoint_url
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] return self.get_endpoint()
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/keystoneauth1/adapter.py", line 223, in get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] return self.session.get_endpoint(auth or self.auth,
**kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/keystoneauth1/session.py", line 942, in get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] return auth.get_endpoint(self, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/opt/stack/nova/nova/context.py", line 78, in
get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] region_name=region_name)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/keystoneauth1/access/service_catalog.py", line 338, in
url_for
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] endpoint_id=endpoint_id).url
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] File "/usr/local/lib/python2.7/dist-
packages/keystoneauth1/access/service_catalog.py", line 400, in
endpoint_data_for
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] raise exceptions.EndpointNotFound(msg)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR
nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-
b98458a862ac] EndpointNotFound: ['internal', 'public'] endpoint for
network service not found
...(snip)...
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746674/+subscriptions
References
