yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #74167
[Bug 1747600] Re: Network cleanups don't work after max retries in a child cell
*** This bug is a duplicate of bug 1777505 ***
https://bugs.launchpad.net/bugs/1777505
This isn't a nova bug, it's a devstack bug, and already fixed.
** Also affects: devstack
Importance: Undecided
Status: New
** Changed in: nova
Status: Confirmed => Invalid
** This bug has been marked a duplicate of bug 1777505
nova_cell1_conf is not configured for [neutron]
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1747600
Title:
Network cleanups don't work after max retries in a child cell
Status in devstack:
New
Status in OpenStack Compute (nova):
Invalid
Bug description:
In cellv2 deployment wit supper-conductor and cell-conductor, the retry logic is done in cell-conductor, in devstack deployment, cell-conductor reads nova_cell1.conf and there is no Neutron endpoint info, also, we cannot get this information from context as network is not included here:
https://github.com/openstack/nova/blob/9519601401ee116a9197fe3b5d571495a96912e9/nova/context.py#L121
if the retry attempts exceeds, we will try to cleanup networks, and
this is happen in cell-conductor, so the below exception will happen:
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: DEBUG nova.network.neutronv2.api [None req-647f2fa0-f66b-4fa4-aef1-2e075b49dc75 admin admin] [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] deallocate_for_instance() {{(pid=27117) deallocate_for_
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [None req-647f2fa0-f66b-4fa4-aef1-2e075b49dc75 admin admin] [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] Failed to deallocate networks: EndpointNotFound: ['internal
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] Traceback (most recent call last):
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py", line 364, in _cleanup_allocated_networks
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] context, instance, requested_networks=requested_networks)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1253, in deallocate_for_instance
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] data = neutron.list_ports(**search_opts)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] ret = obj(*args, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] **_params)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] ret = obj(*args, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in list
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] for r in self._pagination(collection, path, **params):
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in _pagination
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] res = self.get(path, params=params)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] ret = obj(*args, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] headers=headers, params=params)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] ret = obj(*args, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] headers=headers, params=params)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] ret = obj(*args, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] headers=headers)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 342, in do_request
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] self._check_uri_length(url)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 335, in _check_uri_length
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] uri_len = len(self.endpoint_url) + len(url)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 349, in endpoint_url
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] return self.get_endpoint()
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 223, in get_endpoint
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] return self.session.get_endpoint(auth or self.auth, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 942, in get_endpoint
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] return auth.get_endpoint(self, **kwargs)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/nova/context.py", line 78, in get_endpoint
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] region_name=region_name)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 338, in url_for
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] endpoint_id=endpoint_id).url
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 400, in endpoint_data_for
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] raise exceptions.EndpointNotFound(msg)
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd] EndpointNotFound: ['internal', 'public'] endpoint for network service not found
Feb 06 08:36:00 donotdelete-allinone-zhuli-417094 nova-conductor[27093]: ERROR nova.conductor.manager [instance: cf53bacd-4cae-45f3-b0b4-549330a167bd]
Add neutron endpoint info to context will prevent this from happening.
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1747600/+subscriptions
References
