yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1752289] Re: ServiceCatalog does not contain "network" service

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 28 Feb 2018 13:29:32 -0000
Reply-to: Bug 1752289 <1752289@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is likely related to
https://github.com/openstack/nova/commit/6cde77ebbab85bc8ccd2ab7ad977b1d4af4a13fa
if it's a regression in Queens.

** Tags added: queens-backport-potential

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Changed in: nova/queens
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => High

** Changed in: nova/queens
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1752289

Title:
  ServiceCatalog does not contain "network" service

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) queens series:
  Confirmed

Bug description:
  On SLE12SP3, openstack-nova 17.0.0.0~xrc2~dev160-1.1, I try to boot an
  cirros instance and get:

  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance failed to spawn: EndpointNotFound: ['internal', 'public'] endpoint for network service not found
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] Traceback (most recent call last):
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     yield resources
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2019, in _build_and_run_instance
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     block_device_info=block_device_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3011, in spawn
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     mdevs=mdevs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5256, in _get_guest_xml
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     network_info_str = str(network_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return self._sync_wrapper(fn, *args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in _sync_wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     self.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     self[:] = self._gt.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return self._exit_event.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     current.throw(*self._exc)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     result = function(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return func(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in _allocate_network_async
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     six.reraise(*exc_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1397, in _allocate_network_async
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     bind_host_id=bind_host_id)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 918, in allocate_for_instance
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     context, instance, neutron, requested_networks, ordered_networks)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 758, in _validate_requested_network_ids
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     auto_allocate=auto_allocate)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 383, in _get_available_networks
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     nets = neutron.list_networks(**search_opts).get('networks', [])
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 809, in list_networks
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     **_params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     for r in self._pagination(collection, path, **params):
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     res = self.get(path, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     headers=headers, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     headers=headers, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     headers=headers)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 342, in do_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     self._check_uri_length(url)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 335, in _check_uri_length
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     uri_len = len(self.endpoint_url) + len(url)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 349, in endpoint_url
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return self.get_endpoint()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return self.session.get_endpoint(auth or self.auth, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     return auth.get_endpoint(self, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     region_name=region_name)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 338, in url_for
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     endpoint_id=endpoint_id).url
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 399, in endpoint_data_for
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]     raise exceptions.EndpointNotFound(msg)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] EndpointNotFound: ['internal', 'public'] endpoint for network service not found
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] 
  2018-02-28 10:24:07.555 10768 INFO nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Terminating instance
  2018-02-28 10:24:07.560 10768 INFO nova.virt.libvirt.driver [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance destroyed successfully.

  
  /etc/nova/nova.conf.d/100-nova.conf contains:
  [keystone_authtoken]
  auth_type = password
  username = nova
  password = openstack
  user_domain_name = Default
  project_name = service
  project_domain_name = Default
  auth_url = http://192.168.122.80:5000/
  [neutron]
  auth_type = password
  username = neutron
  password = openstack
  user_domain_name = Default
  user_domain_id = default
  project_name = service
  project_domain_name = Default
  project_domain_id = default
  auth_url = http://192.168.122.80:5000/
  service_metadata_proxy = True
  metadata_proxy_shared_secret = a3f876fb3286e8bf304c01064e080fae

  
  Theses credentials are correct. I can list the full service catalog when using these creds.
  The problem seems, that the context which is used has only a subset of the complete service_catalog:

  > /usr/lib/python2.7/site-packages/nova/compute/manager.py(1440)_build_networks_for_instance()
  -> macs = self.driver.macs_for_instance(instance)
  (Pdb) pp context.to_dict()
  {'auth_token': u'gAAAAABaloMiR0qjo1BapDgt93AP5MLLx_VAXY-8GVUHR4iXZkwXmZ-qdnLcba2PKmQETAYqSWEi03ALlvgOCKB1CU6-XWsOXMuqFT9Urnm4D3Zd9Ss_J3QNRAC8furxu7UGtP5XbHymcE8xfjbPDfQNaHfJhTJo511tPd57TkIOcKWbff7JA_M',
   'domain': None,
   'global_request_id': None,
   'instance_lock_checked': False,
   'is_admin': True,
   'is_admin_project': True,
   'project': u'd9685d1130d74a73af6ee213c421d9de',
   'project_domain': u'default',
   'project_id': u'd9685d1130d74a73af6ee213c421d9de',
   'project_name': u'admin',
   'quota_class': None,
   'read_deleted': u'no',
   'read_only': False,
   'remote_address': u'192.168.122.80',
   'request_id': u'req-53fa6935-f60d-4e07-bc75-14b6a5336330',
   'resource_uuid': None,
   'roles': [u'admin'],
   'service_catalog': [{u'endpoints': [{u'adminURL': u'http://192.168.122.80:8780/',
                                        u'internalURL': u'http://192.168.122.80:8780/',
                                        u'publicURL': u'http://192.168.122.80:8780/',
                                        u'region': u'RegionOne'}],
                        u'name': u'placement',
                        u'type': u'placement'},
                       {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9292',
                                        u'internalURL': u'http://192.168.122.80:9292',
                                        u'publicURL': u'http://192.168.122.80:9292',
                                        u'region': u'RegionOne'}],
                        u'name': u'glance',
                        u'type': u'image'},
                       {u'endpoints': [{u'adminURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'internalURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'publicURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'region': u'RegionOne'}],
                        u'name': u'cinderv3',
                        u'type': u'volumev3'},
                       {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9311',
                                        u'internalURL': u'http://192.168.122.80:9311',
                                        u'publicURL': u'http://192.168.122.80:9311',
                                        u'region': u'RegionOne'}],
                        u'name': u'barbican',
                        u'type': u'key-manager'}],
   'show_deleted': False,
   'tenant': u'd9685d1130d74a73af6ee213c421d9de',
   'timestamp': '2018-02-28T10:23:31.133924',
   'user': u'f80483de8573468b869e64262780a903',
   'user_domain': u'default',
   'user_id': u'f80483de8573468b869e64262780a903',
   'user_identity': u'f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default',
   'user_name': u'admin'}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1752289/+subscriptions
References

[Bug 1752289] [NEW] ServiceCatalog does not contain "network" service
From: Thomas Bechtold, 2018-02-28