yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1775075] Re: EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Mon, 04 Jun 2018 22:00:18 -0000
Reply-to: Bug 1775075 <1775075@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Tags added: upgrades volumes

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Changed in: nova/queens
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova/queens
       Status: New => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1775075

Title:
  EndpointNotFound raised by Pike n-cpu when running alongside Queens
  n-api

Status in OpenStack Compute (nova):
  New
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  Description
  ===========
  During a P to Q upgrade n-cpu processes still running P will be unable to find the volumev2 endpoint when running alongside Q n-api processes due to the following change: 

  Update cinder in RequestContext service catalog
  https://review.openstack.org/#/c/510947/

  This results in failures anytime the P n-cpu process attempts to
  interact with the volume service, for example during LM from the node:

  2018-06-02 00:19:17.683 1 WARNING nova.virt.libvirt.driver [req-3712be3d-b883-4fe1-bab0-83ee44bd5bb5 e16a043a84b14e2b8afbdd1b8677259f cb92ed750eac463faf8935cb137f1e60 - default default] [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b] Error monitoring migration: internalURL endpoint for volumev2 service named cinderv2 not found: EndpointNotFound: internalURL endpoint for volumev2 service named cinderv2 not found
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b] Traceback (most recent call last):
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6817, in _live_migration
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     finish_event, disk_paths)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6728, in _live_migration_monitor
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     migrate_data)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     function_name, call_dict, binary)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     self.force_reraise()
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     six.reraise(self.type_, self.value, self.tb)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     return f(self, context, *args, **kw)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 218, in decorated_function
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     kwargs['instance'], e, sys.exc_info())
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     self.force_reraise()
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     six.reraise(self.type_, self.value, self.tb)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 206, in decorated_function
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     return function(self, context, *args, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5684, in _post_live_migration
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     migrate_data)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7169, in post_live_migration
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     connector)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 235, in wrapper
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     res = method(self, ctx, *args, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 257, in wrapper
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     res = method(self, ctx, volume_id, *args, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 410, in initialize_connection
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     context).volumes.initialize_connection(volume_id, connector)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 116, in cinderclient
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     url = _SESSION.get_endpoint(auth, **service_parameters)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 947, in get_endpoint
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     return auth.get_endpoint(self, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     region_name=region_name)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     return wrapped(*args, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 344, in url_for
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     endpoint_id=endpoint_id).url
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     return wrapped(*args, **kwargs)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 407, in endpoint_data_for
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b]     raise exceptions.EndpointNotFound(msg)
  2018-06-02 00:19:17.683 1 ERROR nova.virt.libvirt.driver [instance: 1a3800b3-9f75-4999-b726-a1afb0ebdd9b] EndpointNotFound: internalURL endpoint for volumev2 service named cinderv2 not found

  Steps to reproduce
  ==================
  * Starting with a P environment
  * Upgrade n-api/n-schd/n-cond etc to Q
  * Attempt to attach a volume to an instance on a P n-cpu host

  Expected result
  ===============
  volumev2 endpoint found and used.

  Actual result
  =============
  EndpointNotFound

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     Queens control plane, Pike computes.

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============
  LM between OSP 12 and OSP 13 computes fail during major upgrade
  https://bugzilla.redhat.com/show_bug.cgi?id=1585656

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1775075/+subscriptions
References

[Bug 1775075] [NEW] EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api
From: Lee Yarwood, 2018-06-04