yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Given we're about to cut RC1 for rocky so ocata/pike are old and cells
v1 is deprecated, I don't think anyone cares about fixing this for CI.

** Changed in: nova
       Status: Confirmed => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745838

Title:
  legacy-tempest-dsvm-cells constantly failing on stable pike and ocata
  due to libvirt connection reset

Status in OpenStack Compute (nova):
  Won't Fix

Bug description:
  The cellsv1 job has been failing pretty constantly within the last
  week or two due to a libvirt connection reset:

  http://logs.openstack.org/36/536936/1/check/legacy-tempest-dsvm-
  cells/a9ff792/logs/libvirt/libvirtd.txt.gz#_2018-01-28_01_25_23_762

  2018-01-28 01:25:23.762+0000: 3896: error :
  virKeepAliveTimerInternal:143 : internal error: connection closed due
  to keepalive timeout

  http://logs.openstack.org/36/536936/1/check/legacy-tempest-dsvm-
  cells/a9ff792/logs/screen-n-cpu.txt.gz?level=TRACE#_2018-01-28_01_25_23_766

  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager [req-392410f9-c834-4bdc-a439-ac20476fe212 - -] Error updating resources for node ubuntu-xenial-inap-mtl01-0002208439.
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager Traceback (most recent call last):
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 6590, in update_available_resource_for_node
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 535, in update_available_resource
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5675, in get_available_resource
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     data["vcpus_used"] = self._get_vcpu_used()
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5316, in _get_vcpu_used
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     for guest in self._host.list_guests():
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 573, in list_guests
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     only_running=only_running, only_guests=only_guests)]
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 593, in list_instance_domains
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     alldoms = self.get_connection().listAllDomains(flags)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     rv = execute(f, *args, **kwargs)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     six.reraise(c, e, tb)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     rv = meth(*args, **kwargs)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 4953, in listAllDomains
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager     raise libvirtError("virConnectListAllDomains() failed", conn=self)
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager libvirtError: Cannot recv data: Connection reset by peer
  2018-01-28 01:25:23.766 16360 ERROR nova.compute.manager 

  It seems to be totally random. I'm not sure what is different about
  this job running on stable vs master, but it doesn't appear to be an
  issue on master:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirtError%3A%20Cannot%20recv%20data%3A%20Connection%20reset%20by%20peer%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20build_name%3A%5C
  %22legacy-tempest-dsvm-cells%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745838/+subscriptions