yahoo-eng-team team mailing list archive

[OpenStack Infra <1612875@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/374975
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2ce193e16d7ca5b93671d7f0d2e3b35761f5d386
Submitter: Jenkins
Branch:    master

commit 2ce193e16d7ca5b93671d7f0d2e3b35761f5d386
Author: Matt Riedemann <mriedem@xxxxxxxxxx>
Date:   Thu Sep 22 12:41:46 2016 -0400

    libvirt: ignore conflict when defining network filters
    
    We have a latent race in the libvirt firewall code when setting
    up static filters which is now an error with libvirt>=1.2.7,
    which is why we started seeing this in CI failures starting in
    newton which run on xenial nodes that have libvirt 1.3.1 (but
    didn't see it on trusty nodes with libvirt 1.2.2).
    
    Libvirt commit 46a811db0731cedaea0153fc223faa6096cee5b5 checks
    for an existing filter with the same name but a different uuid
    when defining network filters and raises an error if found. That
    was added in the 1.2.7 release.
    
    This change simply handles the error and ignores it so we don't
    fail to boot the instance.
    
    Unfortunately we don't have a specific error code from libvirt
    when this happens so the best we can do is compare the error
    message from the libvirt error which is only going to work for
    English locales because the error message from libvirt is
    translated.
    
    Change-Id: I161be26d605351f168e351d3ed3d308234346f6f
    Closes-Bug: #1612875


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1612875

Title:
  FixedIPsTestJson fails server build with "was re-scheduled: operation
  failed: filter 'nova-no-nd-reflection' already exists with uuid"

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/98/355098/1/check/gate-tempest-dsvm-full-
  ubuntu-
  xenial/3c301f3/logs/screen-n-cond.txt.gz?level=TRACE#_2016-08-13_00_53_27_621

  2016-08-13 00:53:27.621 15971 ERROR nova.scheduler.utils [req-e7b83619-01ae-43f2-b293-af02c5cb35a8 tempest-FixedIPsTestJson-2017152444 tempest-FixedIPsTestJson-2017152444] [instance: ac2c9a4a-1b07-43e1-8f4c-b75541331307] Error from last host: ubuntu-xenial-rax-ord-3453779 (node ubuntu-xenial-rax-ord-3453779): [u'Traceback (most recent call last):\n', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1778, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/opt/stack/new/nova/nova/compute/manager.py", line 1973, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance ac2c9a4a-1b07-43e1-8f4c-b75541331307 was re-scheduled: operation failed: filter 'nova-no-nd-reflection' already exists with uuid 1f47eeb2-d473-481e-998a-c4d64a44ac5e\n"]
  2016-08-13 00:53:27.676 15971 WARNING nova.scheduler.utils [req-e7b83619-01ae-43f2-b293-af02c5cb35a8 tempest-FixedIPsTestJson-2017152444 tempest-FixedIPsTestJson-2017152444] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 199, in inner
      return func(*args, **kwargs)

    File "/opt/stack/new/nova/nova/scheduler/manager.py", line 104, in select_destinations
      dests = self.driver.select_destinations(ctxt, spec_obj)

    File "/opt/stack/new/nova/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
      raise exception.NoValidHost(reason=reason)

  NoValidHost: No valid host was found. There are not enough hosts
  available.

  2016-08-13 00:53:27.676 15971 WARNING nova.scheduler.utils [req-
  e7b83619-01ae-43f2-b293-af02c5cb35a8 tempest-
  FixedIPsTestJson-2017152444 tempest-FixedIPsTestJson-2017152444]
  [instance: ac2c9a4a-1b07-43e1-8f4c-b75541331307] Setting instance to
  ERROR state.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22was
  %20re-scheduled%3A%20operation%20failed%3A%20filter%20'nova-no-nd-
  reflection'%20already%20exists%20with%20uuid%5C%22%20AND%20tags%3A%5C%22screen-n-cond.txt%5C%22&from=7d

  5 hits in 7 days, check queue only, but multiple changes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1612875/+subscriptions