yahoo-eng-team team mailing list archive

[Pooja Ghumre <poojaghumre@xxxxxxxxx>]

nova-no-nd-reflection is one of the static network filters in libvirt, which only needs to be created once. In this case, there was a race condition between 2 instance creation workflows both of which tried to setup these static filters, so the latter failed with 'nwfilter already exists' error.
The nova libvirt NWFilterFirewall driver has no locking around the static_filters_configured flag used to determine if this one-time setup has been done. Moreover, libvirt driver doesn't look at the exact error code in exception being thrown during lookup; it should only create new filter if lookup fails with error VIR_ERR_NO_NWFILTER = 62.

** Changed in: tripleo
     Assignee: (unassigned) => Pooja Ghumre (pooja-9)

** Project changed: tripleo => nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1632041

Title:
  Concurrent nova instance creation fails with nova-no-nd-reflection
  filter already exists error

Status in OpenStack Compute (nova):
  New

Bug description:
  Immediately after starting nova-compute service on the host, if
  concurrent instance creation operations are triggered on the same
  host, both the workflows try to define the static network filters and
  the latter one fails with 'nova-no-nd-reflection filter already
  exists'.

  Log snippet:

  2016-08-07 05:40:57.628 17601 ERROR nova.scheduler.utils [req-137df929-3b97-4437-b26d-3280dcbcd8e3 b5dc09cde3ac440d9060d6cc89f7ba71 87841dbf785d4ff3a43d5fd5c53ead50 - - -] [insta
  nce: b8d573aa-1aae-4dec-a275-ae8c82f4a274] Error from last host: dc5e6fb9-bcd1-4548-ad2e-4e571b783fe8 (node cinder-glance-centos7-10-4-252-239.platform9.sys): [u'Traceback (most 
  recent call last):\n', u' File "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2128, in _do_build_and_run_instance\n filter_properties)\n', u' File
  "/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2284, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"Resch
  eduledException: Build of instance b8d573aa-1aae-4dec-a275-ae8c82f4a274 was re-scheduled: operation failed: filter 'nova-no-nd-reflection' already exists with uuid 60ba4736-64c7-
  42c3-8efe-e6919dec44d2\n"]
  2016-08-07 05:40:57.865 17601 WARNING nova.scheduler.utils [req-137df929-3b97-4437-b26d-3280dcbcd8e3 b5dc09cde3ac440d9060d6cc89f7ba71 87841dbf785d4ff3a43d5fd5c53ead50 - - -] Fail
  ed to compute_task_build_instances: No valid host was found. 
  Traceback (most recent call last):
  File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
  return func(*args, **kwargs)
  File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/manager.py", line 84, in select_destinations
  filter_properties)
  File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
  raise exception.NoValidHost(reason=msg)
  NoValidHost: No valid host was found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1632041/+subscriptions