yahoo-eng-team team mailing list archive

[Ihar Hrachyshka <1644937@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

http://logs.openstack.org/48/395748/15/check/gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-xenial/366ecda/logs/testr_results.html.gz

Traceback (most recent call last):
  File "tempest/test.py", line 119, in wrapper
    return func(*func_args, **func_kwargs)
  File "tempest/test.py", line 100, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_network_basic_ops.py", line 768, in test_router_rescheduling
    msg='After router rescheduling')
  File "tempest/scenario/test_network_basic_ops.py", line 206, in check_public_network_connectivity
    self.check_floating_ip_status(floating_ip, floatingip_status)
  File "tempest/scenario/manager.py", line 907, in check_floating_ip_status
    st=status))
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': FloatingIP: {u'id': u'41cbfe8e-066c-414b-9b06-f818cc845ff9', u'status': u'DOWN', u'description': u'', u'router_id': u'fa20a117-a4f4-42db-9232-92ed1a3fd211', u'tenant_id': u'2d4bc4eb27984f1f9466af083e6c0929', u'revision_number': 1, u'port_id': u'1204588c-9084-4846-92d8-c0ed0719b16d', u'fixed_ip_address': u'10.1.0.3', u'floating_ip_address': u'172.24.5.17', u'updated_at': u'2016-11-25T16:17:42Z', u'created_at': u'2016-11-25T16:17:42Z', u'floating_network_id': u'3c4a550a-3dd9-4501-9bc8-0719853048d1', u'project_id': u'2d4bc4eb27984f1f9466af083e6c0929'} is at status: DOWN. failed  to reach status: ACTIVE

In tempest log, we see that we rescheduled the same agent for a router,
then check it's indeed in the list of scheduled agents, then check FIP
status to be ACTIVE. The last check fails (initially it's ACTIVE, but
then just before we check for the last time, it flips back to DOWN).

Looking into l3 agent logs, it seems like the agent flipped the status
to DOWN while processing the previous unscheduling event.

It seems like we should wait for the FIP status to flip to DOWN after
unscheduling, to make sure the agent is done with unscheduling event
processing, so that we are safe to proceed with rescheduling without a
risk of some previous update events still sitting in router update
queue.

** Affects: tempest
     Importance: High
     Assignee: Ihar Hrachyshka (ihar-hrachyshka)
         Status: In Progress


** Tags: gate-failure

** Changed in: neutron
       Status: New => Confirmed

** Changed in: neutron
     Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka)

** Changed in: neutron
   Importance: Undecided => High

** Tags added: gate-failure

** Project changed: neutron => tempest

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1644937

Title:
  test_router_rescheduling failed with unexpected FIP status after
  rescheduling

Status in tempest:
  In Progress

Bug description:
  http://logs.openstack.org/48/395748/15/check/gate-tempest-dsvm-
  neutron-linuxbridge-ubuntu-xenial/366ecda/logs/testr_results.html.gz

  Traceback (most recent call last):
    File "tempest/test.py", line 119, in wrapper
      return func(*func_args, **func_kwargs)
    File "tempest/test.py", line 100, in wrapper
      return f(self, *func_args, **func_kwargs)
    File "tempest/scenario/test_network_basic_ops.py", line 768, in test_router_rescheduling
      msg='After router rescheduling')
    File "tempest/scenario/test_network_basic_ops.py", line 206, in check_public_network_connectivity
      self.check_floating_ip_status(floating_ip, floatingip_status)
    File "tempest/scenario/manager.py", line 907, in check_floating_ip_status
      st=status))
    File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': FloatingIP: {u'id': u'41cbfe8e-066c-414b-9b06-f818cc845ff9', u'status': u'DOWN', u'description': u'', u'router_id': u'fa20a117-a4f4-42db-9232-92ed1a3fd211', u'tenant_id': u'2d4bc4eb27984f1f9466af083e6c0929', u'revision_number': 1, u'port_id': u'1204588c-9084-4846-92d8-c0ed0719b16d', u'fixed_ip_address': u'10.1.0.3', u'floating_ip_address': u'172.24.5.17', u'updated_at': u'2016-11-25T16:17:42Z', u'created_at': u'2016-11-25T16:17:42Z', u'floating_network_id': u'3c4a550a-3dd9-4501-9bc8-0719853048d1', u'project_id': u'2d4bc4eb27984f1f9466af083e6c0929'} is at status: DOWN. failed  to reach status: ACTIVE

  In tempest log, we see that we rescheduled the same agent for a
  router, then check it's indeed in the list of scheduled agents, then
  check FIP status to be ACTIVE. The last check fails (initially it's
  ACTIVE, but then just before we check for the last time, it flips back
  to DOWN).

  Looking into l3 agent logs, it seems like the agent flipped the status
  to DOWN while processing the previous unscheduling event.

  It seems like we should wait for the FIP status to flip to DOWN after
  unscheduling, to make sure the agent is done with unscheduling event
  processing, so that we are safe to proceed with rescheduling without a
  risk of some previous update events still sitting in router update
  queue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/tempest/+bug/1644937/+subscriptions