yahoo-eng-team team mailing list archive

[Brian Haley <1538387@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I've been trying to track down failures in the DVR multinode job.  I'm
now tripping over this one.

For context I've been focusing on a single change, but if you see a
failure in the gate-tempest-dsvm-neutron-dvr-multinode-full job you'll
probably be able to find similar info.  This is the change:

http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron-
dvr-multinode-full/5abca7b/logs/

The screen-q-agt log shows a traceback here:

http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron-
dvr-multinode-
full/5abca7b/logs/screen-q-agt.txt.gz#_2016-01-18_10_11_29_715

<snip>
2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc_manager/l2population_rpc.py", line 312, in fdb_chg_ip_tun
2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher     mac_ip.mac_address,
2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'list' object has no attribute 'mac_address'
2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher

The info passed to fdb_chg_ip_tun() should have a "PortInfo" namedtuple
as data, but from the line before we can see it doesn't:

DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc
[req-671e8634-c753-4002-acfd-68515dd44f29 None None]
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method fdb_chg_ip_tun called with arguments (<neutron.context.Context
object at 0x7fa2ee6988d0>,
<neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.br_tun.DeferredOVSTunnelBridge
object at 0x7fa2ee79abd0>, {u'c4ae0757-e3e5-419c-b2ba-4d7817964237':
{u'10.209.192.28': {u'before': [[u'fa:16:3e:8d:2e:48',
u'2003:0:0:1::1']]}}}, '10.208.193.94', {u'7ca0dcf2-fb63-4959-92ee-
cc757da8d120':

So from this it's clear that _unmarshall_fdb_entries() either hasn't
been called, or didn't do anything.

Looking over in screen-q-svc.log for the info before the RPC call finds:

DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-
f32790a5-0160-47b9-89b4-763b9c23bc08 tempest-
TestGettingAddress-2071048693 tempest-TestGettingAddress-1817548879]
Fanout notify l2population agents at q-agent-notifier the message
update_fdb_entries with {'chg_ip': {u'c4ae0757-e3e5-419c-b2ba-
4d7817964237': {u'10.208.193.94': {'before':
[PortInfo(mac_address=u'fa:16:3e:8d:2e:48',
ip_address=u'2003:0:0:1::1')]}}}} _notification_fanout
/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:47

This is the message right before _marshall_fdb_entries() was called to
convert the PortInfo into [<mac>, <ip>] pairs, and from the above it
looks like it did.

I'm just starting to look at this now, but maybe someone more familiar
with l2pop has a guess at what's broken.

** Affects: neutron
     Importance: Undecided
     Assignee: Brian Haley (brian-haley)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Brian Haley (brian-haley)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1538387

Title:
  fdb_chg_ip_tun throwing exception because fdb_entries not in correct
  format

Status in neutron:
  New

Bug description:
  I've been trying to track down failures in the DVR multinode job.  I'm
  now tripping over this one.

  For context I've been focusing on a single change, but if you see a
  failure in the gate-tempest-dsvm-neutron-dvr-multinode-full job you'll
  probably be able to find similar info.  This is the change:

  http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron-
  dvr-multinode-full/5abca7b/logs/

  The screen-q-agt log shows a traceback here:

  http://logs.openstack.org/77/177777/4/check/gate-tempest-dsvm-neutron-
  dvr-multinode-
  full/5abca7b/logs/screen-q-agt.txt.gz#_2016-01-18_10_11_29_715

  <snip>
  2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc_manager/l2population_rpc.py", line 312, in fdb_chg_ip_tun
  2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher     mac_ip.mac_address,
  2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'list' object has no attribute 'mac_address'
  2016-01-18 10:11:29.724 12932 ERROR oslo_messaging.rpc.dispatcher

  The info passed to fdb_chg_ip_tun() should have a "PortInfo"
  namedtuple as data, but from the line before we can see it doesn't:

  DEBUG neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc
  [req-671e8634-c753-4002-acfd-68515dd44f29 None None]
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
  method fdb_chg_ip_tun called with arguments (<neutron.context.Context
  object at 0x7fa2ee6988d0>,
  <neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.br_tun.DeferredOVSTunnelBridge
  object at 0x7fa2ee79abd0>, {u'c4ae0757-e3e5-419c-b2ba-4d7817964237':
  {u'10.209.192.28': {u'before': [[u'fa:16:3e:8d:2e:48',
  u'2003:0:0:1::1']]}}}, '10.208.193.94', {u'7ca0dcf2-fb63-4959-92ee-
  cc757da8d120':

  So from this it's clear that _unmarshall_fdb_entries() either hasn't
  been called, or didn't do anything.

  Looking over in screen-q-svc.log for the info before the RPC call
  finds:

  DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-
  f32790a5-0160-47b9-89b4-763b9c23bc08 tempest-
  TestGettingAddress-2071048693 tempest-TestGettingAddress-1817548879]
  Fanout notify l2population agents at q-agent-notifier the message
  update_fdb_entries with {'chg_ip': {u'c4ae0757-e3e5-419c-b2ba-
  4d7817964237': {u'10.208.193.94': {'before':
  [PortInfo(mac_address=u'fa:16:3e:8d:2e:48',
  ip_address=u'2003:0:0:1::1')]}}}} _notification_fanout
  /opt/stack/new/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:47

  This is the message right before _marshall_fdb_entries() was called to
  convert the PortInfo into [<mac>, <ip>] pairs, and from the above it
  looks like it did.

  I'm just starting to look at this now, but maybe someone more familiar
  with l2pop has a guess at what's broken.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1538387/+subscriptions