yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1610303] [NEW] l2pop mech fails to update_port_postcommit on a loaded cluster

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date: Fri, 05 Aug 2016 15:35:53 -0000
Reply-to: Bug 1610303 <1610303@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On a cluster where VMs boots and deletes happen pretty intensively
following traces can pop up in neutron server log:

2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Mechanism driver 'l2population' failed in update_port_postcommit
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 401, in _call_on_drivers
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 120, in update_port_postcommit
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     self._update_port_up(context)
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 227, in _update_port_up
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     network_id)
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 176, in _create_agent_fdb
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     fdbs.extend(self._get_port_fdb_entries(binding.port))
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 45, in _get_port_fdb_entries
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     for ip in port['fixed_ips']]
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers TypeError: 'NoneType' object has no attribute '__getitem__'
2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers
2016-08-05 14:08:29.578 9560 ERROR neutron.plugins.ml2.rpc [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Failed to update device 4c499a14-7211-4714-afa2-95b280d595a2 up

This leads to device to being set to Active state and hence Nova
timeouts waiting for the interface to be ready.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  On a cluster where VMs boots and deletes happen pretty intensively
  following traces can pop up in neutron server log:
  
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Mechanism driver 'l2population' failed in update_port_postcommit
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 401, in _call_on_drivers
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 120, in update_port_postcommit
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     self._update_port_up(context)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 227, in _update_port_up
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     network_id)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 176, in _create_agent_fdb
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     fdbs.extend(self._get_port_fdb_entries(binding.port))
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 45, in _get_port_fdb_entries
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     for ip in port['fixed_ips']]
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers TypeError: 'NoneType' object has no attribute '__getitem__'
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers
  2016-08-05 14:08:29.578 9560 ERROR neutron.plugins.ml2.rpc [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Failed to update device 4c499a14-7211-4714-afa2-95b280d595a2 up
+ 
+ This leads to device to being set to Active state and hence Nova
+ timeouts waiting for the interface to be ready.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1610303

Title:
  l2pop mech fails to update_port_postcommit on a loaded cluster

Status in neutron:
  New

Bug description:
  On a cluster where VMs boots and deletes happen pretty intensively
  following traces can pop up in neutron server log:

  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Mechanism driver 'l2population' failed in update_port_postcommit
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 401, in _call_on_drivers
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 120, in update_port_postcommit
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     self._update_port_up(context)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 227, in _update_port_up
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     network_id)
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 176, in _create_agent_fdb
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     fdbs.extend(self._get_port_fdb_entries(binding.port))
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 45, in _get_port_fdb_entries
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers     for ip in port['fixed_ips']]
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers TypeError: 'NoneType' object has no attribute '__getitem__'
  2016-08-05 14:08:29.575 9560 ERROR neutron.plugins.ml2.managers
  2016-08-05 14:08:29.578 9560 ERROR neutron.plugins.ml2.rpc [req-1b5e9a29-7f7e-48f8-84ee-19ce217cb556 - - - - -] Failed to update device 4c499a14-7211-4714-afa2-95b280d595a2 up

  This leads to device to being set to Active state and hence Nova
  timeouts waiting for the interface to be ready.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1610303/+subscriptions
Follow ups

[Bug 1610303] Re: l2pop mech fails to update_port_postcommit on a loaded cluster
From: OpenStack Infra, 2016-08-11