← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #23743

[Bug 1381238] [NEW] Race condition on processing DVR floating IPs

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

A race condition can sometimes occur in l-3 agent when a dvr based
floatingip is being deleted from one router and another dvr based
floatingip is being configured on another router in the same node.
Especially if the floatingip being deleted was the last floatingip on
the node.  Although fix for Bug # 1373100 [1] eliminated frequent
observation of this behavior in upstream tests, it still shows up.
Couple of recent examples:

http://logs.openstack.org/88/128288/1/check/check-tempest-dsvm-neutron-
dvr/8fdd1de/

http://logs.openstack.org/03/123403/7/check/check-tempest-dsvm-neutron-
dvr/859534a/

Relevant log messages:

2014-10-14 16:06:15.803 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
2014-10-14 16:06:15.838 22303 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', 'addr', 'show', 'rfp-7ed86ca6-b']
Exit code: 0
Stdout: '2: rfp-7ed86ca6-b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n    link/ether c6:88:ee:71:a7:51 brd ff:ff:ff:ff:ff:ff\n    inet 169.254.30.212/31 scope global rfp-7ed86ca6-b\n       valid_lft forever preferred_lft forever\n    inet6 fe80::c488:eeff:fe71:a751/64 scope link \n       valid_lft forever preferred_lft forever\n'
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81
2014-10-14 16:06:15.839 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
2014-10-14 16:06:16.221 22303 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b']
Exit code: 0
Stdout: ''
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81
2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.l3_agent [-] DVR: unplug: fg-f04e25ef-e3 _destroy_fip_namespace /opt/stack/new/neutron/neutron/agent/l3_agent.py:679
2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
2014-10-14 16:06:16.251 22303 ERROR neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "rfp-7ed86ca6-b"\n'  


[1] https://bugs.launchpad.net/neutron/+bug/1373100

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1381238

Title:
  Race condition on processing DVR floating IPs

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  A race condition can sometimes occur in l-3 agent when a dvr based
  floatingip is being deleted from one router and another dvr based
  floatingip is being configured on another router in the same node.
  Especially if the floatingip being deleted was the last floatingip on
  the node.  Although fix for Bug # 1373100 [1] eliminated frequent
  observation of this behavior in upstream tests, it still shows up.
  Couple of recent examples:

  http://logs.openstack.org/88/128288/1/check/check-tempest-dsvm-
  neutron-dvr/8fdd1de/

  http://logs.openstack.org/03/123403/7/check/check-tempest-dsvm-
  neutron-dvr/859534a/

  Relevant log messages:

  2014-10-14 16:06:15.803 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
  2014-10-14 16:06:15.838 22303 DEBUG neutron.agent.linux.utils [-] 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', 'addr', 'show', 'rfp-7ed86ca6-b']
  Exit code: 0
  Stdout: '2: rfp-7ed86ca6-b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n    link/ether c6:88:ee:71:a7:51 brd ff:ff:ff:ff:ff:ff\n    inet 169.254.30.212/31 scope global rfp-7ed86ca6-b\n       valid_lft forever preferred_lft forever\n    inet6 fe80::c488:eeff:fe71:a751/64 scope link \n       valid_lft forever preferred_lft forever\n'
  Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81
  2014-10-14 16:06:15.839 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
  2014-10-14 16:06:16.221 22303 DEBUG neutron.agent.linux.utils [-] 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'fip-82fb2751-30ba-4015-a5da-6c8563064db9', 'ip', 'link', 'del', 'fpr-7ed86ca6-b']
  Exit code: 0
  Stdout: ''
  Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:81
  2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.l3_agent [-] DVR: unplug: fg-f04e25ef-e3 _destroy_fip_namespace /opt/stack/new/neutron/neutron/agent/l3_agent.py:679
  2014-10-14 16:06:16.222 22303 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:46
  2014-10-14 16:06:16.251 22303 ERROR neutron.agent.linux.utils [-] 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-7ed86ca6-b42d-4ba9-8899-447ff0509174', 'ip', '-4', 'addr', 'add', '172.24.4.91/32', 'brd', '172.24.4.91', 'scope', 'global', 'dev', 'rfp-7ed86ca6-b']
  Exit code: 1
  Stdout: ''
  Stderr: 'Cannot find device "rfp-7ed86ca6-b"\n'  


  [1] https://bugs.launchpad.net/neutron/+bug/1373100

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1381238/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next