← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #28851

[Bug 1426165] [NEW] DVR: "Device or resource busy" error seen when fip namespace is being deleted

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

How to reproduce -

1. Assign 2 routers with network/subnet/etc sharing the same external network for FIPs to a single agent/host.
2.  Disassociate all FIPs
3.  FIP namespace should be deleted but the following trace is seen instead


2015-02-26 15:38:34.457 ^[[00;32mDEBUG neutron.agent.l3.dvr_fip_ns [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mDVR: destroy fip ns: fip-6473ee45-f14f-4b86-a7da-678845a10c08^[[00m ^[[00;33mfrom (pid=6207) destroy /opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py:153^[[00m
2015-02-26 15:38:34.457 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mRunning command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']^[[00m ^[[00;33mfrom (pid=6207) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:50^[[00m
2015-02-26 15:38:34.651 ^[[01;31mERROR neutron.agent.linux.utils [^[[00;36m-^[[01;31m] ^[[01;35m^[[01;31m
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']
Exit code: 1
Stdout:
Stderr: Cannot remove /var/run/netns/fip-6473ee45-f14f-4b86-a7da-678845a10c08: Device or resource busy
^[[00m
2015-02-26 15:38:34.652 ^[[01;31mERROR neutron.agent.l3.dvr_fip_ns [^[[00;36m-^[[01;31m] ^[[01;35m^[[01;31mFailed trying to delete namespace: fip-6473ee45-f14f-4b86-a7da-678845a10c08^[[00m
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mTraceback (most recent call last):
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 157, in destroy
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    ip_wrapper.netns.delete(ns)
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 541, in delete
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    self._as_root('delete', name, use_root_namespace=True)
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 250, in _as_root
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    kwargs.get('use_root_namespace', False))
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 72, in _as_root
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    log_fail_as_error=self.log_fail_as_error)
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 84, in _execute
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    log_fail_as_error=log_fail_as_error)
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 86, in execute
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    raise RuntimeError(m)
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mRuntimeError:
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mCommand: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mExit code: 1
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mStdout:
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mStderr: Cannot remove /var/run/netns/fip-6473ee45-f14f-4b86-a7da-678845a10c08: Device or resource busy
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m
^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-dvr-backlog

** Tags added: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1426165

Title:
  DVR: "Device or resource busy" error seen when fip namespace is being
  deleted

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  How to reproduce -

  1. Assign 2 routers with network/subnet/etc sharing the same external network for FIPs to a single agent/host.
  2.  Disassociate all FIPs
  3.  FIP namespace should be deleted but the following trace is seen instead

  
  2015-02-26 15:38:34.457 ^[[00;32mDEBUG neutron.agent.l3.dvr_fip_ns [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mDVR: destroy fip ns: fip-6473ee45-f14f-4b86-a7da-678845a10c08^[[00m ^[[00;33mfrom (pid=6207) destroy /opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py:153^[[00m
  2015-02-26 15:38:34.457 ^[[00;32mDEBUG neutron.agent.linux.utils [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mRunning command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']^[[00m ^[[00;33mfrom (pid=6207) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:50^[[00m
  2015-02-26 15:38:34.651 ^[[01;31mERROR neutron.agent.linux.utils [^[[00;36m-^[[01;31m] ^[[01;35m^[[01;31m
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']
  Exit code: 1
  Stdout:
  Stderr: Cannot remove /var/run/netns/fip-6473ee45-f14f-4b86-a7da-678845a10c08: Device or resource busy
  ^[[00m
  2015-02-26 15:38:34.652 ^[[01;31mERROR neutron.agent.l3.dvr_fip_ns [^[[00;36m-^[[01;31m] ^[[01;35m^[[01;31mFailed trying to delete namespace: fip-6473ee45-f14f-4b86-a7da-678845a10c08^[[00m
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mTraceback (most recent call last):
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 157, in destroy
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    ip_wrapper.netns.delete(ns)
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 541, in delete
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    self._as_root('delete', name, use_root_namespace=True)
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 250, in _as_root
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    kwargs.get('use_root_namespace', False))
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 72, in _as_root
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    log_fail_as_error=self.log_fail_as_error)
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 84, in _execute
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    log_fail_as_error=log_fail_as_error)
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m  File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 86, in execute
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m    raise RuntimeError(m)
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mRuntimeError:
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mCommand: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'delete', 'fip-6473ee45-f14f-4b86-a7da-678845a10c08']
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mExit code: 1
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mStdout:
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00mStderr: Cannot remove /var/run/netns/fip-6473ee45-f14f-4b86-a7da-678845a10c08: Device or resource busy
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m
  ^[[01;31m2015-02-26 15:38:34.652 TRACE neutron.agent.l3.dvr_fip_ns ^[[01;35m^[[00m

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1426165/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next