yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1260692] Re: LBaaS: stacktraces in q-lbaas while running tempest lbaas api tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 22 Jan 2014 15:10:26 -0000
Reply-to: Bug 1260692 <1260692@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Fix Committed => Fix Released

** Changed in: neutron
    Milestone: None => icehouse-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1260692

Title:
  LBaaS: stacktraces in q-lbaas while running tempest lbaas api tests

Status in OpenStack Neutron (virtual network service):
  Fix Released

Bug description:
  Following stacktraces appeared in q-lbaas after merging
  https://review.openstack.org/#/c/40381:

  2013-12-10 14:02:10.655 6158 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [req-0316f127-ac85-4d14-b6c6-57d58272f9e6 c7798670e00d40f7809a65f04536b969 f21028acd29f43dc935c5aeb3950bb06] Create member b4b0bd55-a70e-48f5-8816-fdbcba372cc1 failed on device driver haproxy_ns
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Traceback (most recent call last):
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 276, in create_member
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     driver.create_member(member)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 308, in create_member
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._refresh_device(member['pool_id'])
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 283, in _refresh_device
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self.deploy_instance(logical_config)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 279, in deploy_instance
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self.create(logical_config)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 90, in create
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._plug(namespace, logical_config['vip']['port'])
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 246, in _plug
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     namespace=namespace
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/interface.py", line 186, in plug
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     ns_dev.link.set_address(mac_address)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 230, in set_address
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._as_root('set', self.name, 'address', mac_address)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 217, in _as_root
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     kwargs.get('use_root_namespace', False))
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     namespace)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     root_helper=root_helper)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 75, in execute
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     raise RuntimeError(m)
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager RuntimeError: 
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tapc8872a60-80', 'address', 'fa:16:3e:fe:37:64']
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Exit code: 1
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stdout: ''
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stderr: 'Cannot find device "tapc8872a60-80"\n'
  2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager 
  2013-12-10 14:02:12.330 6158 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [req-832c5fe5-d69d-47af-bd09-0632d591ce62 c7798670e00d40f7809a65f04536b969 f21028acd29f43dc935c5aeb3950bb06] Create health_monitor {'monitor_id': u'540e405d-8d64-45fb-8494-da4c301923c9', 'pool_id': u'4c4dc051-4d2d-414e-8c7f-7ba281878509'} failed on device driver haproxy_ns
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Traceback (most recent call last):
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 303, in create_pool_health_monitor
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     driver.create_pool_health_monitor(health_monitor, pool_id)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 317, in create_pool_health_monitor
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._refresh_device(pool_id)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 283, in _refresh_device
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self.deploy_instance(logical_config)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 279, in deploy_instance
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self.create(logical_config)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 90, in create
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._plug(namespace, logical_config['vip']['port'])
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 246, in _plug
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     namespace=namespace
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/interface.py", line 195, in plug
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     namespace_obj = ip.ensure_namespace(namespace)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 136, in ensure_namespace
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     ip = self.netns.add(name)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 447, in add
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     self._as_root('add', name, use_root_namespace=True)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 217, in _as_root
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     kwargs.get('use_root_namespace', False))
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     namespace)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     root_helper=root_helper)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 75, in execute
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager     raise RuntimeError(m)
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager RuntimeError: 
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qlbaas-4c4dc051-4d2d-414e-8c7f-7ba281878509']
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Exit code: 1
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stdout: ''
  2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stderr: 'Could not create /var/run/netns/qlbaas-4c4dc051-4d2d-414e-8c7f-7ba281878509: File exists\n'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1260692/+subscriptions
References

[Bug 1260692] [NEW] LBaaS: stacktraces in q-lbaas while running tempest lbaas api tests
From: Oleg Bondarev, 2013-12-13