yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1487972] [NEW] LbbasV2+DVR- we see ERROR message in lbaas log every minute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alex Syafeyev <astafeye@xxxxxxxxxx>
Date: Mon, 24 Aug 2015 06:17:41 -0000
Reply-to: Bug 1487972 <1487972@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After configurinf LBaaSv2 with DVR we see ERRORs every minute in the
LBaas log.

Reproducible: 100%
Setps to reproduce: 
1. AIO+compute node setup with lbaasV2 enabled.- see the there is active lbaasv2 agent. 
2. configure DVR on the setup. restart neutron. - see lbaas logs 

IF NOT REPRODUCIBLE, BEFORE STEP 2 CREATE NETS SUBNETS AND ROUTER AND
THE DELETE. THEN EXECUTE STEP 2.


!!!!!! IF WE FIRST ENBALE DVR AND THEN CONFIGURE LBAAS TO V2 WE DO NOT SEE THOSE ERRORS. 

The log :
2015-08-24 09:12:38.779 16280 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     ready_instances = set(self.plugin_rpc.get_ready_devices())
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     return cctxt.call(self.context, 'get_ready_devices', host=self.host)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=self.retry)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=retry)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     result = self._waiter.wait(msg_id, timeout)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     message = self.waiters.get(msg_id, timeout=timeout)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     'to message ID %s' % msg_id)
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 770300260dc94a218863238b5b49bbc8
2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager
2015-08-24 09:12:38.780 16280 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
2015-08-24 09:12:38.780 16280 WARNING neutron.openstack.common.loopingcall [-] task <bound method LbaasAgentManager.run_periodic_tasks of <neutron_lbaas.services.loadbalancer.agent.agent_manager.LbaasAgentManager object at 0x3f84a10>> run outlasted interval by 50.01 sec
2015-08-24 09:12:39.011 16292 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is c50ffdaafbe7459ebc9be07c4e1ea068. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2015-08-24 09:12:40.252 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
2015-08-24 09:12:40.252 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.periodic_resync run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
2015-08-24 09:12:48.781 16280 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.periodic_resync run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
2015-08-24 09:12:48.781 16280 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 37edfd235247414d910146108d8dd80e _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-08-24 09:12:48.782 16280 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is c07261b036244b95931511c5ba5512fc. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2015-08-24 09:12:50.253 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: lbaas

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1487972

Title:
  LbbasV2+DVR- we see ERROR message in lbaas log every minute

Status in neutron:
  New

Bug description:
  After configurinf LBaaSv2 with DVR we see ERRORs every minute in the
  LBaas log.

  Reproducible: 100%
  Setps to reproduce: 
  1. AIO+compute node setup with lbaasV2 enabled.- see the there is active lbaasv2 agent. 
  2. configure DVR on the setup. restart neutron. - see lbaas logs 

  IF NOT REPRODUCIBLE, BEFORE STEP 2 CREATE NETS SUBNETS AND ROUTER AND
  THE DELETE. THEN EXECUTE STEP 2.

  
  !!!!!! IF WE FIRST ENBALE DVR AND THEN CONFIGURE LBAAS TO V2 WE DO NOT SEE THOSE ERRORS. 

  The log :
  2015-08-24 09:12:38.779 16280 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     ready_instances = set(self.plugin_rpc.get_ready_devices())
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     return cctxt.call(self.context, 'get_ready_devices', host=self.host)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=self.retry)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=retry)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     result = self._waiter.wait(msg_id, timeout)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     message = self.waiters.get(msg_id, timeout=timeout)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager     'to message ID %s' % msg_id)
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 770300260dc94a218863238b5b49bbc8
  2015-08-24 09:12:38.779 16280 TRACE neutron_lbaas.services.loadbalancer.agent.agent_manager
  2015-08-24 09:12:38.780 16280 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
  2015-08-24 09:12:38.780 16280 WARNING neutron.openstack.common.loopingcall [-] task <bound method LbaasAgentManager.run_periodic_tasks of <neutron_lbaas.services.loadbalancer.agent.agent_manager.LbaasAgentManager object at 0x3f84a10>> run outlasted interval by 50.01 sec
  2015-08-24 09:12:39.011 16292 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is c50ffdaafbe7459ebc9be07c4e1ea068. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
  2015-08-24 09:12:40.252 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
  2015-08-24 09:12:40.252 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.periodic_resync run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
  2015-08-24 09:12:48.781 16280 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.periodic_resync run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219
  2015-08-24 09:12:48.781 16280 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 37edfd235247414d910146108d8dd80e _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:311
  2015-08-24 09:12:48.782 16280 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is c07261b036244b95931511c5ba5512fc. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
  2015-08-24 09:12:50.253 16292 DEBUG neutron.openstack.common.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats run_periodic_tasks /usr/lib/python2.7/site-packages/neutron/openstack/common/periodic_task.py:219

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1487972/+subscriptions
Follow ups

[Bug 1487972] Re: LbbasV2+DVR- we see ERROR message in lbaas log every minute
From: Doug Wiegley, 2015-12-03