← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95254

[Bug 2095595] [NEW] The neutron-dhcp-agent resync throttling mechanism doesn't work

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The neutron-dhcp-agent has a resync_throttle mechanism to ensure a minimum event interval if events occur too frequently [0].
However, this mechanism doesn't work because a new throttler object is created each time.

I conducted a simple experiment by forcing a failure within the
'sync_state' call and observed no throttling of sync_state:

--- a/neutron/agent/dhcp/agent.py
+++ b/neutron/agent/dhcp/agent.py
@@ -303,6 +303,7 @@ class DhcpAgent(manager.Manager):
         known_network_ids = set(self.cache.get_network_ids())

         try:
+            raise Exception('test sync_state error')
             active_networks = self.plugin_rpc.get_active_networks_info(
                 enable_dhcp_filter=False)
             LOG.info('All active networks have been fetched through RPC.')

Hundreds attempts to call 'Synchronizing state' per second:

Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent     raise Exception('test sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been scheduled {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.: Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent     raise Exception('test sync_state error')
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been scheduled {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state


[0] https://bugs.launchpad.net/neutron/+bug/1780370

** Affects: neutron
     Importance: Undecided
     Assignee: Anton Kurbatov (akurbatov)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Anton Kurbatov (akurbatov)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2095595

Title:
  The neutron-dhcp-agent resync throttling mechanism doesn't work

Status in neutron:
  New

Bug description:
  The neutron-dhcp-agent has a resync_throttle mechanism to ensure a minimum event interval if events occur too frequently [0].
  However, this mechanism doesn't work because a new throttler object is created each time.

  I conducted a simple experiment by forcing a failure within the
  'sync_state' call and observed no throttling of sync_state:

  --- a/neutron/agent/dhcp/agent.py
  +++ b/neutron/agent/dhcp/agent.py
  @@ -303,6 +303,7 @@ class DhcpAgent(manager.Manager):
           known_network_ids = set(self.cache.get_network_ids())

           try:
  +            raise Exception('test sync_state error')
               active_networks = self.plugin_rpc.get_active_networks_info(
                   enable_dhcp_filter=False)
               LOG.info('All active networks have been fetched through RPC.')

  Hundreds attempts to call 'Synchronizing state' per second:

  Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.: Exception: test sync_state error
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent     raise Exception('test sync_state error')
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
  Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been scheduled {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
  Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
  Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
  Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.: Exception: test sync_state error
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent   File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 306, in sync_state
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent     raise Exception('test sync_state error')
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent Exception: test sync_state error
  Jan 23 07:04:39 ERROR neutron.agent.dhcp.agent
  Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] Resync event has been scheduled {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:385}}
  Jan 23 07:04:39 DEBUG neutron.common.utils [-] Calling throttled function clear {{(pid=798097) wrapper /opt/stack/neutron/neutron/common/utils.py:117}}
  Jan 23 07:04:39 DEBUG neutron.agent.dhcp.agent [-] resync (*): [Exception('test sync_state error')] {{(pid=798097) _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:401}}
  Jan 23 07:04:39 INFO neutron.agent.dhcp.agent [-] Synchronizing state

  
  [0] https://bugs.launchpad.net/neutron/+bug/1780370

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2095595/+subscriptions
  Thread Previous • Date Previous • • Thread Next