yahoo-eng-team team mailing list archive

[OpenStack Infra <1886969@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/742363
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=20b138ff3118029e86f0525695160c4c7ca8b551
Submitter: Zuul
Branch:    master

commit 20b138ff3118029e86f0525695160c4c7ca8b551
Author: Matt Vinall <boyvinall@xxxxxxxxx>
Date:   Thu Jul 9 21:08:21 2020 +0100

    fix dhcp bulk reload exceptions
    
    1886969 - The bulk reload code was written for python2 and caused
    an exception running under python3. This change works under python3.
    
    1890027 - There was an additional exception triggered when
    deleting networks - reading the network from the cache returned 'None'
    and this was not properly checked before use.
    
    Change-Id: I4e546c0e37146b1f34d8b5e6637c407b0c04ad4d
    Closes-Bug: 1886969
    Closes-Bug: 1890027
    Signed-off-by: Matt Vinall <boyvinall@xxxxxxxxx>


** Changed in: neutron
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1886969

Title:
  dhcp bulk reload fails with python3

Status in neutron:
  Fix Released

Bug description:
  In ussuri, configuring the neutron.conf bulk_reload_interval enables
  bulk reload mode. The current code looks to be incompatible with
  python3.

  With the current ussuri code, which looks unchanged on master, I get
  the following error in docker logs:

  ---
  Running command: 'neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini'
  + exec neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini
  Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
      timer()
    File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/usr/lib/python3.6/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
      waiter.switch()
    File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 154, in _reload_bulk_allocations
      for network_id in self._network_bulk_allocations.keys():
  RuntimeError: dictionary changed size during iteration
  ---

  After this, I see no further updates to the dnsmasq hosts file.

  The current code looks like this:

  https://github.com/openstack/neutron/blob/56bb42fcbc43b619c3c07897c7de88f29158e4b8/neutron/agent/dhcp/agent.py#L157
  ---
      def _reload_bulk_allocations(self):
          while True:
              for network_id in self._network_bulk_allocations.keys():
                  network = self.cache.get_network_by_id(network_id)
                  self.call_driver('bulk_reload_allocations', network)
                  del self._network_bulk_allocations[network_id]
              eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
  ---

  I think the problem is the "del" statement .. code like this works in
  python2 but not in python3.  However, I also wonder if there's some
  race hazard here with new IDs being appended.

  I suspect something like this might work better:

  ---
      def _reload_bulk_allocations(self):
          while True:
              deleted = self._network_bulk_allocations.copy()
              self._network_bulk_allocations = {}

              for network_id in deleted:
                  network = self.cache.get_network_by_id(network_id)
                  self.call_driver('bulk_reload_allocations', network)
              eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
  ---

  However, even this is probably susceptible to a race hazard. Probably
  need a mutex around any update to self._network_bulk_allocations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1886969/+subscriptions