yahoo-eng-team team mailing list archive

[Boyvinall <1886969@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

In ussuri, configuring the neutron.conf bulk_reload_interval enables
bulk reload mode. The current code looks to be incompatible with
python3.

With the current ussuri code, which looks unchanged on master, I get the
following error in docker logs:

---
Running command: 'neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini'
+ exec neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
    timer()
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
    cb(*args, **kw)
  File "/usr/lib/python3.6/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
    waiter.switch()
  File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 154, in _reload_bulk_allocations
    for network_id in self._network_bulk_allocations.keys():
RuntimeError: dictionary changed size during iteration
---

After this, I see no further updates to the dnsmasq hosts file.

The current code looks like this:

https://github.com/openstack/neutron/blob/56bb42fcbc43b619c3c07897c7de88f29158e4b8/neutron/agent/dhcp/agent.py#L157
---
    def _reload_bulk_allocations(self):
        while True:
            for network_id in self._network_bulk_allocations.keys():
                network = self.cache.get_network_by_id(network_id)
                self.call_driver('bulk_reload_allocations', network)
                del self._network_bulk_allocations[network_id]
            eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
---

I think the problem is the "del" statement .. code like this works in
python2 but not in python3.  However, I also wonder if there's some race
hazard here with new IDs being appended.

I suspect something like this might work better:

---
    def _reload_bulk_allocations(self):
        while True:
            deleted = self._network_bulk_allocations.copy()
            self._network_bulk_allocations = {}

            for network_id in deleted:
                network = self.cache.get_network_by_id(network_id)
                self.call_driver('bulk_reload_allocations', network)
            eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
---

However, even this is probably susceptible to a race hazard. Probably
need a mutex around any update to self._network_bulk_allocations.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1886969

Title:
  dhcp bulk reload fails with python3

Status in neutron:
  New

Bug description:
  In ussuri, configuring the neutron.conf bulk_reload_interval enables
  bulk reload mode. The current code looks to be incompatible with
  python3.

  With the current ussuri code, which looks unchanged on master, I get
  the following error in docker logs:

  ---
  Running command: 'neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini'
  + exec neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini
  Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
      timer()
    File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/usr/lib/python3.6/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
      waiter.switch()
    File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 154, in _reload_bulk_allocations
      for network_id in self._network_bulk_allocations.keys():
  RuntimeError: dictionary changed size during iteration
  ---

  After this, I see no further updates to the dnsmasq hosts file.

  The current code looks like this:

  https://github.com/openstack/neutron/blob/56bb42fcbc43b619c3c07897c7de88f29158e4b8/neutron/agent/dhcp/agent.py#L157
  ---
      def _reload_bulk_allocations(self):
          while True:
              for network_id in self._network_bulk_allocations.keys():
                  network = self.cache.get_network_by_id(network_id)
                  self.call_driver('bulk_reload_allocations', network)
                  del self._network_bulk_allocations[network_id]
              eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
  ---

  I think the problem is the "del" statement .. code like this works in
  python2 but not in python3.  However, I also wonder if there's some
  race hazard here with new IDs being appended.

  I suspect something like this might work better:

  ---
      def _reload_bulk_allocations(self):
          while True:
              deleted = self._network_bulk_allocations.copy()
              self._network_bulk_allocations = {}

              for network_id in deleted:
                  network = self.cache.get_network_by_id(network_id)
                  self.call_driver('bulk_reload_allocations', network)
              eventlet.greenthread.sleep(self.conf.bulk_reload_interval)
  ---

  However, even this is probably susceptible to a race hazard. Probably
  need a mutex around any update to self._network_bulk_allocations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1886969/+subscriptions