yahoo-eng-team team mailing list archive

[Dave Walker <email@xxxxxxxxxx>]

** Also affects: neutron/kilo
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1528031

Title:
  'NetworkNotFound' exception during listing ports

Status in neutron:
  Fix Released
Status in neutron kilo series:
  New

Bug description:
  There is a problem - when I run tests in parallel then one/two can fail.
  As I see in logs one thread is deleting network while second thread is
  listing all ports. And second thread get exception 'NetworkNotFound'.

  Part of neutron service logs is:

  2015-12-18 06:29:05.151 INFO neutron.wsgi [req-4d303e7d-ae31-47b5-a644-552fceeb03ef user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "DELETE /v2.0/networks/d2d2481a-4c20-452f-8088-6e6815694ac0.json HTTP/1.1" 204 173 0.426808
  2015-12-18 06:29:05.173 ERROR neutron.policy [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] Policy check error while calling <bound method Ml2Plugin.get_network of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7f1ffffaa950>>!
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy Traceback (most recent call last):
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/policy.py", line 258, in __call__
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     fields=[parent_field])
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 713, in get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     result = super(Ml2Plugin, self).get_network(context, id, None)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 385, in get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     network = self._get_network(context, id)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 188, in _get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     raise n_exc.NetworkNotFound(net_id=id)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy NetworkNotFound: Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found.
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy 
  2015-12-18 06:29:05.175 INFO neutron.api.v2.resource [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] index failed (client error): Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found.
  2015-12-18 06:29:05.175 INFO neutron.wsgi [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "GET /v2.0/ports.json?tenant_id=63f912ca152048c6a6b375784d90bd37 HTTP/1.1" 404 359 0.311871

  
  Answer from Kevin Benton (in mailing list):
  Ah, I believe what is happening is that the network is being deleted after the port has been retrieved from the database during the policy check. The policy check retrieves the port's network to be able to enforce the network_owner lookup: https://github.com/openstack/neutron/blob/master/etc/policy.json#L6

  So order of events seems to be:

  port list API call received
  ports retrieved from db
  network delete request is processed
  ports processed by policy engine
  policy engine triggers network lookup and hits 404

  
  This appears to be a legitimate bug. Maybe we need to find a way to cache the network at port retrieval time for the policy engine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1528031/+subscriptions