yahoo-eng-team team mailing list archive

[OpenStack Infra <1528031@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/273034
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=77de9653fd60a802b11f157972f7b3e81497e8a7
Submitter: Jenkins
Branch:    master

commit 77de9653fd60a802b11f157972f7b3e81497e8a7
Author: Kevin Benton <blak111@xxxxxxxxx>
Date:   Wed Jan 27 05:18:13 2016 -0800

    Raise RetryRequest on policy parent not found
    
    During a port list operation, a port and its parent network
    may be concurrently deleted from the database after they have
    been retrieved from the DB but before policy is enforced.
    Then when the policy engine tries to do a get_network to check
    network ownership for a port on a network that no longer exists,
    it will encounter a NetworkNotFound exception from the core plugin.
    
    This exception was being propagated all of the way up to the whole
    API operation as a 404, which made no sense in the context of a
    port list.
    
    This patch adjusts the logic to catch any NotFound exceptions during
    this processing and convert them into a RetryRequest to trigger the
    API to restart the operation. At this point the objects will be gone
    from the database so the problematic items will not be passed to the
    policy engine for enforcement.
    
    Closes-Bug: #1528031
    Change-Id: I89d12fe0767e1c7ecb68138b5f6f17aa68a68769


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1528031

Title:
  'NetworkNotFound' exception during listing ports

Status in neutron:
  Fix Released

Bug description:
  There is a problem - when I run tests in parallel then one/two can fail.
  As I see in logs one thread is deleting network while second thread is
  listing all ports. And second thread get exception 'NetworkNotFound'.

  Part of neutron service logs is:

  2015-12-18 06:29:05.151 INFO neutron.wsgi [req-4d303e7d-ae31-47b5-a644-552fceeb03ef user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "DELETE /v2.0/networks/d2d2481a-4c20-452f-8088-6e6815694ac0.json HTTP/1.1" 204 173 0.426808
  2015-12-18 06:29:05.173 ERROR neutron.policy [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] Policy check error while calling <bound method Ml2Plugin.get_network of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7f1ffffaa950>>!
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy Traceback (most recent call last):
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/policy.py", line 258, in __call__
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     fields=[parent_field])
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 713, in get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     result = super(Ml2Plugin, self).get_network(context, id, None)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 385, in get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     network = self._get_network(context, id)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy   File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 188, in _get_network
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy     raise n_exc.NetworkNotFound(net_id=id)
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy NetworkNotFound: Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found.
  2015-12-18 06:29:05.173 22048 ERROR neutron.policy 
  2015-12-18 06:29:05.175 INFO neutron.api.v2.resource [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] index failed (client error): Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found.
  2015-12-18 06:29:05.175 INFO neutron.wsgi [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "GET /v2.0/ports.json?tenant_id=63f912ca152048c6a6b375784d90bd37 HTTP/1.1" 404 359 0.311871

  
  Answer from Kevin Benton (in mailing list):
  Ah, I believe what is happening is that the network is being deleted after the port has been retrieved from the database during the policy check. The policy check retrieves the port's network to be able to enforce the network_owner lookup: https://github.com/openstack/neutron/blob/master/etc/policy.json#L6

  So order of events seems to be:

  port list API call received
  ports retrieved from db
  network delete request is processed
  ports processed by policy engine
  policy engine triggers network lookup and hits 404

  
  This appears to be a legitimate bug. Maybe we need to find a way to cache the network at port retrieval time for the policy engine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1528031/+subscriptions