yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1658060] [NEW] FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yaohua Yan <1658060@xxxxxxxxxxxxxxxxxx>
Date: Fri, 20 Jan 2017 11:20:21 -0000
Reply-to: Bug 1658060 <1658060@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We have four nodes, and we deploy both the FWaaS and DVR services. When
deleting the firewall, we always get three FirewallNotFound exceptions.
At present, we believe that, in DVR environment, evey node would run a
L3-agent service. This causes a plugin corresponding to multiple agents.
And each agent will call back the plugin's firewall_deleted()
(neutron_fwaas/services/firewall/fwaas_plugin.py) to delete the instance
in DB, but only the first agent will succeed.

How to reproduce:
- first create a firewall applied to a DVR router
- then delete it

$ neutron router-show test-fwaas
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| distributed           | True                                 |
| external_gateway_info |                                      |
| ha                    | False                                |
| id                    | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
| name                  | test-fwaas                           |
| routes                |                                      |
| status                | ACTIVE                               |
| tenant_id             | fc170b1b8a9a467b9e1a63d85ced5a86     |
+-----------------------+--------------------------------------+
$ neutron firewall-create --name fw --router test-fwaas policy
Created a new firewall:
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| admin_state_up     | True                                 |
| description        |                                      |
| firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
| id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
| name               | fw                                   |
| router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
| status             | CREATED                              |
| tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
+--------------------+--------------------------------------+
$ neutron firewall-show fw
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| admin_state_up     | True                                 |
| description        |                                      |
| firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
| id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
| name               | fw                                   |
| router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
| status             | ACTIVE                               |
| tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
+--------------------+--------------------------------------+
$ neutron firewall-delete fw

$ less neutron-service_error.log
2017-01-20 17:19:44.526 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher
2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
2017-01-20 17:19:44.532 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher
2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
2017-01-20 17:19:44.627 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher
2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

- We have four nodes, and we deploy both the FWaaS and DVR services. When deleting the firewall, we always get three FirewallNotFound exceptions. At present, we believe that, in DVR environment, evey node would run a L3-agent service. This causes a plugin corresponding to multiple agents. And each agent will call back the plugin's firewall_deleted() (neutron_fwaas/services/firewall/fwaas_plugin.py) to delete the instance in DB, but only the first agent will succeed.
- 	
+ We have four nodes, and we deploy both the FWaaS and DVR services. When
+ deleting the firewall, we always get three FirewallNotFound exceptions.
+ At present, we believe that, in DVR environment, evey node would run a
+ L3-agent service. This causes a plugin corresponding to multiple agents.
+ And each agent will call back the plugin's firewall_deleted()
+ (neutron_fwaas/services/firewall/fwaas_plugin.py) to delete the instance
+ in DB, but only the first agent will succeed.
+ 
  How to reproduce:
  - first create a firewall applied to a DVR router
  - then delete it
  
  $ neutron router-show test-fwaas
  +-----------------------+--------------------------------------+
  | Field                 | Value                                |
  +-----------------------+--------------------------------------+
  | admin_state_up        | True                                 |
  | distributed           | True                                 |
  | external_gateway_info |                                      |
  | ha                    | False                                |
  | id                    | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | name                  | test-fwaas                           |
  | routes                |                                      |
  | status                | ACTIVE                               |
  | tenant_id             | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +-----------------------+--------------------------------------+
  $ neutron firewall-create --name fw --router test-fwaas policy
  Created a new firewall:
  +--------------------+--------------------------------------+
  | Field              | Value                                |
  +--------------------+--------------------------------------+
  | admin_state_up     | True                                 |
  | description        |                                      |
  | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
  | id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
  | name               | fw                                   |
  | router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | status             | CREATED                              |
  | tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +--------------------+--------------------------------------+
  $ neutron firewall-show fw
  +--------------------+--------------------------------------+
  | Field              | Value                                |
  +--------------------+--------------------------------------+
  | admin_state_up     | True                                 |
  | description        |                                      |
  | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
  | id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
  | name               | fw                                   |
  | router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | status             | ACTIVE                               |
  | tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +--------------------+--------------------------------------+
  $ neutron firewall-delete fw
  
  $ less neutron-service_error.log
  2017-01-20 17:19:44.526 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
  2017-01-20 17:19:44.532 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
  2017-01-20 17:19:44.627 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1658060

Title:
  FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR

Status in neutron:
  New

Bug description:
  We have four nodes, and we deploy both the FWaaS and DVR services.
  When deleting the firewall, we always get three FirewallNotFound
  exceptions. At present, we believe that, in DVR environment, evey node
  would run a L3-agent service. This causes a plugin corresponding to
  multiple agents. And each agent will call back the plugin's
  firewall_deleted() (neutron_fwaas/services/firewall/fwaas_plugin.py)
  to delete the instance in DB, but only the first agent will succeed.

  How to reproduce:
  - first create a firewall applied to a DVR router
  - then delete it

  $ neutron router-show test-fwaas
  +-----------------------+--------------------------------------+
  | Field                 | Value                                |
  +-----------------------+--------------------------------------+
  | admin_state_up        | True                                 |
  | distributed           | True                                 |
  | external_gateway_info |                                      |
  | ha                    | False                                |
  | id                    | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | name                  | test-fwaas                           |
  | routes                |                                      |
  | status                | ACTIVE                               |
  | tenant_id             | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +-----------------------+--------------------------------------+
  $ neutron firewall-create --name fw --router test-fwaas policy
  Created a new firewall:
  +--------------------+--------------------------------------+
  | Field              | Value                                |
  +--------------------+--------------------------------------+
  | admin_state_up     | True                                 |
  | description        |                                      |
  | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
  | id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
  | name               | fw                                   |
  | router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | status             | CREATED                              |
  | tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +--------------------+--------------------------------------+
  $ neutron firewall-show fw
  +--------------------+--------------------------------------+
  | Field              | Value                                |
  +--------------------+--------------------------------------+
  | admin_state_up     | True                                 |
  | description        |                                      |
  | firewall_policy_id | 1eb3fff7-240f-4f9d-adf6-766e2cad7f59 |
  | id                 | afd38a9e-cf0a-4667-94e0-853a888fd981 |
  | name               | fw                                   |
  | router_ids         | cfa3e65e-d101-4cc7-80e5-39daf72c6572 |
  | status             | ACTIVE                               |
  | tenant_id          | fc170b1b8a9a467b9e1a63d85ced5a86     |
  +--------------------+--------------------------------------+
  $ neutron firewall-delete fw

  $ less neutron-service_error.log
  2017-01-20 17:19:44.526 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.526 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.527 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
  2017-01-20 17:19:44.532 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.532 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.533 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']
  2017-01-20 17:19:44.627 19338 ERROR oslo_messaging.rpc.dispatcher [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Exception during message handling: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     fw_db = self.plugin._get_firewall(context, firewall_id)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher   File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.
  2017-01-20 17:19:44.627 19338 TRACE oslo_messaging.rpc.dispatcher
  2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] Returning exception Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found. to caller
  2017-01-20 17:19:44.628 19338 ERROR oslo_messaging._drivers.common [req-fa4e7f8c-20e7-4e67-8cd5-fef4c0738498 ] ['Traceback (most recent call last):\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/opt/openstack/.venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted\n    fw_db = self.plugin._get_firewall(context, firewall_id)\n', '  File "/opt/openstack/neutron-fwaas/neutron_fwaas/db/firewall/firewall_db.py", line 101, in _get_firewall\n    raise fw_ext.FirewallNotFound(firewall_id=id)\n', 'FirewallNotFound: Firewall 22c13294-b204-42bc-b592-beefe2b9f3c9 could not be found.\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1658060/+subscriptions
Follow ups

[Bug 1658060] Re: FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR
From: Reedip, 2017-03-21
[Bug 1658060] Re: FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR
From: Kevin Benton, 2017-03-21
[Bug 1658060] Re: FirewallNotFound exceptions when deleting the firewall in FWaaS-DVR
From: OpenStack Infra, 2017-02-09