yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622460] [NEW] neutron-server report 'FirewallNotFound' when delete firewall under l3_ha mode

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yi Zhao <zhaoyi44@xxxxxxxxxxx>
Date: Mon, 12 Sep 2016 07:04:33 -0000
Reply-to: Bug 1622460 <1622460@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When we delete a firewall under l3-ha mode, some neutron-servers reports
error logs: 'FirewallNotFound':

Environment:
* Openstack Kilo version
* Three neutron servers using ha-proxy in balance roundrobin mode, and provides VIP (keepalived)
* l3_ha=True is set in neutron-servers to provide L3 HA.
* Three l3-agents on 3 network nodes

We found 2 out of 3 neutron-servers print the following error logs

-----------
Error logs:
===========
2016-09-12 14:33:34.250 22722 DEBUG oslo_messaging._drivers.amqp [-] unpacked context: {u'read_deleted': u'no', u'project_name': u'zhaoyi', u'user_id': u'5f228382dd8d4001bd079cfab624e870', u'roles': [u'_member_', u'Member', u'user', u'admin'], u'tenant_id': u'd3147020bd1f4a709654b7e62885bd9f', u'auth_token': u'***', u'request_id': u'req-89116778-b4fb-4232-8249-500f1db5d3f8', u'is_admin': True, u'user': u'5f228382dd8d4001bd079cfab624e870', u'timestamp': u'2016-09-12 06:33:33.570294', u'tenant_name': u'zhaoyi', u'project_id': u'd3147020bd1f4a709654b7e62885bd9f', u'user_name': u'zhaoyi', u'tenant': u'd3147020bd1f4a709654b7e62885bd9f'} unpack_context /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203
2016-09-12 14:33:34.253 22722 DEBUG neutron_fwaas.services.firewall.fwaas_plugin [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] firewall_deleted() called firewall_deleted /usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py:62
2016-09-12 14:33:34.260 22722 DEBUG neutron_fwaas.db.firewall.firewall_db [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] delete_firewall() called delete_firewall /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:362
2016-09-12 14:33:34.303 22759 DEBUG keystoneclient.session [-] RESP: [200] content-length: 3084 x-subject-token: {SHA1}fb0b83f9b5d3a4b459a1f2845c1a1bd4bba3c008 vary: X-Auth-Token date: Mon, 12 Sep 2016 06:33:34 GMT content-type: application/json x-openstack-request-id: req-38cd15ab-f193-4d04-80bd-088638497b26
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "bf49f6a34f5b4ec8843efee8a840a8b3", "name": "Member"}, {"id": "a104bc435a5d4031b9712dd702cb8672", "name": "user"}, {"id": "11b9aa45b311407ba9460b95eb1534c2", "name": "admin"}], "expires_at": "2016-09-12T07:03:12.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "d3147020bd1f4a709654b7e62885bd9f", "name": "zhaoyi"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "5f228382dd8d4001bd079cfab624e870", "name": "zhaoyi"}, "audit_ids": ["Q9kbnQe8SzOcZ7ig6_-2ew", "guy9k1VsThmCGwfavbmvjw"], "issued_at": "2016-09-12T06:03:12.784462"}}
 _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-09-12 14:33:34.307 22759 DEBUG neutron_fwaas.db.firewall.firewall_db [req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] get_firewall_rules() called get_firewall_rules /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:534
2016-09-12 14:33:34.322 22759 INFO neutron.wsgi [req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] 10.65.0.99,10.65.0.99 - - [12/Sep/2016 14:33:34] "GET /v2.0/fw/firewall_rules.json HTTP/1.1" 200 6555 0.163946
2016-09-12 14:33:34.447 22722 ERROR oslo_messaging.rpc.dispatcher [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] Exception during message handling: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     self.plugin.delete_db_firewall_object(context, firewall_id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py", line 323, in delete_db_firewall_object
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     super(FirewallPlugin, self).delete_firewall(context, id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py", line 368, in delete_firewall
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: fwaas sg-fw

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1622460

Title:
  neutron-server report 'FirewallNotFound' when delete firewall under
  l3_ha mode

Status in neutron:
  New

Bug description:
  When we delete a firewall under l3-ha mode, some neutron-servers
  reports error logs: 'FirewallNotFound':

  Environment:
  * Openstack Kilo version
  * Three neutron servers using ha-proxy in balance roundrobin mode, and provides VIP (keepalived)
  * l3_ha=True is set in neutron-servers to provide L3 HA.
  * Three l3-agents on 3 network nodes

  We found 2 out of 3 neutron-servers print the following error logs

  -----------
  Error logs:
  ===========
  2016-09-12 14:33:34.250 22722 DEBUG oslo_messaging._drivers.amqp [-] unpacked context: {u'read_deleted': u'no', u'project_name': u'zhaoyi', u'user_id': u'5f228382dd8d4001bd079cfab624e870', u'roles': [u'_member_', u'Member', u'user', u'admin'], u'tenant_id': u'd3147020bd1f4a709654b7e62885bd9f', u'auth_token': u'***', u'request_id': u'req-89116778-b4fb-4232-8249-500f1db5d3f8', u'is_admin': True, u'user': u'5f228382dd8d4001bd079cfab624e870', u'timestamp': u'2016-09-12 06:33:33.570294', u'tenant_name': u'zhaoyi', u'project_id': u'd3147020bd1f4a709654b7e62885bd9f', u'user_name': u'zhaoyi', u'tenant': u'd3147020bd1f4a709654b7e62885bd9f'} unpack_context /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203
  2016-09-12 14:33:34.253 22722 DEBUG neutron_fwaas.services.firewall.fwaas_plugin [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] firewall_deleted() called firewall_deleted /usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py:62
  2016-09-12 14:33:34.260 22722 DEBUG neutron_fwaas.db.firewall.firewall_db [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] delete_firewall() called delete_firewall /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:362
  2016-09-12 14:33:34.303 22759 DEBUG keystoneclient.session [-] RESP: [200] content-length: 3084 x-subject-token: {SHA1}fb0b83f9b5d3a4b459a1f2845c1a1bd4bba3c008 vary: X-Auth-Token date: Mon, 12 Sep 2016 06:33:34 GMT content-type: application/json x-openstack-request-id: req-38cd15ab-f193-4d04-80bd-088638497b26
  RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "bf49f6a34f5b4ec8843efee8a840a8b3", "name": "Member"}, {"id": "a104bc435a5d4031b9712dd702cb8672", "name": "user"}, {"id": "11b9aa45b311407ba9460b95eb1534c2", "name": "admin"}], "expires_at": "2016-09-12T07:03:12.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "d3147020bd1f4a709654b7e62885bd9f", "name": "zhaoyi"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "5f228382dd8d4001bd079cfab624e870", "name": "zhaoyi"}, "audit_ids": ["Q9kbnQe8SzOcZ7ig6_-2ew", "guy9k1VsThmCGwfavbmvjw"], "issued_at": "2016-09-12T06:03:12.784462"}}
   _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
  2016-09-12 14:33:34.307 22759 DEBUG neutron_fwaas.db.firewall.firewall_db [req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] get_firewall_rules() called get_firewall_rules /usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py:534
  2016-09-12 14:33:34.322 22759 INFO neutron.wsgi [req-b354359e-4536-43bd-a0ca-6ffc27ef72d7 ] 10.65.0.99,10.65.0.99 - - [12/Sep/2016 14:33:34] "GET /v2.0/fw/firewall_rules.json HTTP/1.1" 200 6555 0.163946
  2016-09-12 14:33:34.447 22722 ERROR oslo_messaging.rpc.dispatcher [req-89116778-b4fb-4232-8249-500f1db5d3f8 ] Exception during message handling: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py", line 67, in firewall_deleted
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     self.plugin.delete_db_firewall_object(context, firewall_id)
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/services/firewall/fwaas_plugin.py", line 323, in delete_db_firewall_object
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     super(FirewallPlugin, self).delete_firewall(context, id)
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron_fwaas/db/firewall/firewall_db.py", line 368, in delete_firewall
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher     raise fw_ext.FirewallNotFound(firewall_id=id)
  2016-09-12 14:33:34.447 22722 TRACE oslo_messaging.rpc.dispatcher FirewallNotFound: Firewall 6278942e-6485-4ceb-92e5-8ddfa9fb4d25 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1622460/+subscriptions
Follow ups

[Bug 1622460] Re: neutron-server report 'FirewallNotFound' when delete firewall under l3_ha mode
From: Cedric Brandily, 2017-02-22