yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1575402] [NEW] VPNaaS update NAT rules can generate stack trace

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Al Miller <al.miller@xxxxxxxxxxxx>
Date: Tue, 26 Apr 2016 22:36:01 -0000
Reply-to: Bug 1575402 <1575402@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

neutron-vpn-agent can generate stack traces with AttributeError:
'NoneType' object has no attribute 'ipv4' in two different locations in
ipsec.py, in add_nat_rule() and remove_nat_rule(), during sync()
operations while site connections are being created.

Here is an example stack trace (based on a Liberty distribution, but I
believe this is still an issue in master/newton):

2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 675, in vpnservice_updated
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self.sync(context, [router] if router else [])
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 830, in sync
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self._delete_vpn_processes(sync_router_ids, router_ids)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 860, in _delete_vpn_processes
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self.destroy_process(process_id)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 728, in destroy_process
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self._update_nat(vpnservice, self.remove_nat_rule)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 664, in _update_nat
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     top=True)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 629, in remove_nat_rule
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     iptables_manager.ipv4['nat'].remove_rule(chain, rule, top=top)
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'NoneType' object has no attribute 'ipv4'
2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1575402

Title:
  VPNaaS update NAT rules can generate stack trace

Status in neutron:
  New

Bug description:
  neutron-vpn-agent can generate stack traces with AttributeError:
  'NoneType' object has no attribute 'ipv4' in two different locations
  in ipsec.py, in add_nat_rule() and remove_nat_rule(), during sync()
  operations while site connections are being created.

  Here is an example stack trace (based on a Liberty distribution, but I
  believe this is still an issue in master/newton):

  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 675, in vpnservice_updated
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self.sync(context, [router] if router else [])
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 830, in sync
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self._delete_vpn_processes(sync_router_ids, router_ids)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 860, in _delete_vpn_processes
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self.destroy_process(process_id)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 728, in destroy_process
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     self._update_nat(vpnservice, self.remove_nat_rule)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 664, in _update_nat
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     top=True)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/venv/neutron-20160426T025546Z/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 629, in remove_nat_rule
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher     iptables_manager.ipv4['nat'].remove_rule(chain, rule, top=top)
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'NoneType' object has no attribute 'ipv4'
  2016-04-26 20:26:27.894 28022 ERROR oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1575402/+subscriptions
Follow ups

[Bug 1575402] Re: VPNaaS update NAT rules can generate stack trace
From: Ryan Moats, 2016-04-27