yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1978571] [NEW] VPNaaS: creating ipsec site connection fails (in master)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bodo Petermann <1978571@xxxxxxxxxxxxxxxxxx>
Date: Tue, 14 Jun 2022 13:35:23 -0000
Reply-to: Bug 1978571 <1978571@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

With the current master branch of neutron and neutron-vpnaas requests to
create VPN ipsec site connections fail with a 500 Internal Server Error.
I noticed that in a devstack with vpnaas enabled.

The automated tests (neutron-tempest-plugin-vpnaas) also started to fail. 
See https://zuul.opendev.org/t/openstack/build/7f680cca45544cec9e3418bda7399324

In the neutron-server log there is an exception (see below).

This seems to be related to neutron commit 6b529ea3c559ed1a016e2aa2c30769429192d528
"[sqlalchemy-20] Use session.transaction information to decide if active"
https://review.opendev.org/c/openstack/neutron/+/843256

Probably the neutron-vpnaas code needs to be adapted to avoid the error
of remove_reservation inside a transaction.

Exception log:

Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] create failed: No details.: RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     context_reference.session.rollback()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return notify({self._resource: self._view(request.context,
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 504, in notify
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     quota.QUOTAS.commit_reservation(
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/__init__.py", line 103, in commit_reservation
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.get_driver().commit_reservation(context, reservation_id)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/quota/driver.py", line 271, in commit_reservation
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     quota_api.remove_reservation(context, reservation_id,
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/common/utils.py", line 700, in inner
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise RuntimeError(_("Method %s cannot be called within a "
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource 
Jun 12 08:46:58.912701 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: INFO neutron.wsgi [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] 149.202.162.36,149.202.162.36 "POST /v2.0/vpn/ipsec-site-connections HTTP/1.1" status: 500  len: 368 time: 0.2223299

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1978571

Title:
  VPNaaS: creating ipsec site connection fails (in master)

Status in neutron:
  New

Bug description:
  With the current master branch of neutron and neutron-vpnaas requests
  to create VPN ipsec site connections fail with a 500 Internal Server
  Error. I noticed that in a devstack with vpnaas enabled.

  The automated tests (neutron-tempest-plugin-vpnaas) also started to fail. 
  See https://zuul.opendev.org/t/openstack/build/7f680cca45544cec9e3418bda7399324

  In the neutron-server log there is an exception (see below).

  This seems to be related to neutron commit 6b529ea3c559ed1a016e2aa2c30769429192d528
  "[sqlalchemy-20] Use session.transaction information to decide if active"
  https://review.opendev.org/c/openstack/neutron/+/843256

  Probably the neutron-vpnaas code needs to be adapted to avoid the
  error of remove_reservation inside a transaction.

  Exception log:

  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] create failed: No details.: RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     context_reference.session.rollback()
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.force_reraise()
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise self.value
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     return notify({self._resource: self._view(request.context,
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 504, in notify
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     quota.QUOTAS.commit_reservation(
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/__init__.py", line 103, in commit_reservation
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     self.get_driver().commit_reservation(context, reservation_id)
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/quota/driver.py", line 271, in commit_reservation
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     quota_api.remove_reservation(context, reservation_id,
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/common/utils.py", line 700, in inner
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource     raise RuntimeError(_("Method %s cannot be called within a "
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
  Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource 
  Jun 12 08:46:58.912701 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: INFO neutron.wsgi [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] 149.202.162.36,149.202.162.36 "POST /v2.0/vpn/ipsec-site-connections HTTP/1.1" status: 500  len: 368 time: 0.2223299

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1978571/+subscriptions
Follow ups

[Bug 1978571] Re: VPNaaS: creating ipsec site connection fails (in master)
From: OpenStack Infra, 2022-06-20