yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #86483
[Bug 1933350] Re: [neutron-vpnaas] IPSec site connection freeze at PENDING_CREATE status after upgrade
*** This bug is a duplicate of bug 1794718 ***
https://bugs.launchpad.net/bugs/1794718
I confirmed this is a duplicate of bug 1794718 as mentioned above. Note
that the fix is included in stable/train. While stable/stein is in the
Extended-Maintenance phase, it may be good to backport it.
** This bug has been marked a duplicate of bug 1794718
Neutron VPNAAS don't update site connections on python3
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1933350
Title:
[neutron-vpnaas] IPSec site connection freeze at PENDING_CREATE status
after upgrade
Status in neutron:
New
Bug description:
[High level description]
After upgrade to Stein from Rocky(py2 to py3), we can observe that newly created IPSec Site Connections remains in status PENDING_CREATE even when everything is working correctly(connectivity from both sides of tunnel is reachable)
[Pre-conditions]
Created IPSec connection from one side and second side.
[Step-by-step reproduction steps]
All CLI commands where taken from here: https://docs.syseleven.de/syseleven-stack/en/howtos/vpnaas
From one side we have a Rocky OpenStack environment and on the other side there is a Stein OpenStack environment
[Expected output]
IPSec connection status should change from PENDING_CREATE to ACTIVE when connection is established from both ends
[Actual output]
Connection works as expected, but status is hanging in PENDING_CREATE state:
openstack vpn ipsec site connection show conn
+--------------------------+----------------------------------------------------+
| Field | Value |
+--------------------------+----------------------------------------------------+
| Authentication Algorithm | psk |
| Description | |
| ID | c2ae6b41-7eba-4b09-8263-07f5029aa7fb |
| IKE Policy | 82a303ac-2a26-4115-8b50-8f6bd6d18012 |
| IPSec Policy | 22608ad5-0490-4353-b826-06799ef4a584 |
| Initiator | bi-directional |
| Local Endpoint Group ID | 2abb9a94-72d8-4589-b25a-15fdb3b8f347 |
| Local ID | |
| MTU | 1500 |
| Name | conn |
| Peer Address | 195.167.157.152 |
| Peer CIDRs | |
| Peer Endpoint Group ID | 70d225f4-b63c-40f2-906e-bf4f11acd2cc |
| Peer ID | 195.167.157.152 |
| Pre-shared Key | secret |
| Project | 175e079b3aef47a38da16d125863fd9d |
| Route Mode | static |
| State | True |
| Status | PENDING_CREATE |
| VPN Service | 156059f1-155b-48c0-a389-e74b24f780be |
| dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} |
| project_id | 175e079b3aef47a38da16d125863fd9d |
+--------------------------+----------------------------------------------------+
[Version]
Affected system:
OpenStack Stein:
dpkg -l | grep neutron | grep ii
ii neutron-bgp-dragent 2:14.0.0-0ubuntu1~cloud0 all OpenStack Neutron Dynamic Routing - Agent
ii neutron-common 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-dynamic-routing-common 2:14.0.0-0ubuntu1~cloud0 all OpenStack Neutron Dynamic Routing - common files
ii neutron-fwaas-common 1:14.0.1-0ubuntu1~cloud0 all Firewall-as-a-Service driver for OpenStack Neutron
ii neutron-l3-agent 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-linuxbridge-agent 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - linuxbridge agent
ii neutron-metadata-agent 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - server
ii neutron-vpnaas-common 2:14.0.1-0ubuntu1~cloud0 all VPN-as-a-Service driver for OpenStack Neutron
ii python-neutronclient 1:6.11.0-0ubuntu1~cloud0 all client API library for Neutron - Python 2.7
ii python3-neutron 2:14.4.2-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - Python library
ii python3-neutron-dynamic-routing 2:14.0.0-0ubuntu1~cloud0 all OpenStack Neutron Dynamic Routing - Python 3 library
ii python3-neutron-fwaas 1:14.0.1-0ubuntu1~cloud0 all Firewall-as-a-Service driver for OpenStack Neutron
ii python3-neutron-lib 1.25.0-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x
ii python3-neutron-vpnaas 2:14.0.1-0ubuntu1~cloud0 all VPN-as-a-Service driver for OpenStack Neutron
ii python3-neutronclient 1:6.11.0-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x
[Logs]
There is no matching logs which can tell us more. The only difference which we could observe is an output from netnswrapper after upgrade to Stein. For Rocky it was parsed in every single line, when for Stein it is one-liner. Below you can see a difference:
Rocky:
ip netns exec qrouter-9e98710d-7c49-4dd5-b8ad-1bd18d4b1505 neutron-vpn-netns-wrapper --mount_paths=/etc:/var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1505/etc,/var/run:/var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1un --rootwrap_config=/etc/neutron/rootwrap.conf --cmd=ipsec,statusall
2021-06-23 15:53:39.774 20362 INFO neutron.common.config [-] Logging enabled!
2021-06-23 15:53:39.775 20362 INFO neutron.common.config [-] /usr/local/bin/neutron-vpn-netns-wrapper version 13.0.7
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1505/etc', '/etc'] Exit code: 0 Stdout: Stderr: 2021-06-23 13:53:39.789 20362 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1505/etc has been bind-mounted in /etc
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1505/var/run', '/var/run'] Exit code: 0 Stdout: Stderr: 2021-06-23 13:53:39.798 20362 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/9e98710d-7c49-4dd5-b8ad-1bd18d4b1505/var/run has been bind-mounted in /var/run
Command: ['ipsec', 'statusall'] Exit code: 0 Stdout: Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-128-generic, x86_64):
uptime: 27 minutes, since Jun 23 13:26:04 2021
malloc: sbrk 2715648, mmap 0, used 756384, free 1959264
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 8
loaded plugins: charon aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
169.254.192.31
169.254.0.106
10.2.0.1
195.167.157.152
Connections:
92f2ec5f-4560-44c0-91f6-f4377732cf64: 195.167.157.152...128.204.221.44 IKEv1, dpddelay=30s
92f2ec5f-4560-44c0-91f6-f4377732cf64: local: [195.167.157.152] uses pre-shared key authentication
92f2ec5f-4560-44c0-91f6-f4377732cf64: remote: [128.204.221.44] uses pre-shared key authentication
92f2ec5f-4560-44c0-91f6-f4377732cf64: child: 10.2.0.0/24 === 10.1.0.0/24 TUNNEL, dpdaction=hold
Routed Connections:
92f2ec5f-4560-44c0-91f6-f4377732cf64{1}: ROUTED, TUNNEL, reqid 1
92f2ec5f-4560-44c0-91f6-f4377732cf64{1}: 10.2.0.0/24 === 10.1.0.0/24
Security Associations (1 up, 0 connecting):
92f2ec5f-4560-44c0-91f6-f4377732cf64[2]: ESTABLISHED 27 minutes ago, 195.167.157.152[195.167.157.152]...128.204.221.44[128.204.221.44]
92f2ec5f-4560-44c0-91f6-f4377732cf64[2]: IKEv1 SPIs: 03ffeca9c34a2091_i 0b5453dd1f87684a_r*, pre-shared key reauthentication in 27 minutes
92f2ec5f-4560-44c0-91f6-f4377732cf64[2]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536
92f2ec5f-4560-44c0-91f6-f4377732cf64{4}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cc66ee1e_i c5178bb0_o
92f2ec5f-4560-44c0-91f6-f4377732cf64{4}: AES_CBC_128/HMAC_SHA1_96/MODP_1536, 0 bytes_i, 0 bytes_o, rekeying in 72 seconds
92f2ec5f-4560-44c0-91f6-f4377732cf64{4}: 10.2.0.0/24 === 10.1.0.0/24
Stein:
ip netns exec qrouter-a7e77168-d01f-4ace-ba3d-52c75962ffa3 neutron-vpn-netns-wrapper --mount_paths=/etc:/var/lib/neutron/ipsec/a7e77168-d01f-4ace-ba3d-52c75962ffa3/etc,/var/run:/var/lib/neutron/ipsec/a7e77168-d01f-4ace-ba3d-52c75962ffa3/var/run --rootwrap_config=/etc/neutron/rootwrap.conf --cmd=ipsec,statusall
2021-06-23 16:01:29.422 23058 INFO neutron.common.config [-] Logging enabled!
2021-06-23 16:01:29.424 23058 INFO neutron.common.config [-] /usr/bin/neutron-vpn-netns-wrapper version 14.4.2
Command: <map object at 0x7f9c78d96080> Exit code: 0 Stdout: b'' Stderr: b''2021-06-23 16:01:29.440 23058 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/a7e77168-d01f-4ace-ba3d-52c75962ffa3/etc has been bind-mounted in /etc
Command: <map object at 0x7f9c78d96048> Exit code: 0 Stdout: b'' Stderr: b''2021-06-23 16:01:29.449 23058 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/a7e77168-d01f-4ace-ba3d-52c75962ffa3/var/run has been bind-mounted in /var/run
Command: <map object at 0x7f9c78d96780> Exit code: 0 Stdout: b'Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-135-generic, x86_64):\n uptime: 35 minutes, since Jun 23 13:25:45 2021\n malloc: sbrk 2703360, mmap 0, used 770608, free 1932752\n worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 5\n loaded plugins: charon aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters\nListening IP addresses:\n 169.254.192.166\n 169.254.0.215\n 10.1.0.1\n 128.204.221.44\nConnections:\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb: 128.204.221.44...195.167.157.152 IKEv1, dpddelay=30s\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb: local: [128.204.221.44] uses pre-shared key authentication\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb: remote: [195.167.157.152] uses pre-shared key authentication\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb: child: 10.1.0.0/24 === 10.2.0.0/24 TUNNEL, dpdaction=hold\nRouted Connections:\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb{1}: ROUTED, TUNNEL, reqid 1\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb{1}: 10.1.0.0/24 === 10.2.0.0/24\nSecurity Associations (1 up, 0 connecting):\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb[1]: ESTABLISHED 35 minutes ago, 128.204.221.44[128.204.221.44]...195.167.157.152[195.167.157.152]\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb[1]: IKEv1 SPIs: 03ffeca9c34a2091_i* 0b5453dd1f87684a_r, pre-shared key reauthentication in 21 minutes\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb[1]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb{5}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: c0e9476e_i c46722ef_o\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb{5}: AES_CBC_128/HMAC_SHA1_96/MODP_1536, 0 bytes_i, 0 bytes_o, rekeying in 7 minutes\nc2ae6b41-7eba-4b09-8263-07f5029aa7fb{5}: 10.1.0.0/24 === 10.2.0.0/24\n' Stderr: b''[
If some other logs can be useful - do not hesitate to inform me, I will try to collect it, as I have debug set to True for neutron.conf.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1933350/+subscriptions
References
