yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1940071] [NEW] Neutron VPNaaS - growing memory consumption

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Tomas <1940071@xxxxxxxxxxxxxxxxxx>
Date: Mon, 16 Aug 2021 10:43:53 -0000
Reply-to: Bug 1940071 <1940071@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I have problem with neutron vpnaas in Kolla-ansible (Victoria, from
source) on Ubuntu 20.04.2 LTS, kernel 5.4.0-65-generic #73-Ubuntu SMP
Mon Jan 18 17:25:17 UTC 2021 x86_64. Neutron client version 7.1.1,
neutron server version 17.1.3.dev22.


After enabling vpnaas plugin everything was ok at first. I was able to
create VPN connection and communication is correct. But after a week of
running vpnaas (only one VPN connection created/working!) I’ve noticed,
that neutron-vpnaas takes more and more memory. I have 5 processes on
each controller (there should be always five? Or it is changing
dynamically?):

42435    1545384  0.6  8.6 5802516 5712412 ?     S    Jun15 110:50 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
42435    1545389  0.6  8.5 5735832 5645856 ?     S    Jun15 112:16 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
42435    1545378  0.5  8.5 5734192 5643620 ?     S    Jun15 108:09 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
42435    1545372  0.5  8.5 5731128 5641436 ?     S    Jun15 109:07 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
42435    1545369  0.6  8.4 5637084 5547392 ?     S    Jun15 114:21 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf

now neutron_server takes over 27G of RAM on each controller:

neutron_server         running  10.2  27.2G

and forced controllers to use swapfile.

After stopping all neutron containers and starting it again neutron
takes a lot less memory:

neutron_server         running   0.5   583M

but memory usage keeps growing (about 5-6 M every minute).


There is only one, bidirectional vpn connection between project networks in two regions.
There is no (or almost no) traffic on this vpn link.
Second region is on one host (kolla all-in-one) and situation looks the same- with vpn service enabled there is excessive Memory use (growing also about 5-6MB/ minute and forcing the system to use swapfile).

There is no big cpu usage (ofcourse also due to no traffic inside VPN)
 It's the same case even if there's no vpn connection configured.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1940071

Title:
  Neutron VPNaaS - growing memory consumption

Status in neutron:
  New

Bug description:
  I have problem with neutron vpnaas in Kolla-ansible (Victoria, from
  source) on Ubuntu 20.04.2 LTS, kernel 5.4.0-65-generic #73-Ubuntu SMP
  Mon Jan 18 17:25:17 UTC 2021 x86_64. Neutron client version 7.1.1,
  neutron server version 17.1.3.dev22.


  After enabling vpnaas plugin everything was ok at first. I was able to
  create VPN connection and communication is correct. But after a week
  of running vpnaas (only one VPN connection created/working!) I’ve
  noticed, that neutron-vpnaas takes more and more memory. I have 5
  processes on each controller (there should be always five? Or it is
  changing dynamically?):

  42435    1545384  0.6  8.6 5802516 5712412 ?     S    Jun15 110:50 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
  42435    1545389  0.6  8.5 5735832 5645856 ?     S    Jun15 112:16 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
  42435    1545378  0.5  8.5 5734192 5643620 ?     S    Jun15 108:09 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
  42435    1545372  0.5  8.5 5731128 5641436 ?     S    Jun15 109:07 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf
  42435    1545369  0.6  8.4 5637084 5547392 ?     S    Jun15 114:21 /var/lib/kolla/venv/bin/python3.8 /var/lib/kolla/venv/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_vpnaas.conf

  now neutron_server takes over 27G of RAM on each controller:

  neutron_server         running  10.2  27.2G

  and forced controllers to use swapfile.

  After stopping all neutron containers and starting it again neutron
  takes a lot less memory:

  neutron_server         running   0.5   583M

  but memory usage keeps growing (about 5-6 M every minute).

  
  There is only one, bidirectional vpn connection between project networks in two regions.
  There is no (or almost no) traffic on this vpn link.
  Second region is on one host (kolla all-in-one) and situation looks the same- with vpn service enabled there is excessive Memory use (growing also about 5-6MB/ minute and forcing the system to use swapfile).

  There is no big cpu usage (ofcourse also due to no traffic inside VPN)
   It's the same case even if there's no vpn connection configured.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1940071/+subscriptions