yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1524916] [NEW] neutron-ns-metadata-proxy uses ~65MB/router in production, generating OOM situations

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miguel Angel Ajo <majopela@xxxxxxxxxx>
Date: Thu, 10 Dec 2015 17:26:07 -0000
Reply-to: Bug 1524916 <1524916@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

pid	uid	tgid	total_vm	rss	nr_ptes	swapents	oom_score_adj	name
18903	0	18903	123420	15890	60	0	0	ovs-vswitchd
1887	998	1887	83584	12030	114	1155	0	neutron-
4516	998	4516	76571	12535	104	1247	0	neutron-l3-
4517	998	4517	74964	11477	102	1402	0	neutron-dhcp-
2179	0	2179	67815	6909	84	0	0	neutron-ns-
10912	0	10912	67813	6866	84	0	0	neutron-ns-
28570	0	28570	67783	6860	86	0	0	neutron-ns-
27834	0	27834	67752	6860	84	0	0	neutron-ns-
7196	0	7196	67752	6822	83	0	0	neutron-ns-
19443	0	19443	67752	6820	83	0	0	neutron-ns-
18629	0	18629	67752	6562	85	275	0	neutron-ns-
19307	0	19307	67750	6844	84	0	0	neutron-ns-
27887	0	27887	67750	6832	84	0	0	neutron-ns-
10733	0	10733	67750	6824	85	0	0	neutron-ns-
3929	0	3929	67750	6799	85	0	0	neutron-ns-
6399	0	6399	67750	6797	86	0	0	neutron-ns-
20011	0	20011	67750	6796	84	0	0	neutron-ns-
18142	0	18142	67750	2077	84	4753	0	neutron-ns-
16404	0	16404	67750	478	85	6330	0	neutron-ns-
28529	0	28529	67749	6833	86	0	0	neutron-ns-
9930	0	9930	67749	6813	85	0	0	neutron-ns-
18327	0	18327	67749	6812	87	0	0	neutron-ns-
27973	0	27973	67749	6809	85	0	0	neutron-ns-
31444	0	31444	67749	6785	82	0	0	neutron-ns-
28359	0	28359	67749	6784	84	0	0	neutron-ns-
6320	0	6320	67749	499	84	6329	0	neutron-ns-
13629	0	13629	67747	6829	82	0	0	neutron-ns-
24105	0	24105	67747	6813	84	0	0	neutron-ns-
12936	0	12936	67747	6811	86	0	0	neutron-ns-
17217	0	17217	67747	6809	84	0	0	neutron-ns-
8935	0	8935	67747	6809	85	0	0	neutron-ns-
25269	0	25269	67747	6809	85	0	0	neutron-ns-
24489	0	24489	67747	6793	85	0	0	neutron-ns-
19105	0	19105	67747	6776	83	0	0	neutron-ns-
18218	0	18218	67747	3084	84	3693	0	neutron-ns-
16140	0	16140	67747	491	85	6330	0	neutron-ns-
31726	0	31726	67578	6645	85	0	0	neutron-ns-
30083	0	30083	67578	6639	85	0	0	neutron-ns-
28205	0	28205	67578	6637	84	0	0	neutron-ns-
27765	0	27765	67578	6633	85	0	0	neutron-ns-
16708	0	16708	67578	6629	85	0	0	neutron-ns-
23659	0	23659	67578	6619	84	0	0	neutron-ns-
28103	0	28103	67578	6618	84	0	0	neutron-ns-
14483	0	14483	67578	6617	85	0	0	neutron-ns-
30295	0	30295	67578	6616	84	0	0	neutron-ns-
26095	0	26095	67578	6606	82	0	0	neutron-ns-
25878	0	25878	67578	6602	84	0	0	neutron-ns-
29938	0	29938	67578	6602	84	0	0	neutron-ns-
26124	0	26124	67578	6597	84	0	0	neutron-ns-
28612	0	28612	67578	6591	84	0	0	neutron-ns-
30012	0	30012	67578	6591	87	0	0	neutron-ns-
28760	0	28760	67578	6583	84	0	0	neutron-ns-
18570	0	18570	67578	6230	84	389	0	neutron-ns-
21065	0	21065	67578	6062	85	566	0	neutron-ns-
25686	0	25686	67578	6054	83	591	0	neutron-ns-
20718	0	20718	67578	6005	83	588	0	neutron-ns-
25615	0	25615	67578	5975	86	653	0	neutron-ns-
21082	0	21082	67578	5959	84	625	0	neutron-ns-
24981	0	24981	67578	5937	84	677	0	neutron-ns-
20634	0	20634	67578	5812	83	821	0	neutron-ns-
18060	0	18060	67578	1186	84	5439	0	neutron-ns-
5863	0	5863	67578	318	84	6308	0	neutron-ns-
5959	0	5959	67578	312	84	6302	0	neutron-ns-
13016	0	13016	67578	309	84	6331	0	neutron-ns-
16058	0	16058	67578	309	82	6330	0	neutron-ns-
5870	0	5870	67578	308	84	6330	0	neutron-ns-
15960	0	15960	67578	305	84	6328	0	neutron-ns-
13531	0	13531	67578	303	83	6331	0	neutron-ns-
7914	0	7914	67578	298	84	6330	0	neutron-ns-
8716	0	8716	67578	294	84	6320	0	neutron-ns-
16011	0	16011	67578	286	84	6330	0	neutron-ns-
11016	0	11016	67578	281	83	6329	0	neutron-ns-
15734	0	15734	67578	280	84	6332	0	neutron-ns-
11003	0	11003	67578	273	85	6328	0	neutron-ns-
13587	0	13587	67578	272	84	6329	0	neutron-ns-
13498	0	13498	67578	270	84	6330	0	neutron-ns-
31775	0	31775	67555	6640	84	0	0	neutron-ns-
28001	0	28001	67555	6609	82	0	0	neutron-ns-
30294	0	30294	67555	6608	87	0	0	neutron-ns-
6840	0	6840	67555	6600	84	0	0	neutron-ns-
29743	0	29743	67555	6593	84	0	0	neutron-ns-
30774	0	30774	67555	6585	84	0	0	neutron-ns-
25922	0	25922	67555	6582	82	0	0	neutron-ns-
23379	0	23379	67555	6292	82	308	0	neutron-ns-
18512	0	18512	67555	6111	84	511	0	neutron-ns-
21033	0	21033	67555	5952	84	680	0	neutron-ns-
23223	0	23223	67555	5951	85	646	0	neutron-ns-
23490	0	23490	67555	5826	84	778	0	neutron-ns-
23808	0	23808	67555	5769	87	873	0	neutron-ns-
20898	0	20898	67555	5590	84	995	0	neutron-ns-
8888	0	8888	67555	317	84	6328	0	neutron-ns-
8619	0	8619	67555	306	85	6319	0	neutron-ns-
8613	0	8613	67555	299	84	6322	0	neutron-ns-
15582	0	15582	67555	296	84	6332	0	neutron-ns-
13678	0	13678	67555	295	83	6333	0	neutron-ns-
8352	0	8352	67555	293	84	6330	0	neutron-ns-
5952	0	5952	67555	287	83	6320	0	neutron-ns-
13331	0	13331	67555	285	84	6331	0	neutron-ns-
8516	0	8516	67555	284	84	6327	0	neutron-ns-
11090	0	11090	67555	282	86	6331	0	neutron-ns-
6329	0	6329	67555	274	84	6332	0	neutron-ns-
10780	0	10780	67555	274	82	6326	0	neutron-ns-
15722	0	15722	67555	261	84	6332	0	neutron-ns-


In this dump we had a total usage of 6.9GB for 99 routers.

I would be glad to see a go or C replacement for this simple process,
other options could be just relaying all the requests to a single host
via any netns networking trick, but I couldn't think of any reasonable
option for that yet.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1524916

Title:
  neutron-ns-metadata-proxy uses ~65MB/router in production, generating
  OOM situations

Status in neutron:
  New

Bug description:
  pid	uid	tgid	total_vm	rss	nr_ptes	swapents	oom_score_adj	name
  18903	0	18903	123420	15890	60	0	0	ovs-vswitchd
  1887	998	1887	83584	12030	114	1155	0	neutron-
  4516	998	4516	76571	12535	104	1247	0	neutron-l3-
  4517	998	4517	74964	11477	102	1402	0	neutron-dhcp-
  2179	0	2179	67815	6909	84	0	0	neutron-ns-
  10912	0	10912	67813	6866	84	0	0	neutron-ns-
  28570	0	28570	67783	6860	86	0	0	neutron-ns-
  27834	0	27834	67752	6860	84	0	0	neutron-ns-
  7196	0	7196	67752	6822	83	0	0	neutron-ns-
  19443	0	19443	67752	6820	83	0	0	neutron-ns-
  18629	0	18629	67752	6562	85	275	0	neutron-ns-
  19307	0	19307	67750	6844	84	0	0	neutron-ns-
  27887	0	27887	67750	6832	84	0	0	neutron-ns-
  10733	0	10733	67750	6824	85	0	0	neutron-ns-
  3929	0	3929	67750	6799	85	0	0	neutron-ns-
  6399	0	6399	67750	6797	86	0	0	neutron-ns-
  20011	0	20011	67750	6796	84	0	0	neutron-ns-
  18142	0	18142	67750	2077	84	4753	0	neutron-ns-
  16404	0	16404	67750	478	85	6330	0	neutron-ns-
  28529	0	28529	67749	6833	86	0	0	neutron-ns-
  9930	0	9930	67749	6813	85	0	0	neutron-ns-
  18327	0	18327	67749	6812	87	0	0	neutron-ns-
  27973	0	27973	67749	6809	85	0	0	neutron-ns-
  31444	0	31444	67749	6785	82	0	0	neutron-ns-
  28359	0	28359	67749	6784	84	0	0	neutron-ns-
  6320	0	6320	67749	499	84	6329	0	neutron-ns-
  13629	0	13629	67747	6829	82	0	0	neutron-ns-
  24105	0	24105	67747	6813	84	0	0	neutron-ns-
  12936	0	12936	67747	6811	86	0	0	neutron-ns-
  17217	0	17217	67747	6809	84	0	0	neutron-ns-
  8935	0	8935	67747	6809	85	0	0	neutron-ns-
  25269	0	25269	67747	6809	85	0	0	neutron-ns-
  24489	0	24489	67747	6793	85	0	0	neutron-ns-
  19105	0	19105	67747	6776	83	0	0	neutron-ns-
  18218	0	18218	67747	3084	84	3693	0	neutron-ns-
  16140	0	16140	67747	491	85	6330	0	neutron-ns-
  31726	0	31726	67578	6645	85	0	0	neutron-ns-
  30083	0	30083	67578	6639	85	0	0	neutron-ns-
  28205	0	28205	67578	6637	84	0	0	neutron-ns-
  27765	0	27765	67578	6633	85	0	0	neutron-ns-
  16708	0	16708	67578	6629	85	0	0	neutron-ns-
  23659	0	23659	67578	6619	84	0	0	neutron-ns-
  28103	0	28103	67578	6618	84	0	0	neutron-ns-
  14483	0	14483	67578	6617	85	0	0	neutron-ns-
  30295	0	30295	67578	6616	84	0	0	neutron-ns-
  26095	0	26095	67578	6606	82	0	0	neutron-ns-
  25878	0	25878	67578	6602	84	0	0	neutron-ns-
  29938	0	29938	67578	6602	84	0	0	neutron-ns-
  26124	0	26124	67578	6597	84	0	0	neutron-ns-
  28612	0	28612	67578	6591	84	0	0	neutron-ns-
  30012	0	30012	67578	6591	87	0	0	neutron-ns-
  28760	0	28760	67578	6583	84	0	0	neutron-ns-
  18570	0	18570	67578	6230	84	389	0	neutron-ns-
  21065	0	21065	67578	6062	85	566	0	neutron-ns-
  25686	0	25686	67578	6054	83	591	0	neutron-ns-
  20718	0	20718	67578	6005	83	588	0	neutron-ns-
  25615	0	25615	67578	5975	86	653	0	neutron-ns-
  21082	0	21082	67578	5959	84	625	0	neutron-ns-
  24981	0	24981	67578	5937	84	677	0	neutron-ns-
  20634	0	20634	67578	5812	83	821	0	neutron-ns-
  18060	0	18060	67578	1186	84	5439	0	neutron-ns-
  5863	0	5863	67578	318	84	6308	0	neutron-ns-
  5959	0	5959	67578	312	84	6302	0	neutron-ns-
  13016	0	13016	67578	309	84	6331	0	neutron-ns-
  16058	0	16058	67578	309	82	6330	0	neutron-ns-
  5870	0	5870	67578	308	84	6330	0	neutron-ns-
  15960	0	15960	67578	305	84	6328	0	neutron-ns-
  13531	0	13531	67578	303	83	6331	0	neutron-ns-
  7914	0	7914	67578	298	84	6330	0	neutron-ns-
  8716	0	8716	67578	294	84	6320	0	neutron-ns-
  16011	0	16011	67578	286	84	6330	0	neutron-ns-
  11016	0	11016	67578	281	83	6329	0	neutron-ns-
  15734	0	15734	67578	280	84	6332	0	neutron-ns-
  11003	0	11003	67578	273	85	6328	0	neutron-ns-
  13587	0	13587	67578	272	84	6329	0	neutron-ns-
  13498	0	13498	67578	270	84	6330	0	neutron-ns-
  31775	0	31775	67555	6640	84	0	0	neutron-ns-
  28001	0	28001	67555	6609	82	0	0	neutron-ns-
  30294	0	30294	67555	6608	87	0	0	neutron-ns-
  6840	0	6840	67555	6600	84	0	0	neutron-ns-
  29743	0	29743	67555	6593	84	0	0	neutron-ns-
  30774	0	30774	67555	6585	84	0	0	neutron-ns-
  25922	0	25922	67555	6582	82	0	0	neutron-ns-
  23379	0	23379	67555	6292	82	308	0	neutron-ns-
  18512	0	18512	67555	6111	84	511	0	neutron-ns-
  21033	0	21033	67555	5952	84	680	0	neutron-ns-
  23223	0	23223	67555	5951	85	646	0	neutron-ns-
  23490	0	23490	67555	5826	84	778	0	neutron-ns-
  23808	0	23808	67555	5769	87	873	0	neutron-ns-
  20898	0	20898	67555	5590	84	995	0	neutron-ns-
  8888	0	8888	67555	317	84	6328	0	neutron-ns-
  8619	0	8619	67555	306	85	6319	0	neutron-ns-
  8613	0	8613	67555	299	84	6322	0	neutron-ns-
  15582	0	15582	67555	296	84	6332	0	neutron-ns-
  13678	0	13678	67555	295	83	6333	0	neutron-ns-
  8352	0	8352	67555	293	84	6330	0	neutron-ns-
  5952	0	5952	67555	287	83	6320	0	neutron-ns-
  13331	0	13331	67555	285	84	6331	0	neutron-ns-
  8516	0	8516	67555	284	84	6327	0	neutron-ns-
  11090	0	11090	67555	282	86	6331	0	neutron-ns-
  6329	0	6329	67555	274	84	6332	0	neutron-ns-
  10780	0	10780	67555	274	82	6326	0	neutron-ns-
  15722	0	15722	67555	261	84	6332	0	neutron-ns-

  
  In this dump we had a total usage of 6.9GB for 99 routers.

  I would be glad to see a go or C replacement for this simple process,
  other options could be just relaying all the requests to a single host
  via any netns networking trick, but I couldn't think of any reasonable
  option for that yet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1524916/+subscriptions
Follow ups

[Bug 1524916] Re: neutron-ns-metadata-proxy uses ~25MB/router in production
From: Ben Nemec, 2017-08-09
[Bug 1524916] Re: neutron-ns-metadata-proxy uses ~25MB/router in production
From: OpenStack Infra, 2017-03-14
[Bug 1524916] Re: neutron-ns-metadata-proxy uses ~25MB/router in production
From: Jakub Libosvar, 2017-03-07