yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #62291
[Bug 1524916] Re: neutron-ns-metadata-proxy uses ~25MB/router in production
Reviewed: https://review.openstack.org/431691
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3b22541a2aa9a5b06e2bff256701dbe24554c17c
Submitter: Jenkins
Branch: master
commit 3b22541a2aa9a5b06e2bff256701dbe24554c17c
Author: Daniel Alvarez <dalvarez@xxxxxxxxxx>
Date: Thu Feb 9 18:30:23 2017 +0000
Switch ns-metadata-proxy to haproxy
Due to the high memory footprint of current Python ns-metadata-proxy,
it has to be replaced with a lighter process to avoid OOM conditions in
large environments.
This patch spawns haproxy through a process monitor using a pidfile.
This allows tracking the process and respawn it if necessary as it was
done before. Also, it implements an upgrade path which consists of
detecting any running Python instance of ns-metadata-proxy and
replacing them by haproxy. Therefore, upgrades will take place by
simply restarting neutron-l3-agent and neutron-dhcp-agent.
According to /proc/<pid>/smaps, memory footprint goes down from ~50MB
to ~1.5MB.
Also, haproxy is added to bindep in order to ensure that it's installed.
UpgradeImpact
Depends-On: I36a5531cacc21c0d4bb7f20d4bec6da65d04c262
Depends-On: Ia37368a7ff38ea48c683a7bad76f87697e194b04
Closes-Bug: #1524916
Change-Id: I5a75cc582dca48defafb440207d10e2f7b4f218b
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1524916
Title:
neutron-ns-metadata-proxy uses ~25MB/router in production
Status in neutron:
Fix Released
Status in tripleo:
Triaged
Bug description:
[root@mac6cae8b61e442 memexplore]# ./memexplore.py all metadata-proxy | cut -c 1-67
25778 kB (pid 420) /usr/bin/python /bin/neutron-ns-metadata-proxy -
25774 kB (pid 1468) /usr/bin/python /bin/neutron-ns-metadata-proxy
25778 kB (pid 1472) /usr/bin/python /bin/neutron-ns-metadata-proxy
25770 kB (pid 1474) /usr/bin/python /bin/neutron-ns-metadata-proxy
26528 kB (pid 1489) /usr/bin/python /bin/neutron-ns-metadata-proxy
25778 kB (pid 1520) /usr/bin/python /bin/neutron-ns-metadata-proxy
25778 kB (pid 1738) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 1814) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 2024) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 3961) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 4076) /usr/bin/python /bin/neutron-ns-metadata-proxy
25770 kB (pid 4099) /usr/bin/python /bin/neutron-ns-metadata-proxy
[...]
25778 kB (pid 31386) /usr/bin/python /bin/neutron-ns-metadata-proxy
25778 kB (pid 31403) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 31416) /usr/bin/python /bin/neutron-ns-metadata-proxy
25778 kB (pid 31453) /usr/bin/python /bin/neutron-ns-metadata-proxy
25770 kB (pid 31483) /usr/bin/python /bin/neutron-ns-metadata-proxy
25770 kB (pid 31647) /usr/bin/python /bin/neutron-ns-metadata-proxy
25774 kB (pid 31743) /usr/bin/python /bin/neutron-ns-metadata-proxy
2,581,230 kB Total PSS
if we look explicitly at one of those processes we see:
# ./memexplore.py pss 24039
0 kB 7f97db981000-7f97dbb81000 ---p 0005f000 fd:00 4298776438 /usr/lib64/libpcre.so.1.2.0
0 kB 7f97dbb83000-7f97dbba4000 r-xp 00000000 fd:00 4298776486 /usr/lib64/libselinux.so.1
0 kB 7fff16ffe000-7fff17000000 r-xp 00000000 00:00 0 [vdso]
0 kB 7f97dacb5000-7f97dacd1000 r-xp 00000000 fd:00 4298779123 /usr/lib64/python2.7/lib-dynload/_io.so
0 kB 7f97d6a06000-7f97d6c05000 ---p 000b1000 fd:00 4298777149 /usr/lib64/libsqlite3.so.0.8.6
[...]
0 kB 7f97d813a000-7f97d8339000 ---p 0000b000 fd:00 4298779157 /usr/lib64/python2.7/lib-dynload/pyexpat.so
0 kB 7f97dbba4000-7f97dbda4000 ---p 00021000 fd:00 4298776486 /usr/lib64/libselinux.so.1
0 kB 7f9
7db4f7000-7f97db4fb000 r-xp 00000000 fd:00 4298779139 /usr/lib64/python2.7/lib-dynload/cStringIO.so
0 kB 7f97dc81e000-7f97dc81f000 rw-p 00000000 00:00 0
0 kB 7f97d8545000-7f97d8557000 r-xp 00000000 fd:00 4298779138 /usr/lib64/python2.7/lib-dynload/cPickle.so
0 kB 7f97d9fd3000-7f97d9fd7000 r-xp 00000000 fd:00 4298779165 /usr/lib64/python2.7/lib-dynload/timemodule.so
0 kB 7f97d99c4000-7f97d9bc3000 ---p 00002000 fd:00 4298779147 /usr/lib64/python2.7/lib-dynload/grpmodule.so
0 kB 7f97daedb000-7f97daede000 r-xp 00000000 fd:00 4298779121 /usr/lib64/python2.7/lib-dynload/_heapq.so
0 kB 7f97ddfd4000-7f97ddfd7000 r-xp 00000000 fd:00 4298779119 /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so
0 kB 7f97d8b67000-7f97d8b78000 r-xp 00000000 fd:00 4298779141 /usr/lib64/python2.7/lib-dynload/datetime.so
0 kB 7f97d7631000-7f97d7635000 r-xp 00000000 fd:00 4298776496 /usr/lib64/libuuid.so.1.3.0
0 kB 7f97dd59e000-7f97dd5a6000 r-xp 00000000 fd:00 4298779132 /usr/lib64/python2.7/lib-dynload/_ssl.so
0 kB 7f97dbfc0000-7f97dbfc2000 rw-p 00000000 00:00 0
0 kB 7f97dd332000-7f97dd394000 r-xp 00000000 fd:00 4298776137 /usr/lib64/libssl.so.1.0.1e
0 kB 7f97d6e22000-7f97d7021000 ---p 00004000 fd:00 6442649369 /usr/lib64/python2.7/site-packages/sqlalchemy/cresultproxy.so
0 kB 7f97d95bb000-7f97d97ba000 ---p 0000b000 fd:00 4298779156 /usr/lib64/python2.7/lib-dynload/parsermodule.so
0 kB 7f97da3dd000-7f97da3e0000 r-xp 00000000 fd:00 4298779129 /usr/lib64/python2.7/lib-dynload/_randommodule.so
0 kB 7f97dddcf000-7f97dddd3000 r-xp 00000000 fd:00 4298779125 /usr/lib64/python2.7/lib-dynload/_localemodule.so
0 kB 7f97da7e5000-7f97da7ea000 r-xp 00000000 fd:00 4298779136 /usr/lib64/python2.7/lib-dynload/binascii.so
2 kB 7f97e490a000-7f97e4ac0000 r-xp 00000000 fd:00 4299921917 /usr/lib64/libc-2.17.so
3 kB 7f97d6955000-7f97d6a06000 r-xp 00000000 fd:00 4298777149 /usr/lib64/libsqlite3.so.0.8.6
4 kB 7f97d7428000-7f97d7429000 r--p 00002000 fd:00 6442649368 /usr/lib64/python2.7/site-packages/sqlalchemy/cprocessors.so
4 kB 7f97d7630000-7f97d7631000 rw-p 00006000 fd:00 4298779128 /usr/lib64/python2.7/lib-dynload/_multiprocessing.so
4 kB 7f97d95a8000-7f97d95a9000 r--p 00010000 fd:00 2147488545 [...]
/usr/lib64/python2.7/site-packages/OpenSSL/SSL.so
16 kB 7f97d7c58000-7f97d7c5c000 rw-p 0001a000 fd:00 4298779115 /usr/lib64/python2.7/lib-dynload/_ctypes.so
16 kB 7f97dd32e000-7f97dd332000 rw-p 00000000 00:00 0
16 kB 7f97dd9b6000-7f97dd9bb000 rw-p 0000f000 fd:00 4298779130 /usr/lib64/python2.7/lib-dynload/_socketmodule.so
16 kB 7f97dd593000-7f97dd597000 r--p 00061000 fd:00 4298776137 /usr/lib64/libssl.so.1.0.1e
16 kB 7f97e4cc0000-7f97e4cc4000 r--p 001b6000 fd:00 4299921917 /usr/lib64/libc-2.17.so
20 kB 7f97db2ea000-7f97db2ef000 rw-p 0000a000 fd:00 4298779149 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
28 kB 7f97dd597000-7f97dd59e000 rw-p 00065000 fd:00 4298776137 /usr/lib64/libssl.so.1.0.1e
28 kB 7f97d95a9000-7f97d95b0000 rw-p 00011000 fd:00 2147488545 /usr/lib64/python2.7/site-packages/OpenSSL/crypto.so
40 kB 7f97daed1000-7f97daedb000 rw-p 0001c000 fd:00 4298779123 /usr/lib64/python2.7/lib-dynload/_io.so
48 kB 7f97dd322000-7f97dd32e000 rw-p 001d5000 fd:00 4298776134 /usr/lib64/libcrypto.so.1.0.1e
48 kB 7fff16e88000-7fff16ea9000 rw-p 00000000 00:00 0 [stack]
52 kB 7f97dccf3000-7f97dcd00000 r--p 000d0000 fd:00 4298778191 /usr/lib64/libkrb5.so.3.3
60 kB 7f97e59a7000-7f97e59b6000 rw-p 00000000 00:00 0
104 kB 7f97dd308000-7f97dd322000 r--p 001bb000 fd:00 4298776134 /usr/lib64/libcrypto.so.1.0.1e
156 kB 03b92000-03bd4000 rw-p 00000000 00:00 0 [heap]
220 kB 7f97e5969000-7f97e59a7000 rw-p 00179000 fd:00 4298778899 /usr/lib64/libpython2.7.so.1.0
532 kB 7f97e5b48000-7f97e5bcf000 rw-p 00000000 00:00 0
768 kB 7f97e5a54000-7f97e5b17000 rw-p 00000000 00:00 0
772 kB 7f97dabf4000-7f97dacb5000 rw-p 00000000 00:00 0
22192 kB 025d2000-03b92000 rw-p 00000000 00:00 0 [heap]
Total Pss: 25778 kB
Being the maximum responsible python's heap (the tool doesn't show
who's heap that is, but if I look manually it's /usr/bin/python2.7's
heap).
For reference, a bare python waiting on command line Pss is 2930kB,
984kB due to python heap.
memexplore can be found here: https://github.com/mangelajo/memexplore
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1524916/+subscriptions
References
