yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1595878] [NEW] Memory leak in unit tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date: Fri, 24 Jun 2016 08:56:56 -0000
Reply-to: Bug 1595878 <1595878@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

tests.unit.agent.ovsdb.native.test_connection.TestOVSNativeConnection
calls Connection.start() which starts a daemon with a while True loop
full of mocks. mock._CallList of those mocks start to grow very quick
and finally eat all available memory.

mem_top output during unit tests run:

refs:
18118	<class 'mock.mock._CallList'> [call(1),
 call().get_nowait(),
 call().get_nowait().do_commit(),
 call().get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
 call().task_do
18117	<class 'mock.mock._CallList'> [call.get_nowait(),
 call.get_nowait().do_commit(),
 call.get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
 call.task_done(),
 call.get_no
17990	<class 'mock.mock._CallList'> [call(1),
 call().get_nowait(),
 call().get_nowait().do_commit(),
 call().get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
 call().task_do
17989	<class 'mock.mock._CallList'> [call.get_nowait(),
 call.get_nowait().do_commit(),
 call.get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
 call.task_done(),
 call.get_no
13592	<class 'mock.mock._CallList'> [call(),
 call().fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
 call().timer_wait(<Mock id='139952614993680'>),
 call().block(),
 call().fd_wait(<MagicMock name=
13591	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
 call.timer_wait(<Mock id='139952614993680'>),
 call.block(),
 call.fd_wait(<MagicMock name='TransactionQueue
13591	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
 call.timer_wait(<Mock id='139952614993680'>),
 call.block(),
 call.fd_wait(<MagicMock name='TransactionQueue
13588	<class 'mock.mock._CallList'> [call(),
 call().do_commit(),
 call().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
 call(),
 call().do_commit(),
 call().results.put(<MagicMock nam
13496	<class 'mock.mock._CallList'> [call(),
 call().fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
 call().timer_wait(<Mock id='139952687478800'>),
 call().block(),
 call().fd_wait(<MagicMock name=
13495	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
 call.timer_wait(<Mock id='139952687478800'>),
 call.block(),
 call.fd_wait(<MagicMock name='TransactionQueue
13495	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
 call.timer_wait(<Mock id='139952687478800'>),
 call.block(),
 call.fd_wait(<MagicMock name='TransactionQueue
13492	<class 'mock.mock._CallList'> [call(),
 call().do_commit(),
 call().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
 call(),
 call().do_commit(),
 call().results.put(<MagicMock nam
11550	<type 'dict'> {'keystoneclient.service_catalog': <module 'keystoneclient.service_catalog' from '/opt/stack/neutron/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/service_catalog.pyc'>, 'oslo_messaging.r
9061	<class 'mock.mock._CallList'> [call(<Mock id='139952630613712'>, <Mock name='get_schema_helper()' id='139952630616016'>),
 call().wait(<MagicMock name='Poller()' id='139952641693968'>),
 call().run(),
 call().wait(<MagicMock name=
9060	<class 'mock.mock._CallList'> [call.wait(<MagicMock name='Poller()' id='139952641693968'>),
 call.run(),
 call.wait(<MagicMock name='Poller()' id='139952641693968'>),
 call.run(),
 call.wait(<MagicMock name='Poller()' id='13995264
9060	<class 'mock.mock._CallList'> [call.wait(<MagicMock name='Poller()' id='139952641693968'>),
 call.run(),
 call.wait(<MagicMock name='Poller()' id='139952641693968'>),
 call.run(),
 call.wait(<MagicMock name='Poller()' id='13995264
9059	<class 'mock.mock._CallList'> [call.do_commit(),
 call.results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
 call.do_commit(),
 call.results.put(<MagicMock name='TransactionQueue().get_
9059	<class 'mock.mock._CallList'> [call.do_commit(),
 call.results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
 call.do_commit(),
 call.results.put(<MagicMock name='TransactionQueue().get_
9059	<class 'mock.mock._CallList'> [call.get_nowait(),
 call.task_done(),
 call.get_nowait(),
 call.task_done(),
 call.get_nowait(),
 call.task_done(),
 call.get_nowait(),
 call.task_done(),
 call.get_nowait(),
 call.task_done(),
 call
8997	<class 'mock.mock._CallList'> [call(<Mock id='139952686333712'>, <Mock name='get_schema_helper()' id='139952686333840'>),
 call().wait(<MagicMock name='Poller()' id='139952689107984'>),
 call().run(),
 call().wait(<MagicMock name=

types:
460418	 <class 'mock.mock._Call'>
79091	 <type 'tuple'>
47269	 <type 'function'>
45542	 <type 'dict'>
30758	 <type 'list'>
14696	 <type 'weakref'>
8601	 <type 'set'>
6579	 <type 'cell'>
5639	 <type 'type'>
4940	 <class 'mock.mock.MagicProxy'>
3858	 <type 'getset_descriptor'>
3291	 <class 'unittest.suite.TestSuite'>
3275	 <type 'itertools.count'>
3267	 <class 'unittest2.case._TypeEqualityDict'>
2439	 <type 'module'>
2304	 <class '_weakrefset.WeakSet'>
2222	 <type 'instancemethod'>
2219	 <type 'builtin_function_or_method'>
1869	 <type 'property'>
1424	 <type 'wrapper_descriptor'>

** Affects: neutron
     Importance: High
     Assignee: Oleg Bondarev (obondarev)
         Status: In Progress


** Tags: unittest

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1595878

Title:
  Memory leak in unit tests

Status in neutron:
  In Progress

Bug description:
  tests.unit.agent.ovsdb.native.test_connection.TestOVSNativeConnection
  calls Connection.start() which starts a daemon with a while True loop
  full of mocks. mock._CallList of those mocks start to grow very quick
  and finally eat all available memory.

  mem_top output during unit tests run:

  refs:
  18118	<class 'mock.mock._CallList'> [call(1),
   call().get_nowait(),
   call().get_nowait().do_commit(),
   call().get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
   call().task_do
  18117	<class 'mock.mock._CallList'> [call.get_nowait(),
   call.get_nowait().do_commit(),
   call.get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
   call.task_done(),
   call.get_no
  17990	<class 'mock.mock._CallList'> [call(1),
   call().get_nowait(),
   call().get_nowait().do_commit(),
   call().get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
   call().task_do
  17989	<class 'mock.mock._CallList'> [call.get_nowait(),
   call.get_nowait().do_commit(),
   call.get_nowait().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
   call.task_done(),
   call.get_no
  13592	<class 'mock.mock._CallList'> [call(),
   call().fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
   call().timer_wait(<Mock id='139952614993680'>),
   call().block(),
   call().fd_wait(<MagicMock name=
  13591	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
   call.timer_wait(<Mock id='139952614993680'>),
   call.block(),
   call.fd_wait(<MagicMock name='TransactionQueue
  13591	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952638377296'>, 1),
   call.timer_wait(<Mock id='139952614993680'>),
   call.block(),
   call.fd_wait(<MagicMock name='TransactionQueue
  13588	<class 'mock.mock._CallList'> [call(),
   call().do_commit(),
   call().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
   call(),
   call().do_commit(),
   call().results.put(<MagicMock nam
  13496	<class 'mock.mock._CallList'> [call(),
   call().fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
   call().timer_wait(<Mock id='139952687478800'>),
   call().block(),
   call().fd_wait(<MagicMock name=
  13495	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
   call.timer_wait(<Mock id='139952687478800'>),
   call.block(),
   call.fd_wait(<MagicMock name='TransactionQueue
  13495	<class 'mock.mock._CallList'> [call.fd_wait(<MagicMock name='TransactionQueue().alert_fileno' id='139952689448208'>, 1),
   call.timer_wait(<Mock id='139952687478800'>),
   call.block(),
   call.fd_wait(<MagicMock name='TransactionQueue
  13492	<class 'mock.mock._CallList'> [call(),
   call().do_commit(),
   call().results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952663067600'>),
   call(),
   call().do_commit(),
   call().results.put(<MagicMock nam
  11550	<type 'dict'> {'keystoneclient.service_catalog': <module 'keystoneclient.service_catalog' from '/opt/stack/neutron/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/service_catalog.pyc'>, 'oslo_messaging.r
  9061	<class 'mock.mock._CallList'> [call(<Mock id='139952630613712'>, <Mock name='get_schema_helper()' id='139952630616016'>),
   call().wait(<MagicMock name='Poller()' id='139952641693968'>),
   call().run(),
   call().wait(<MagicMock name=
  9060	<class 'mock.mock._CallList'> [call.wait(<MagicMock name='Poller()' id='139952641693968'>),
   call.run(),
   call.wait(<MagicMock name='Poller()' id='139952641693968'>),
   call.run(),
   call.wait(<MagicMock name='Poller()' id='13995264
  9060	<class 'mock.mock._CallList'> [call.wait(<MagicMock name='Poller()' id='139952641693968'>),
   call.run(),
   call.wait(<MagicMock name='Poller()' id='139952641693968'>),
   call.run(),
   call.wait(<MagicMock name='Poller()' id='13995264
  9059	<class 'mock.mock._CallList'> [call.do_commit(),
   call.results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
   call.do_commit(),
   call.results.put(<MagicMock name='TransactionQueue().get_
  9059	<class 'mock.mock._CallList'> [call.do_commit(),
   call.results.put(<MagicMock name='TransactionQueue().get_nowait().do_commit()' id='139952637975312'>),
   call.do_commit(),
   call.results.put(<MagicMock name='TransactionQueue().get_
  9059	<class 'mock.mock._CallList'> [call.get_nowait(),
   call.task_done(),
   call.get_nowait(),
   call.task_done(),
   call.get_nowait(),
   call.task_done(),
   call.get_nowait(),
   call.task_done(),
   call.get_nowait(),
   call.task_done(),
   call
  8997	<class 'mock.mock._CallList'> [call(<Mock id='139952686333712'>, <Mock name='get_schema_helper()' id='139952686333840'>),
   call().wait(<MagicMock name='Poller()' id='139952689107984'>),
   call().run(),
   call().wait(<MagicMock name=

  types:
  460418	 <class 'mock.mock._Call'>
  79091	 <type 'tuple'>
  47269	 <type 'function'>
  45542	 <type 'dict'>
  30758	 <type 'list'>
  14696	 <type 'weakref'>
  8601	 <type 'set'>
  6579	 <type 'cell'>
  5639	 <type 'type'>
  4940	 <class 'mock.mock.MagicProxy'>
  3858	 <type 'getset_descriptor'>
  3291	 <class 'unittest.suite.TestSuite'>
  3275	 <type 'itertools.count'>
  3267	 <class 'unittest2.case._TypeEqualityDict'>
  2439	 <type 'module'>
  2304	 <class '_weakrefset.WeakSet'>
  2222	 <type 'instancemethod'>
  2219	 <type 'builtin_function_or_method'>
  1869	 <type 'property'>
  1424	 <type 'wrapper_descriptor'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1595878/+subscriptions
Follow ups

[Bug 1595878] Re: Memory leak in unit tests
From: OpenStack Infra, 2016-06-29