yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1665061] [NEW] keepalived sanity check tries to disable process monitoring, but doesn't

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Terry Wilson <1665061@xxxxxxxxxxxxxxxxxx>
Date: Wed, 15 Feb 2017 17:58:36 -0000
Reply-to: Bug 1665061 <1665061@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

KeepalivedIPv6Test.start_keepalived_process overrides
AGENT.check_child_processes_interval to 0 in an attempt to disable
process monitoring. The default value is 60 and the decision on whether
to spawn the process monitoring thread is made in
ProcessMonitor.__init__(), which is called by
KeepalivedIPv6Test.__init__() before start_keepalived_process is called,
so the process monitor is already started. Example logs:

2017-02-15 11:20:28.913 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid get_value_from_file neutron/agent/linux/utils.py:274
2017-02-15 11:20:28.914 16433 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid router1 not found. The process should not have died
2017-02-15 11:20:28.914 16433 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid router1
2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid get_value_from_file neutron/agent/linux/utils.py:274
2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid-vrrp get_value_from_file neutron/agent/linux/utils.py:274
2017-02-15 11:20:28.915 16433 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'keepalivedtest-31b1e0ad-37df-46c9-9aee-ed2cbfc117bf', 'keepalived', '-P', '-f', '/tmp/tmppDICas/tmpJpJ77T/router1/keepalived.conf', '-p', '/tmp/tmppDICas/tmpJpJ77T/router1.pid', '-r', '/tmp/tmppDICas/tmpJpJ77T/router1.pid-vrrp'] execute_rootwrap_daemon neutron/agent/linux/utils.py:113
2017-02-15 11:20:28.998 16433 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute neutron/agent/linux/utils.py:154
2017-02-15 11:20:28.998 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.085s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1665061

Title:
  keepalived sanity check tries to disable process monitoring, but
  doesn't

Status in neutron:
  New

Bug description:
  KeepalivedIPv6Test.start_keepalived_process overrides
  AGENT.check_child_processes_interval to 0 in an attempt to disable
  process monitoring. The default value is 60 and the decision on
  whether to spawn the process monitoring thread is made in
  ProcessMonitor.__init__(), which is called by
  KeepalivedIPv6Test.__init__() before start_keepalived_process is
  called, so the process monitor is already started. Example logs:

  2017-02-15 11:20:28.913 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
  2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid get_value_from_file neutron/agent/linux/utils.py:274
  2017-02-15 11:20:28.914 16433 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid router1 not found. The process should not have died
  2017-02-15 11:20:28.914 16433 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid router1
  2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid get_value_from_file neutron/agent/linux/utils.py:274
  2017-02-15 11:20:28.914 16433 DEBUG neutron.agent.linux.utils [-] Unable to access /tmp/tmppDICas/tmpJpJ77T/router1.pid-vrrp get_value_from_file neutron/agent/linux/utils.py:274
  2017-02-15 11:20:28.915 16433 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'keepalivedtest-31b1e0ad-37df-46c9-9aee-ed2cbfc117bf', 'keepalived', '-P', '-f', '/tmp/tmppDICas/tmpJpJ77T/router1/keepalived.conf', '-p', '/tmp/tmppDICas/tmpJpJ77T/router1.pid', '-r', '/tmp/tmppDICas/tmpJpJ77T/router1.pid-vrrp'] execute_rootwrap_daemon neutron/agent/linux/utils.py:113
  2017-02-15 11:20:28.998 16433 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute neutron/agent/linux/utils.py:154
  2017-02-15 11:20:28.998 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.085s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
  2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2017-02-15 11:20:28.999 16433 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /home/terry/src/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1665061/+subscriptions
Follow ups

[Bug 1665061] Re: keepalived sanity check tries to disable process monitoring, but doesn't
From: OpenStack Infra, 2017-03-08