← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #77829

[Bug 1823038] [NEW] Neutron-keepalived-state-change fails to check initial router state

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

As fix for bug https://bugs.launchpad.net/neutron/+bug/1818614 we added
to neutron-keepalived-state-change monitor possibility to check initial
status of router (master or slave).

Unfortunately for some reason I see now in journal log of functional job
that this check is failing with error like:

Apr 03 09:19:09 ubuntu-bionic-ovh-gra1-0004666718 neutron-keepalived-state-change[1553]: 2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change [-] Failed to get initial status of router cd300e6b-8222-4100-8f6a-3b5c4d5fe37b: FailedToDropPrivileges: privsep helper command exited non-zero (96)
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change Traceback (most recent call last):
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/l3/keepalived_state_change.py", line 98, in handle_initial_state
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     for address in ip.addr.list():
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 540, in list
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     **kwargs)
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 1412, in get_devices_with_ip
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     devices = privileged.get_link_devices(namespace, **link_args)
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 240, in _wrap
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     self.start()
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 251, in start
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     channel = daemon.RootwrapClientChannel(context=self)
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 328, in __init__
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     raise FailedToDropPrivileges(msg)
                                                                                         2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change FailedToDropPrivileges: privsep helper command exited non-zero (96)


Example of such error: http://logs.openstack.org/25/645225/8/check/neutron-functional-python27/0704654/controller/logs/journal_log.txt.gz#_Apr_03_09_19_09

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: gate-failure l3-dvr-backlog

** Changed in: neutron
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1823038

Title:
  Neutron-keepalived-state-change fails to check initial router state

Status in neutron:
  Confirmed

Bug description:
  As fix for bug https://bugs.launchpad.net/neutron/+bug/1818614 we
  added to neutron-keepalived-state-change monitor possibility to check
  initial status of router (master or slave).

  Unfortunately for some reason I see now in journal log of functional
  job that this check is failing with error like:

  Apr 03 09:19:09 ubuntu-bionic-ovh-gra1-0004666718 neutron-keepalived-state-change[1553]: 2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change [-] Failed to get initial status of router cd300e6b-8222-4100-8f6a-3b5c4d5fe37b: FailedToDropPrivileges: privsep helper command exited non-zero (96)
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change Traceback (most recent call last):
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/l3/keepalived_state_change.py", line 98, in handle_initial_state
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     for address in ip.addr.list():
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 540, in list
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     **kwargs)
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 1412, in get_devices_with_ip
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     devices = privileged.get_link_devices(namespace, **link_args)
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 240, in _wrap
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     self.start()
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 251, in start
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     channel = daemon.RootwrapClientChannel(context=self)
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change   File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 328, in __init__
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change     raise FailedToDropPrivileges(msg)
                                                                                           2019-04-03 09:19:09.778 1553 ERROR neutron.agent.l3.keepalived_state_change FailedToDropPrivileges: privsep helper command exited non-zero (96)

  
  Example of such error: http://logs.openstack.org/25/645225/8/check/neutron-functional-python27/0704654/controller/logs/journal_log.txt.gz#_Apr_03_09_19_09

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1823038/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next