yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1786213] [NEW] Metering agent: failed to running ip netns commad

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dongcan Ye <yedongcan@xxxxxxxx>
Date: Thu, 09 Aug 2018 09:58:46 -0000
Reply-to: Bug 1786213 <1786213@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

While neutron-metering-agent start, it fails as following:
2018-08-09 03:15:08.504 10637 INFO oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/metering_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-metering-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp_JYWYs/privsep.sock']
2018-08-09 03:15:08.525 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.526 10637 WARNING oslo.privsep.daemon [-] privsep log: We trust you have received the usual lecture from the local System
2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log: Administrator. It usually boils down to these three things:
2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #1) Respect the privacy of others.
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #2) Think before you type.
2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #3) With great power comes great responsibility.
2018-08-09 03:15:08.529 10637 WARNING oslo.privsep.daemon [-] privsep log:
2018-08-09 03:15:08.531 10637 WARNING oslo.privsep.daemon [-] privsep log: sudo: no tty present and no askpass program specified
2018-08-09 03:15:08.544 10637 CRITICAL oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] privsep helper command exited non-zero (1)

2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Error during MeteringAgentWithStateReport._sync_routers_task: FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     task(self, context)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 189, in _sync_routers_task
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._update_routers(context, routers)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 212, in _update_routers
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     'update_routers')
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 166, in _invoke_driver
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return getattr(self.metering_driver, func_name)(context, meterings)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return method(*args, **kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 151, in update_routers
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._process_associate_metering_label(router)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 249, in _process_associate_metering_label
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._update_router(router)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 118, in _update_router
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     RouterWithMetering(self.conf, router))
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 94, in __init__
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     if ip_lib.network_namespace_exists(self.ns_name):
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1057, in network_namespace_exists
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     output = list_network_namespaces(**kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1046, in list_network_namespaces
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return privileged.list_netns(**kwargs)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self.start()
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 217, in start
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     channel = daemon.RootwrapClientChannel(context=self)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     raise FailedToDropPrivileges(msg)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task

It seems missing setup_privsep() call in metering agent init.

** Affects: neutron
     Importance: Undecided
     Assignee: Dongcan Ye (hellochosen)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Dongcan Ye (hellochosen)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1786213

Title:
  Metering agent: failed to running ip netns commad

Status in neutron:
  New

Bug description:
  While neutron-metering-agent start, it fails as following:
  2018-08-09 03:15:08.504 10637 INFO oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/metering_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-metering-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp_JYWYs/privsep.sock']
  2018-08-09 03:15:08.525 10637 WARNING oslo.privsep.daemon [-] privsep log:
  2018-08-09 03:15:08.526 10637 WARNING oslo.privsep.daemon [-] privsep log: We trust you have received the usual lecture from the local System
  2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log: Administrator. It usually boils down to these three things:
  2018-08-09 03:15:08.527 10637 WARNING oslo.privsep.daemon [-] privsep log:
  2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #1) Respect the privacy of others.
  2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #2) Think before you type.
  2018-08-09 03:15:08.528 10637 WARNING oslo.privsep.daemon [-] privsep log:     #3) With great power comes great responsibility.
  2018-08-09 03:15:08.529 10637 WARNING oslo.privsep.daemon [-] privsep log:
  2018-08-09 03:15:08.531 10637 WARNING oslo.privsep.daemon [-] privsep log: sudo: no tty present and no askpass program specified
  2018-08-09 03:15:08.544 10637 CRITICAL oslo.privsep.daemon [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] privsep helper command exited non-zero (1)

  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task [req-6dd24e96-e82b-49a5-9d33-80e1ff572502 - - - - -] Error during MeteringAgentWithStateReport._sync_routers_task: FailedToDropPrivileges: privsep helper command exited non-zero (1)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     task(self, context)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 189, in _sync_routers_task
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._update_routers(context, routers)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 212, in _update_routers
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     'update_routers')
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/agents/metering_agent.py", line 166, in _invoke_driver
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return getattr(self.metering_driver, func_name)(context, meterings)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return method(*args, **kwargs)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 151, in update_routers
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._process_associate_metering_label(router)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 249, in _process_associate_metering_label
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self._update_router(router)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 118, in _update_router
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     RouterWithMetering(self.conf, router))
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/services/metering/drivers/iptables/iptables_driver.py", line 94, in __init__
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     if ip_lib.network_namespace_exists(self.ns_name):
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1057, in network_namespace_exists
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     output = list_network_namespaces(**kwargs)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1046, in list_network_namespaces
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     return privileged.list_netns(**kwargs)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 206, in _wrap
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     self.start()
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 217, in start
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     channel = daemon.RootwrapClientChannel(context=self)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 327, in __init__
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task     raise FailedToDropPrivileges(msg)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task FailedToDropPrivileges: privsep helper command exited non-zero (1)
  2018-08-09 03:15:08.546 10637 ERROR oslo_service.periodic_task

  It seems missing setup_privsep() call in metering agent init.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1786213/+subscriptions
Follow ups

[Bug 1786213] Re: Metering agent: failed to run ip netns command
From: OpenStack Infra, 2018-08-10