yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1873240] [NEW] Missing rootwrap filter rule for radvd kill script

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1873240@xxxxxxxxxxxxxxxxxx>
Date: Thu, 16 Apr 2020 10:21:07 -0000
Reply-to: Bug 1873240 <1873240@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In commit https://opendev.org/openstack/neutron/commit/93015527f02cb54b0b2bc07c8c9c239c63878264 we added support for custom kill scripts and we added rootwrap filter rules for scripts to kill dnsmasq, haproxy, keepalived but we missed same rule for radvd.
Now if such radvd-kill script is used, it fails in Neutron-L3-agent with error like:

2020-04-13 11:07:43.064 31530 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['radvd-kill', '9', '43476']: multiprocessing.managers.RemoteError: 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent [-] Error while deleting router f5735a8c-ede9-47a4-b2c8-2c93fc50ac07: multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
    send(msg)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
    s = self.dumps(obj)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
    return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
    return super(RpcJSONEncoder, self).default(o)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ValueError' is not JSON serializable

---------------------------------------------------------------------------
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 506, in _safe_router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self._router_removed(ri, router_id)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 542, in _router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.router_info[router_id] = ri
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 539, in _router_removed
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     ri.delete()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 470, in delete
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.disable_radvd()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 587, in disable_radvd
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.radvd.disable()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ra.py", line 195, in disable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     pm.disable()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/external_process.py", line 113, in disable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=self.run_as_root)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 122, in execute
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     LOG.error("Rootwrap error running command: %s", cmd)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in execute
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = self._run_one_command(proxy, cmd, stdin)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 139, in _run_one_command
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent multiprocessing.managers.RemoteError: 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Unserializable message: Traceback (most recent call last):
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     send(msg)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     s = self.dumps(obj)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     **kw).encode(obj)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     chunks = self.iterencode(o, _one_shot=True)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return _iterencode(o, 0)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return super(RpcJSONEncoder, self).default(o)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     o.__class__.__name__)
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent TypeError: Object of type 'ValueError' is not JSON serializable
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent 
2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------

** Affects: neutron
     Importance: Low
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1873240

Title:
  Missing rootwrap filter rule for radvd kill script

Status in neutron:
  Confirmed

Bug description:
  In commit https://opendev.org/openstack/neutron/commit/93015527f02cb54b0b2bc07c8c9c239c63878264 we added support for custom kill scripts and we added rootwrap filter rules for scripts to kill dnsmasq, haproxy, keepalived but we missed same rule for radvd.
  Now if such radvd-kill script is used, it fails in Neutron-L3-agent with error like:

  2020-04-13 11:07:43.064 31530 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['radvd-kill', '9', '43476']: multiprocessing.managers.RemoteError: 
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent [-] Error while deleting router f5735a8c-ede9-47a4-b2c8-2c93fc50ac07: multiprocessing.managers.RemoteError: 
  ---------------------------------------------------------------------------
  Unserializable message: Traceback (most recent call last):
    File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
      send(msg)
    File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
      s = self.dumps(obj)
    File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
      return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
    File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
      **kw).encode(obj)
    File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
      return _iterencode(o, 0)
    File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
      return super(RpcJSONEncoder, self).default(o)
    File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
      o.__class__.__name__)
  TypeError: Object of type 'ValueError' is not JSON serializable

  ---------------------------------------------------------------------------
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 506, in _safe_router_removed
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self._router_removed(ri, router_id)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 542, in _router_removed
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.router_info[router_id] = ri
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 539, in _router_removed
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     ri.delete()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 470, in delete
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.disable_radvd()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 587, in disable_radvd
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.radvd.disable()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ra.py", line 195, in disable
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     pm.disable()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/external_process.py", line 113, in disable
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=self.run_as_root)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 122, in execute
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     LOG.error("Rootwrap error running command: %s", cmd)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise value
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in execute
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = self._run_one_command(proxy, cmd, stdin)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 139, in _run_one_command
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent multiprocessing.managers.RemoteError: 
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent Unserializable message: Traceback (most recent call last):
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     send(msg)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     s = self.dumps(obj)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     **kw).encode(obj)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     chunks = self.iterencode(o, _one_shot=True)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return _iterencode(o, 0)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     return super(RpcJSONEncoder, self).default(o)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent     o.__class__.__name__)
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent TypeError: Object of type 'ValueError' is not JSON serializable
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent 
  2020-04-13 11:07:43.065 31530 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1873240/+subscriptions
Follow ups

[Bug 1873240] Re: Missing rootwrap filter rule for radvd kill script
From: OpenStack Infra, 2020-04-18