yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1893627] [NEW] For neutron-l3-agent, after the execution of the linux command fails, it is not displayed which command failed to execute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: boluan ace <1893627@xxxxxxxxxxxxxxxxxx>
Date: Mon, 31 Aug 2020 09:33:45 -0000
Reply-to: Bug 1893627 <1893627@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

For neutron-l3-agent, there are many scenarios where linux commands must
be executed. When a linux command execution fails and an exception
occurs, the exception information only shows the reason for the error,
but it does not show which command was executed incorrectly, so it is
difficult to locate the cause of the error.

For example, neutron-l3-agent fails to set floating ip, the error
message is as follows:

2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info [req-23d047f9-97bb-4d37-8509-ee5658ef241f - ddcde2e097774530a2180da53c1afa23 - - -] L3 agent failure to setup floating IPs: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-c9ff34e2-20" does not exist.
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 432, in configure_fip_addresses
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self.process_floating_ip_addresses(interface_name)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 372, in process_floating_ip_addresses
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     existing_cidrs = self.get_router_cidrs(device)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 353, in get_router_cidrs
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return set([addr['cidr'] for addr in device.addr.list()])
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 679, in list
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     self.name, scope, to, filters, ip_version)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 644, in get_devices_with_ip
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     for line in self._run(options, tuple(args)).split('\n'):
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 379, in _run
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self._parent._run(options, self.COMMAND, args)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _run
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self._as_root(options, command, args)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 101, in _as_root
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     namespace=namespace)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 109, in _execute
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 149, in execute
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     returncode=returncode)
2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-c9ff34e2-20" does not exist.


But it is impossible to know which command execution error is, so it is difficult to troubleshoot the problem

Our expectation is similar to ProcessExecutionError: Exit code: 1; Cmd:
['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip',
'netns', 'exec', 'qrouter-648a4b83-b900-4b34-a01d-f9c7fd3f443d', 'ip',
'addr', 'show', 'qg-c9ff34e2-20']; Stdin: ; Stdout: ; Stderr: Device
"qg-c9ff34e2-20" does not exist.

** Affects: neutron
     Importance: Undecided
     Assignee: boluan ace (boluanace)
         Status: New

** Summary changed:

- Command detail is lost when command execution fails
+ For neutron-l3-agent, after the execution of the linux command fails, it is not displayed which command failed to execute

** Changed in: neutron
     Assignee: (unassigned) => boluan ace (boluanace)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1893627

Title:
  For neutron-l3-agent, after the execution of the linux command fails,
  it is not displayed which command failed to execute

Status in neutron:
  New

Bug description:
  For neutron-l3-agent, there are many scenarios where linux commands
  must be executed. When a linux command execution fails and an
  exception occurs, the exception information only shows the reason for
  the error, but it does not show which command was executed
  incorrectly, so it is difficult to locate the cause of the error.

  For example, neutron-l3-agent fails to set floating ip, the error
  message is as follows:

  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info [req-23d047f9-97bb-4d37-8509-ee5658ef241f - ddcde2e097774530a2180da53c1afa23 - - -] L3 agent failure to setup floating IPs: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-c9ff34e2-20" does not exist.
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 432, in configure_fip_addresses
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self.process_floating_ip_addresses(interface_name)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 372, in process_floating_ip_addresses
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     existing_cidrs = self.get_router_cidrs(device)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 353, in get_router_cidrs
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return set([addr['cidr'] for addr in device.addr.list()])
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 679, in list
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     self.name, scope, to, filters, ip_version)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 644, in get_devices_with_ip
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     for line in self._run(options, tuple(args)).split('\n'):
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 379, in _run
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self._parent._run(options, self.COMMAND, args)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _run
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     return self._as_root(options, command, args)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 101, in _as_root
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     namespace=namespace)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 109, in _execute
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 149, in execute
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info     returncode=returncode)
  2020-08-28 11:06:50.090 13 ERROR neutron.agent.l3.router_info ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Device "qg-c9ff34e2-20" does not exist.

  
  But it is impossible to know which command execution error is, so it is difficult to troubleshoot the problem

  Our expectation is similar to ProcessExecutionError: Exit code: 1;
  Cmd: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip',
  'netns', 'exec', 'qrouter-648a4b83-b900-4b34-a01d-f9c7fd3f443d', 'ip',
  'addr', 'show', 'qg-c9ff34e2-20']; Stdin: ; Stdout: ; Stderr: Device
  "qg-c9ff34e2-20" does not exist.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1893627/+subscriptions
Follow ups

[Bug 1893627] Re: For neutron-l3-agent, after the execution of the linux command fails, it is not displayed which command failed to execute
From: OpenStack Infra, 2020-09-04