yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1887147] Re: neutron-linuxbridge-agent looping same as dhcp

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brian Haley <1887147@xxxxxxxxxxxxxxxxxx>
Date: Wed, 19 Aug 2020 18:58:44 -0000
Reply-to: Bug 1887147 <1887147@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1887147

Title:
  neutron-linuxbridge-agent looping same as dhcp

Status in neutron:
  Invalid

Bug description:
  I am trying to install https://docs.openstack.org/install-guide
  /openstack-services.html#minimal-deployment-for-ussuri on CentOS 8,
  with network provider option 1.

  
  For the reproduction steps i followed install-guide deployment-for-ussuri.

  Other components with trial's and errors did worked, neutron
  linuxbridge-agent keeps looping and returning errors.

  Logs from linuxbridge-agent.log in the attachment additionally dhcp
  agent log returns similar problem in logs:


  
  2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 action: enable call_driver /usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py:163
  2020-07-10 12:21:15.060 68787 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid; Error: [Errno 2] No such file or directory: '/var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5/pid' get_value_from_file /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:262
  2020-07-10 12:21:15.061 68787 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/dhcp_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-dhcp-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmphzark8uo/privsep.sock']
  2020-07-10 12:21:16.353 68787 CRITICAL oslo.privsep.daemon [-] privsep helper command exited non-zero (1)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 34dc4390-9448-4eba-8be2-a5c3f4cb94a5.: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 178, in call_driver
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 256, in enable
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     common_utils.wait_until_true(self._enable, timeout=300)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 703, in wait_until_true
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     while not predicate():
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 268, in _enable
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1652, in setup
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     ip_lib.IPWrapper().ensure_namespace(network.namespace)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 249, in ensure_namespace
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     if not self.netns.exists(name):
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 728, in exists
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     return network_namespace_exists(name)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 936, in network_namespace_exists
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     output = list_network_namespaces(**kwargs)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 922, in list_network_namespaces
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     return privileged.list_netns(**kwargs)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 246, in _wrap
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     self.start()
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 258, in start
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     channel = daemon.RootwrapClientChannel(context=self)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 331, in __init__
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent     raise FailedToDropPrivileges(msg)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
  2020-07-10 12:21:16.354 68787 ERROR neutron.agent.dhcp.agent 


  It tries to get /var/lib/neutron/dhcp/34dc4390-9448-4eba-
  8be2-a5c3f4cb94a5/pid but it's not there

  
  [root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# pwd
  /var/lib/neutron/dhcp/34dc4390-9448-4eba-8be2-a5c3f4cb94a5
  [root@34dc4390-9448-4eba-8be2-a5c3f4cb94a5]# ls -a
  .  ..

  
  nmcli con show
  NAME       UUID                                  TYPE      DEVICE 
  eno3       ad51daed-db15-4380-b559-ccda28fec65e  ethernet  eno3   
  eno4       f4d476b2-edaf-45e6-bbb5-84d97b6b5e2f  ethernet  eno4  




  
   openstack network list
  +--------------------------------------+----------+--------------------------------------+
  | ID                                   | Name     | Subnets                              |
  +--------------------------------------+----------+--------------------------------------+
  | 34dc4390-9448-4eba-8be2-a5c3f4cb94a5 | provider | 26401d1b-38c4-4887-bef3-64b746f86e96 |
  +--------------------------------------+----------+--------------------------------------+
   neutron agent-list

  
  neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
  +--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
  | id                                   | agent_type     | host                        | availability_zone | alive | admin_state_up | binary                 |
  +--------------------------------------+----------------+-----------------------------+-------------------+-------+----------------+------------------------+
  | 1ff1b3be-555c-4ef2-a004-b1299f2a8d3d | DHCP agent     | X | nova              | :-)   | True           | neutron-dhcp-agent     |
  | ff6f0baf-e507-4df2-8cc3-a3eb4bc064e0 | Metadata agent | X |                   | :-)   | True           | neutron-metadata-agent |
  +--------------------------------------+----------------+----


  eno4 is interface used for provider it has'nt any ip address attached.

  sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1
  sysctl net.bridge.bridge-nf-call-ip6tables net.bridge.bridge-nf-call-ip6tables = 1

  
  journalctl -u neutron-linuxbridge-agent.service -xe --no-pager


  Jul 10 12:31:02 X sudo[87036]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmpj1_5u88q/privsep.sock
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 CRITICAL privsep [-] Unhandled error: PermissionError: [Errno 13] Permission denied
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep Traceback (most recent call last):
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep   File "/bin/privsep-helper", line 10, in <module>
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep     sys.exit(helper_main())
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 536, in helper_main
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep     sock.connect(cfg.CONF.privsep_sock_path)
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep PermissionError: [Errno 13] Permission denied
  Jul 10 12:31:03 X neutron-linuxbridge-agent[87013]: 2020-07-10 12:31:03.535 87040 ERROR privsep 
  Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Main process exited, code=exited, status=1/FAILURE
  Jul 10 12:31:03 X systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.
  Jul 10 12:31:04 X systemd[1]: neutron-linuxbridge-agent.service: Service RestartSec=100ms expired, scheduling restart

  
  cat /etc/sudoers.d/neutron 
  Defaults:neutron !requiretty

  neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
  neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
  neutron ALL = (root) NOPASSWD: ALL


  In sudoers file there is directive #includedir /etc/sudoers.d

  OpenStack version: Ussuri
  Linux distro: CentOS 8

  I am trying to install this controller/compute node on the same bare
  metal machine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1887147/+subscriptions
References

[Bug 1887147] [NEW] neutron-linuxbridge-agent looping same as dhcp
From: Rafał Radziejewski, 2020-07-10