← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #88496

[Bug 1965140] [NEW] Eventlet fails when starting network agents

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I have a two nodes openstack setup, where one node (w3) runs all
controller services (keystone, glance, placement, nova, neutron,
horizon, cinder) as well as nova-compute and cinder-volume and the
second (w6) runs nova-compute and linuxbridge agent.

All network agents on w3 are dead
[root@w3 ~]# openstack network agent list
+--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host          | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+
| 330269b7-b73c-4207-abc7-21f1a2972b7b | Linux bridge agent | w6.int.lunarc | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 83d16241-8a3a-42b0-beda-87246d945dc1 | L3 agent           | w3.int.lunarc | nova              | XXX   | UP    | neutron-l3-agent          |
| a52ab60f-d893-491d-a43e-823a0d482810 | Linux bridge agent | w3.int.lunarc | None              | XXX   | UP    | neutron-linuxbridge-agent |
| abd75644-d895-41ae-94fa-6c4351cbc4bf | Metadata agent     | w3.int.lunarc | None              | XXX   | UP    | neutron-metadata-agent    |
| c05c65bc-779e-4fe5-a19e-350c44900be4 | DHCP agent         | w3.int.lunarc | nova              | XXX   | UP    | neutron-dhcp-agent        |
+--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+

, and I cannot start them anymore. I tried restarting said agent alone,
restarting all openstack daemon on w3 and even restarting the whole node
but nothing seems to help and I always have teh same issue and the same
trace as show below.

I could not find any useful info in the logs, but systemd does report an issue with eventlet/greenlet:
[root@w3 ~]# journalctl -fu neutron-linuxbridge-agent
-- Logs begin at Wed 2022-03-16 04:32:31 EDT. --
Mar 16 09:14:09 w3.int.lunarc sudo[37085]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmp93tzwqg3/privsep.sock
Mar 16 09:14:12 w3.int.lunarc sudo[37107]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.link_cmd --privsep_sock_path /tmp/tmp81iy5eni/privsep.sock
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]: Traceback (most recent call last):
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     timer()
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     cb(*args, **kw)
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     waiter.switch()
Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]: greenlet.error: cannot switch to a different thread

I am running OpenStack Xena on CentOS Stream 8 freshly installed. Here are other details:
[root@w3 ~]# uname -a
Linux w3.int.lunarc 4.18.0-365.el8.x86_64 #1 SMP Thu Feb 10 16:11:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Any clue on how I can find out what makes this happen, or just how I can
get past this crippling greenlet/eventlet error, and get these agents to
run again?

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "journalctl -fu neutron-linuxbridge-agent"
   https://bugs.launchpad.net/bugs/1965140/+attachment/5569663/+files/journalctl

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1965140

Title:
  Eventlet fails when starting network agents

Status in neutron:
  New

Bug description:
  I have a two nodes openstack setup, where one node (w3) runs all
  controller services (keystone, glance, placement, nova, neutron,
  horizon, cinder) as well as nova-compute and cinder-volume and the
  second (w6) runs nova-compute and linuxbridge agent.

  All network agents on w3 are dead
  [root@w3 ~]# openstack network agent list
  +--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host          | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+
  | 330269b7-b73c-4207-abc7-21f1a2972b7b | Linux bridge agent | w6.int.lunarc | None              | :-)   | UP    | neutron-linuxbridge-agent |
  | 83d16241-8a3a-42b0-beda-87246d945dc1 | L3 agent           | w3.int.lunarc | nova              | XXX   | UP    | neutron-l3-agent          |
  | a52ab60f-d893-491d-a43e-823a0d482810 | Linux bridge agent | w3.int.lunarc | None              | XXX   | UP    | neutron-linuxbridge-agent |
  | abd75644-d895-41ae-94fa-6c4351cbc4bf | Metadata agent     | w3.int.lunarc | None              | XXX   | UP    | neutron-metadata-agent    |
  | c05c65bc-779e-4fe5-a19e-350c44900be4 | DHCP agent         | w3.int.lunarc | nova              | XXX   | UP    | neutron-dhcp-agent        |
  +--------------------------------------+--------------------+---------------+-------------------+-------+-------+---------------------------+

  , and I cannot start them anymore. I tried restarting said agent
  alone, restarting all openstack daemon on w3 and even restarting the
  whole node but nothing seems to help and I always have teh same issue
  and the same trace as show below.

  I could not find any useful info in the logs, but systemd does report an issue with eventlet/greenlet:
  [root@w3 ~]# journalctl -fu neutron-linuxbridge-agent
  -- Logs begin at Wed 2022-03-16 04:32:31 EDT. --
  Mar 16 09:14:09 w3.int.lunarc sudo[37085]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmp93tzwqg3/privsep.sock
  Mar 16 09:14:12 w3.int.lunarc sudo[37107]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --privsep_context neutron.privileged.link_cmd --privsep_sock_path /tmp/tmp81iy5eni/privsep.sock
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]: Traceback (most recent call last):
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     timer()
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     cb(*args, **kw)
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:   File "/usr/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]:     waiter.switch()
  Mar 16 09:14:15 w3.int.lunarc neutron-linuxbridge-agent[37073]: greenlet.error: cannot switch to a different thread

  I am running OpenStack Xena on CentOS Stream 8 freshly installed. Here are other details:
  [root@w3 ~]# uname -a
  Linux w3.int.lunarc 4.18.0-365.el8.x86_64 #1 SMP Thu Feb 10 16:11:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

  Any clue on how I can find out what makes this happen, or just how I
  can get past this crippling greenlet/eventlet error, and get these
  agents to run again?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1965140/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next