yahoo-eng-team team mailing list archive

[Geoffrey <2114724@xxxxxxxxxxxxxxxxxx>]

Public bug reported:


.. bug report for OpenStack Nova: RLock not greened error

Title:
nova-manage fails with AssertionError: 3 RLock(s) were not greened when installed via packages

Bug Description:
Running nova-manage api_db sync on a fresh Ubuntu-based OpenStack installation causes a crash:

AssertionError: 3 RLock(s) were not greened

This is due to eventlet.monkey_patch() not being called early enough in
the process, causing threading.RLock to be loaded before patching, which
breaks compatibility.

Environment:

OS: Ubuntu 22.04 / 24.04

Python: 3.10+ (tested on 3.10.12)

Eventlet: 0.33.3+

Nova: 2025.1 (Bobcat)

Deployment: Official install guide for controller node

Steps to Reproduce:

Install Nova via apt per official install doc:

sudo apt install nova-api nova-conductor nova-scheduler nova-novncproxy

Run DB sync:

sudo nova-manage api_db sync

Observe error:

AssertionError: 3 RLock(s) were not greened

Root Cause:

nova-manage does not call eventlet.monkey_patch() early enough.

By the time it is called (via module imports), Python’s threading module
has already loaded RLock, and eventlet cannot green it retroactively.

Workaround:
Manually edit /usr/bin/nova-manage to add monkey patching right after the hashbang:

#!/usr/bin/python3
import eventlet
eventlet.monkey_patch()

import sys
from nova.cmd.manage import main

This ensures monkey patching is applied before any threading-related
imports.

Suggested Fix:

Modify the nova-manage entry point (in setup.cfg or wrapper script) to
import eventlet and patch before any other imports.

Ensure the patching happens at the top level of the executable.

Importance:

Affects default package-based installs using official documentation

Breaks controller bootstrapping for new operators

Error message is unclear to users unfamiliar with eventlet

References:

Related: https://github.com/eventlet/eventlet/issues/401

Eventlet monkey_patch limitations with Python 3.10+

Priority: High

Status: Workaround confirmed, patch recommended
This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes:

- [ ] This doc is inaccurate in this way: ______
- [ ] This is a doc addition request.
- [ ] I have a fix to the document that I can paste below including example: input and output. 

If you have a troubleshooting or support issue, use the following
resources:

 - The mailing list: https://lists.openstack.org
 - IRC: 'openstack' channel on OFTC

-----------------------------------
Release: 31.0.1.dev12 on 2023-02-15 22:37:40
SHA: c73aa889d38f534f965f207b23394b64f598c800
Source: https://opendev.org/openstack/nova/src/doc/source/install/controller-install-ubuntu.rst
URL: https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: doc

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2114724

Title:
  Install and configure controller node for Ubuntu in nova

Status in OpenStack Compute (nova):
  New

Bug description:

  .. bug report for OpenStack Nova: RLock not greened error

  Title:
  nova-manage fails with AssertionError: 3 RLock(s) were not greened when installed via packages

  Bug Description:
  Running nova-manage api_db sync on a fresh Ubuntu-based OpenStack installation causes a crash:

  AssertionError: 3 RLock(s) were not greened

  This is due to eventlet.monkey_patch() not being called early enough
  in the process, causing threading.RLock to be loaded before patching,
  which breaks compatibility.

  Environment:

  OS: Ubuntu 22.04 / 24.04

  Python: 3.10+ (tested on 3.10.12)

  Eventlet: 0.33.3+

  Nova: 2025.1 (Bobcat)

  Deployment: Official install guide for controller node

  Steps to Reproduce:

  Install Nova via apt per official install doc:

  sudo apt install nova-api nova-conductor nova-scheduler nova-
  novncproxy

  Run DB sync:

  sudo nova-manage api_db sync

  Observe error:

  AssertionError: 3 RLock(s) were not greened

  Root Cause:

  nova-manage does not call eventlet.monkey_patch() early enough.

  By the time it is called (via module imports), Python’s threading
  module has already loaded RLock, and eventlet cannot green it
  retroactively.

  Workaround:
  Manually edit /usr/bin/nova-manage to add monkey patching right after the hashbang:

  #!/usr/bin/python3
  import eventlet
  eventlet.monkey_patch()

  import sys
  from nova.cmd.manage import main

  This ensures monkey patching is applied before any threading-related
  imports.

  Suggested Fix:

  Modify the nova-manage entry point (in setup.cfg or wrapper script) to
  import eventlet and patch before any other imports.

  Ensure the patching happens at the top level of the executable.

  Importance:

  Affects default package-based installs using official documentation

  Breaks controller bootstrapping for new operators

  Error message is unclear to users unfamiliar with eventlet

  References:

  Related: https://github.com/eventlet/eventlet/issues/401

  Eventlet monkey_patch limitations with Python 3.10+

  Priority: High

  Status: Workaround confirmed, patch recommended
  This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes:

  - [ ] This doc is inaccurate in this way: ______
  - [ ] This is a doc addition request.
  - [ ] I have a fix to the document that I can paste below including example: input and output. 

  If you have a troubleshooting or support issue, use the following
  resources:

   - The mailing list: https://lists.openstack.org
   - IRC: 'openstack' channel on OFTC

  -----------------------------------
  Release: 31.0.1.dev12 on 2023-02-15 22:37:40
  SHA: c73aa889d38f534f965f207b23394b64f598c800
  Source: https://opendev.org/openstack/nova/src/doc/source/install/controller-install-ubuntu.rst
  URL: https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2114724/+subscriptions