yahoo-eng-team team mailing list archive

[Uggla <2118397@xxxxxxxxxxxxxxxxxx>]

Thank you for submitting this bug report.

Please note that Nova Yoga is not supported anymore.

Based on the information provided, this issue appears to be a
configuration or environment problem rather than a bug in the Nova
software itself.

For these reasons, we are marking this bug as 'Invalid'. If you still
believe this is a Nova bug and you can reproduce it on a supported
OpenStack version, please feel free to update this report with the
necessary details (referencing our bug reporting template:
https://wiki.openstack.org/wiki/Nova/BugsTeam/BugReportTemplate) and set
its status back to 'New'.


** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2118397

Title:
  Nova restart Virtual machines on service restart if hitting
  max_replies_per_connection limit with libvirtd

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  This is happening on Nova 3:25.2.1 (Yoga)

  
  We have some nodes running 300+ VMs, when restarting nova-compute we saw on journal following errors 

  
  Jul 21 08:50:44 ra2-n5 dbus-daemon[37971]: [system] The maximum number of pending replies for ":1.131331" (uid=0 pid=2649667 comm="/usr/sbin/libvirtd --timeout 120 " label="libvirtd (enforce)") has been reached (max_replies_per_connection=128)

  Jul 21 08:50:44 ra2-n5 libvirtd[2649667]: error from service:
  GDBus.Error:org.freedesktop.DBus.Error.LimitsExceeded: The maximum
  number of pending replies per connection has been reached

  
  At the same time nova-compute which is configured with resume_guests_state_on_host_boot = True
  started rebooting half of VMs running on the node logging the following 

  
  2025-07-21 09:07:33.912 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Checking state _get_power_state /usr/lib/python3/dist-packages/nova/compute/manager.py:1674
   
  2025-07-21 09:07:33.914 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Current state is 4, state in DB is 1. _init_instance /usr/lib/python3/dist-packages/nova/compute/manager.py:1302
   
  2025-07-21 09:07:33.914 2665574 INFO nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Rebooting instance after nova-compute restart.
   
  2025-07-21 09:07:33.946 2665574 INFO nova.virt.libvirt.driver [-] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Instance destroyed successfully.  


  
  A possible workaround is to tune that dbus parameter controlling the max_replies_per_connection with

  # /etc/dbus-1/system-local.conf
  <!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
   "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd";>
  <busconfig>
    <limit name="max_replies_per_connection">1000</limit>
  </busconfig>
   
  # restart dbus
  sudo systemctl restart dbus

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2118397/+subscriptions