yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2118397] [NEW] Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with libvirtd

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Giuseppe Petralia <2118397@xxxxxxxxxxxxxxxxxx>
Date: Thu, 24 Jul 2025 10:25:45 -0000
Reply-to: Bug 2118397 <2118397@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

This is happening on Nova 3:25.2.1 (Yoga)


We have some nodes running 300+ VMs, when restarting nova-compute we saw on journal following errors 


Jul 21 08:50:44 ra2-n5 dbus-daemon[37971]: [system] The maximum number of pending replies for ":1.131331" (uid=0 pid=2649667 comm="/usr/sbin/libvirtd --timeout 120 " label="libvirtd (enforce)") has been reached (max_replies_per_connection=128)

Jul 21 08:50:44 ra2-n5 libvirtd[2649667]: error from service:
GDBus.Error:org.freedesktop.DBus.Error.LimitsExceeded: The maximum
number of pending replies per connection has been reached


At the same time nova-compute which is configured with resume_guests_state_on_host_boot = True
started rebooting half of VMs running on the node logging the following 


2025-07-21 09:07:33.912 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Checking state _get_power_state /usr/lib/python3/dist-packages/nova/compute/manager.py:1674
 
2025-07-21 09:07:33.914 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Current state is 4, state in DB is 1. _init_instance /usr/lib/python3/dist-packages/nova/compute/manager.py:1302
 
2025-07-21 09:07:33.914 2665574 INFO nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Rebooting instance after nova-compute restart.
 
2025-07-21 09:07:33.946 2665574 INFO nova.virt.libvirt.driver [-] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Instance destroyed successfully.  


A possible workaround is to tune that dbus parameter controlling the
max_replies_per_connection with

# /etc/dbus-1/system-local.conf
<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd";>
<busconfig>
  <limit name="max_replies_per_connection">1000</limit>
</busconfig>
 
# restart dbus
sudo systemctl restart dbus

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with livirtd
+ Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with libvirtd

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2118397

Title:
  Nova restart Virtual machines on service restart if hitting
  max_replies_per_connection limit with libvirtd

Status in OpenStack Compute (nova):
  New

Bug description:
  This is happening on Nova 3:25.2.1 (Yoga)

  
  We have some nodes running 300+ VMs, when restarting nova-compute we saw on journal following errors 

  
  Jul 21 08:50:44 ra2-n5 dbus-daemon[37971]: [system] The maximum number of pending replies for ":1.131331" (uid=0 pid=2649667 comm="/usr/sbin/libvirtd --timeout 120 " label="libvirtd (enforce)") has been reached (max_replies_per_connection=128)

  Jul 21 08:50:44 ra2-n5 libvirtd[2649667]: error from service:
  GDBus.Error:org.freedesktop.DBus.Error.LimitsExceeded: The maximum
  number of pending replies per connection has been reached

  
  At the same time nova-compute which is configured with resume_guests_state_on_host_boot = True
  started rebooting half of VMs running on the node logging the following 

  
  2025-07-21 09:07:33.912 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Checking state _get_power_state /usr/lib/python3/dist-packages/nova/compute/manager.py:1674
   
  2025-07-21 09:07:33.914 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Current state is 4, state in DB is 1. _init_instance /usr/lib/python3/dist-packages/nova/compute/manager.py:1302
   
  2025-07-21 09:07:33.914 2665574 INFO nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Rebooting instance after nova-compute restart.
   
  2025-07-21 09:07:33.946 2665574 INFO nova.virt.libvirt.driver [-] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Instance destroyed successfully.  


  
  A possible workaround is to tune that dbus parameter controlling the max_replies_per_connection with

  # /etc/dbus-1/system-local.conf
  <!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
   "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd";>
  <busconfig>
    <limit name="max_replies_per_connection">1000</limit>
  </busconfig>
   
  # restart dbus
  sudo systemctl restart dbus

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2118397/+subscriptions