yahoo-eng-team team mailing list archive

[bryan <2051598@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We are creating a new Fedora 38 instance in Zed release, and we attach a
new port to the instance when the VM status is active, but we got the
following error in libvirt:

2024-01-24 17:41:13.999+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:41:37.001+0000: 451529: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (52s, 0s, 0s)
2024-01-24 17:41:37.001+0000: 451529: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
2024-01-24 17:41:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
2024-01-24 17:41:40.251+0000: 451533: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:41:44.639+0000: 2433335: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:42:06.802+0000: 451531: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (82s, 0s, 0s)
2024-01-24 17:42:06.802+0000: 451531: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
2024-01-24 17:42:09.434+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
2024-01-24 17:42:10.859+0000: 451529: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:42:14.394+0000: 451535: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:42:36.585+0000: 2433335: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (111s, 0s, 0s)
2024-01-24 17:42:36.585+0000: 2433335: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
2024-01-24 17:42:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
2024-01-24 17:42:40.446+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:42:43.935+0000: 451531: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
2024-01-24 17:43:06.803+0000: 451533: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (142s, 0s, 0s)

It seems the instance is stuck in the attaching interface jobs, and we
also checked the nova instance-action-list output, it reports the start
time of the interface attaching event but no ends and may last for
hours, when we hard reboot the instance, the interface attaching job
gets failed immediately, seems all the blocked jobs are then released
because of the hard reboot. I also attach the
/var/log/kolla/libvirt.log.1 for more information. Thanks.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2051598

Title:
  virDomainObjBeginJobInternal:433 : Cannot start job (query, none,
  none) in API remoteDispatchDomainMemoryStats

Status in OpenStack Compute (nova):
  New

Bug description:
  We are creating a new Fedora 38 instance in Zed release, and we attach
  a new port to the instance when the VM status is active, but we got
  the following error in libvirt:

  2024-01-24 17:41:13.999+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:41:37.001+0000: 451529: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (52s, 0s, 0s)
  2024-01-24 17:41:37.001+0000: 451529: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
  2024-01-24 17:41:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
  2024-01-24 17:41:40.251+0000: 451533: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:41:44.639+0000: 2433335: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:42:06.802+0000: 451531: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (82s, 0s, 0s)
  2024-01-24 17:42:06.802+0000: 451531: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
  2024-01-24 17:42:09.434+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
  2024-01-24 17:42:10.859+0000: 451529: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:42:14.394+0000: 451535: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:42:36.585+0000: 2433335: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (111s, 0s, 0s)
  2024-01-24 17:42:36.585+0000: 2433335: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags)
  2024-01-24 17:42:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error
  2024-01-24 17:42:40.446+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:42:43.935+0000: 451531: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks
  2024-01-24 17:43:06.803+0000: 451533: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (142s, 0s, 0s)

  It seems the instance is stuck in the attaching interface jobs, and we
  also checked the nova instance-action-list output, it reports the
  start time of the interface attaching event but no ends and may last
  for hours, when we hard reboot the instance, the interface attaching
  job gets failed immediately, seems all the blocked jobs are then
  released because of the hard reboot. I also attach the
  /var/log/kolla/libvirt.log.1 for more information. Thanks.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2051598/+subscriptions