yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1979047] [NEW] Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1979047@xxxxxxxxxxxxxxxxxx>
Date: Fri, 17 Jun 2022 09:48:55 -0000
Reply-to: Bug 1979047 <1979047@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The tempest-integrated-compute-centos-9-stream job is broken since
2022-06-16 02:26:37 [1]. Multiple interface attach tempest test fails
with:

libvirt.libvirtError: internal error: unable to execute QEMU command
'netdev_add': File descriptor named '(null)' has not been found

Full exception stack trace:

[None req-6f95599e-022a-42ab-a4de-07c7b8f73daf tempest-
AttachInterfacesTestJSON-2035965943 tempest-
AttachInterfacesTestJSON-2035965943-project] [instance:
b41513e6-cb4e-4441-af14-c272393cdafb] attaching network adapter failed.:
libvirt.libvirtError: internal error: unable to execute QEMU command
'netdev_add': File descriptor named '(null)' has not been found

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb] Traceback (most recent call last):

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2850, in
attach_interface

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     guest.attach_device(cfg,
persistent=True, live=live)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 321, in attach_device

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]
self._domain.attachDeviceFlags(device_xml, flags=flags)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in
doit

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     result =
proxy_call(self._autowrap, f, *args, **kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in
proxy_call

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     rv = execute(f, *args,
**kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in
execute

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     six.reraise(c, e, tb)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     raise value

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File
"/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in
tworker

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     rv = meth(*args, **kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]   File "/usr/lib64/python3.9/site-
packages/libvirt.py", line 706, in attachDeviceFlags

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb]     raise
libvirtError('virDomainAttachDeviceFlags() failed')

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
compute[70195]: ERROR nova.virt.libvirt.driver [instance:
b41513e6-cb4e-4441-af14-c272393cdafb] libvirt.libvirtError: internal
error: unable to execute QEMU command 'netdev_add': File descriptor
named '(null)' has not been found

Example run:
https://zuul.opendev.org/t/openstack/build/129de28a7a374013a03070f0742c46ac/log/controller/logs/screen-
n-cpu.txt?severity=4

libvirt version: libvirt x86_64  8.4.0-1.el9
qemu version: qemu-kvm x86_64  17:7.0.0-4.el9 

[1]https://zuul.opendev.org/t/openstack/builds?job_name=tempest-
integrated-compute-centos-9-stream&skip=0

** Affects: nova
     Importance: Critical
         Status: Triaged


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1979047

Title:
  Interface attach fails with libvirt.libvirtError: internal error:
  unable to execute QEMU command                'netdev_add': File
  descriptor named '(null)' has not been found

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  The tempest-integrated-compute-centos-9-stream job is broken since
  2022-06-16 02:26:37 [1]. Multiple interface attach tempest test fails
  with:

  libvirt.libvirtError: internal error: unable to execute QEMU command
  'netdev_add': File descriptor named '(null)' has not been found

  Full exception stack trace:

  [None req-6f95599e-022a-42ab-a4de-07c7b8f73daf tempest-
  AttachInterfacesTestJSON-2035965943 tempest-
  AttachInterfacesTestJSON-2035965943-project] [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb] attaching network adapter
  failed.: libvirt.libvirtError: internal error: unable to execute QEMU
  command 'netdev_add': File descriptor named '(null)' has not been
  found

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb] Traceback (most recent call
  last):

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2850, in
  attach_interface

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     guest.attach_device(cfg,
  persistent=True, live=live)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 321, in
  attach_device

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]
  self._domain.attachDeviceFlags(device_xml, flags=flags)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 193,
  in doit

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 151,
  in proxy_call

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     rv = execute(f, *args,
  **kwargs)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 132,
  in execute

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     six.reraise(c, e, tb)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     raise value

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 86,
  in tworker

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     rv = meth(*args, **kwargs)

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]   File
  "/usr/lib64/python3.9/site-packages/libvirt.py", line 706, in
  attachDeviceFlags

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb]     raise
  libvirtError('virDomainAttachDeviceFlags() failed')

  Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-
  compute[70195]: ERROR nova.virt.libvirt.driver [instance:
  b41513e6-cb4e-4441-af14-c272393cdafb] libvirt.libvirtError: internal
  error: unable to execute QEMU command 'netdev_add': File descriptor
  named '(null)' has not been found

  Example run:
  https://zuul.opendev.org/t/openstack/build/129de28a7a374013a03070f0742c46ac/log/controller/logs/screen-
  n-cpu.txt?severity=4

  libvirt version: libvirt x86_64  8.4.0-1.el9
  qemu version: qemu-kvm x86_64  17:7.0.0-4.el9 

  [1]https://zuul.opendev.org/t/openstack/builds?job_name=tempest-
  integrated-compute-centos-9-stream&skip=0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1979047/+subscriptions
Follow ups

[Bug 1979047] Re: Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
From: yatin, 2022-10-20
[Bug 1979047] Re: Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
From: Balazs Gibizer, 2022-07-12
[Bug 1979047] Re: Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
From: yatin, 2022-06-21