← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #45014

[Bug 1536671] [NEW] libvirt detach_interface logs errors for network device not found after neutron network-vif-deleted event

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I've started noticing a lot of these in the neutron job logs:

http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron-
src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_667

2016-01-21 05:38:48.667 ERROR nova.virt.libvirt.driver [req-c8971f87-303e-460b-894b-7e2ccde9944f nova service] [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] detaching network adapter failed.
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Traceback (most recent call last):
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1354, in detach_interface
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     guest.detach_device(cfg, persistent=True, live=live)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 341, in detach_device
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     self._domain.detachDeviceFlags(conf.to_xml(), flags=flags)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     rv = execute(f, *args, **kwargs)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     six.reraise(c, e, tb)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     rv = meth(*args, **kwargs)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 985, in detachDeviceFlags
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] libvirtError: operation failed: no matching network device was found
2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] 

Following the request ID, it's coming from a neutron vif deleted event:

http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron-
src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_361

It looks like we should just handle and not log that network device not
found as an error, but libvirt doesn't provide a specific error code for
that case:

http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=e04a32841807851fd1386ab8a7fd91672f114dce;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l6764

And the error message is translatable so we can't just parse the message
either.

It seems there should be a way to verify that the network device exists
before we try to detach it from the config so we can avoid this error
which doesn't actually result in job failures (it's probably happening
during an instance delete asynchronously where we're deleting the ports
in neutron).

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Caught%20error%3A%20%3Ctype%20'exceptions.AttributeError'%3E%20'module'%20object%20has%20no%20attribute%20'ServiceList'%5C%22%20AND%20tags%3A%5C%22screen-c-api.txt%5C%22&from=7d

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt network neutron

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1536671

Title:
  libvirt detach_interface logs errors for network device not found
  after neutron network-vif-deleted event

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I've started noticing a lot of these in the neutron job logs:

  http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron-
  src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_667

  2016-01-21 05:38:48.667 ERROR nova.virt.libvirt.driver [req-c8971f87-303e-460b-894b-7e2ccde9944f nova service] [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] detaching network adapter failed.
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Traceback (most recent call last):
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1354, in detach_interface
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     guest.detach_device(cfg, persistent=True, live=live)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 341, in detach_device
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     self._domain.detachDeviceFlags(conf.to_xml(), flags=flags)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     rv = execute(f, *args, **kwargs)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     six.reraise(c, e, tb)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     rv = meth(*args, **kwargs)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 985, in detachDeviceFlags
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d]     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] libvirtError: operation failed: no matching network device was found
  2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] 

  Following the request ID, it's coming from a neutron vif deleted
  event:

  http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron-
  src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_361

  It looks like we should just handle and not log that network device
  not found as an error, but libvirt doesn't provide a specific error
  code for that case:

  http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=e04a32841807851fd1386ab8a7fd91672f114dce;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l6764

  And the error message is translatable so we can't just parse the
  message either.

  It seems there should be a way to verify that the network device
  exists before we try to detach it from the config so we can avoid this
  error which doesn't actually result in job failures (it's probably
  happening during an instance delete asynchronously where we're
  deleting the ports in neutron).

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Caught%20error%3A%20%3Ctype%20'exceptions.AttributeError'%3E%20'module'%20object%20has%20no%20attribute%20'ServiceList'%5C%22%20AND%20tags%3A%5C%22screen-c-api.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1536671/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next