← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #56666

[Bug 1624383] [NEW] vif plugging fails with ovn when trying to set mtu on qvb device that does not exist

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Richard Theis reported the regression here:

https://review.openstack.org/#/c/370681/9/nova/virt/libvirt/vif.py

Shown here:

http://logs.openstack.org/75/371175/1/check/gate-tempest-dsvm-
networking-
ovn/7e52927/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-16_11_20_39_627

2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [req-78de89a8-2053-42d9-899f-04aa2b8f25c7 tempest-FloatingIPsTestJSON-1913520192 tempest-FloatingIPsTestJSON-1913520192] [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Instance failed to spawn
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Traceback (most recent call last):
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     yield resources
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     block_device_info=block_device_info)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     post_xml_callback=gen_confdrive)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4814, in _create_domain_and_network
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.plug_vifs(instance, network_info)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 684, in plug_vifs
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.vif_driver.plug(instance, vif)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 817, in plug
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self._plug_os_vif(instance, vif_obj, vif)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 799, in _plug_os_vif
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     linux_net._set_device_mtu(veth, mtu)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1237, in _set_device_mtu
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     check_exit_code=[0, 2, 254])
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 295, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 178, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return processutils.execute(*cmd, **kwargs)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     cmd=sanitized_cmd)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] ProcessExecutionError: Unexpected error while running command.
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvb00b6a38f-60 mtu 1442
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Exit code: 1
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stdout: u''
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stderr: u'Cannot find device "qvb00b6a38f-60"\n'
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] 

The nova team isn't sure if ovn should have those devices or not by the
time we've plugged the vif, or if ovn is a special case and those just
won't exist.

We're going to add device exists checks around that code to be safe and
get that into newton-rc2 and move forward unless someone more that knows
more about OVN can tell us otherwise.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt neutron newton-rc-potential ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1624383

Title:
  vif plugging fails with ovn when trying to set mtu on qvb device that
  does not exist

Status in OpenStack Compute (nova):
  New

Bug description:
  Richard Theis reported the regression here:

  https://review.openstack.org/#/c/370681/9/nova/virt/libvirt/vif.py

  Shown here:

  http://logs.openstack.org/75/371175/1/check/gate-tempest-dsvm-
  networking-
  ovn/7e52927/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-16_11_20_39_627

  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [req-78de89a8-2053-42d9-899f-04aa2b8f25c7 tempest-FloatingIPsTestJSON-1913520192 tempest-FloatingIPsTestJSON-1913520192] [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Instance failed to spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Traceback (most recent call last):
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     yield resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     block_device_info=block_device_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     post_xml_callback=gen_confdrive)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4814, in _create_domain_and_network
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.plug_vifs(instance, network_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 684, in plug_vifs
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.vif_driver.plug(instance, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 817, in plug
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self._plug_os_vif(instance, vif_obj, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 799, in _plug_os_vif
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     linux_net._set_device_mtu(veth, mtu)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1237, in _set_device_mtu
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     check_exit_code=[0, 2, 254])
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 295, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 178, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return processutils.execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     cmd=sanitized_cmd)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] ProcessExecutionError: Unexpected error while running command.
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvb00b6a38f-60 mtu 1442
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Exit code: 1
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stdout: u''
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stderr: u'Cannot find device "qvb00b6a38f-60"\n'
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] 

  The nova team isn't sure if ovn should have those devices or not by
  the time we've plugged the vif, or if ovn is a special case and those
  just won't exist.

  We're going to add device exists checks around that code to be safe
  and get that into newton-rc2 and move forward unless someone more that
  knows more about OVN can tell us otherwise.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1624383/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next