yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1624383] Re: vif plugging fails with ovn when trying to set mtu on qvb device that does not exist

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Sun, 18 Sep 2016 18:31:26 -0000
Reply-to: Bug 1624383 <1624383@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/newton
   Importance: High
     Assignee: John Garbutt (johngarbutt)
       Status: Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1624383

Title:
  vif plugging fails with ovn when trying to set mtu on qvb device that
  does not exist

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Released

Bug description:
  Richard Theis reported the regression here:

  https://review.openstack.org/#/c/370681/9/nova/virt/libvirt/vif.py

  Shown here:

  http://logs.openstack.org/75/371175/1/check/gate-tempest-dsvm-
  networking-
  ovn/7e52927/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-16_11_20_39_627

  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [req-78de89a8-2053-42d9-899f-04aa2b8f25c7 tempest-FloatingIPsTestJSON-1913520192 tempest-FloatingIPsTestJSON-1913520192] [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Instance failed to spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Traceback (most recent call last):
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     yield resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     block_device_info=block_device_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     post_xml_callback=gen_confdrive)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4814, in _create_domain_and_network
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.plug_vifs(instance, network_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 684, in plug_vifs
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.vif_driver.plug(instance, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 817, in plug
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self._plug_os_vif(instance, vif_obj, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 799, in _plug_os_vif
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     linux_net._set_device_mtu(veth, mtu)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1237, in _set_device_mtu
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     check_exit_code=[0, 2, 254])
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 295, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 178, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return processutils.execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     cmd=sanitized_cmd)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] ProcessExecutionError: Unexpected error while running command.
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvb00b6a38f-60 mtu 1442
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Exit code: 1
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stdout: u''
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stderr: u'Cannot find device "qvb00b6a38f-60"\n'
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] 

  The nova team isn't sure if ovn should have those devices or not by
  the time we've plugged the vif, or if ovn is a special case and those
  just won't exist.

  We're going to add device exists checks around that code to be safe
  and get that into newton-rc2 and move forward unless someone more that
  knows more about OVN can tell us otherwise.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1624383/+subscriptions
References

[Bug 1624383] [NEW] vif plugging fails with ovn when trying to set mtu on qvb device that does not exist
From: Matt Riedemann, 2016-09-16