yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1624383] Re: vif plugging fails with ovn when trying to set mtu on qvb device that does not exist

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1624383@xxxxxxxxxxxxxxxxxx>
Date: Fri, 16 Sep 2016 23:24:47 -0000
Reply-to: Bug 1624383 <1624383@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/371543
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc0e281743917dbda25fb1911500439abed192ca
Submitter: Jenkins
Branch:    master

commit fc0e281743917dbda25fb1911500439abed192ca
Author: John Garbutt <john.garbutt@xxxxxxxxxxxxx>
Date:   Fri Sep 16 14:51:10 2016 +0100

    Stop ovn networking failing on mtu
    
    The type check needed to be more specific. There are some subclasses
    that don't create the devices we were trying to set the mtu on.
    
    Change-Id: Icc628a2dbde137d320fb78ad45b2ee0f7b5775fa
    Closes-Bug: #1624383


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1624383

Title:
  vif plugging fails with ovn when trying to set mtu on qvb device that
  does not exist

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Richard Theis reported the regression here:

  https://review.openstack.org/#/c/370681/9/nova/virt/libvirt/vif.py

  Shown here:

  http://logs.openstack.org/75/371175/1/check/gate-tempest-dsvm-
  networking-
  ovn/7e52927/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-16_11_20_39_627

  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [req-78de89a8-2053-42d9-899f-04aa2b8f25c7 tempest-FloatingIPsTestJSON-1913520192 tempest-FloatingIPsTestJSON-1913520192] [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Instance failed to spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Traceback (most recent call last):
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     yield resources
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     block_device_info=block_device_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     post_xml_callback=gen_confdrive)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4814, in _create_domain_and_network
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.plug_vifs(instance, network_info)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 684, in plug_vifs
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self.vif_driver.plug(instance, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 817, in plug
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     self._plug_os_vif(instance, vif_obj, vif)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 799, in _plug_os_vif
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     linux_net._set_device_mtu(veth, mtu)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1237, in _set_device_mtu
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     check_exit_code=[0, 2, 254])
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 295, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/opt/stack/new/nova/nova/utils.py", line 178, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     return processutils.execute(*cmd, **kwargs)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]     cmd=sanitized_cmd)
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] ProcessExecutionError: Unexpected error while running command.
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvb00b6a38f-60 mtu 1442
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Exit code: 1
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stdout: u''
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stderr: u'Cannot find device "qvb00b6a38f-60"\n'
  2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] 

  The nova team isn't sure if ovn should have those devices or not by
  the time we've plugged the vif, or if ovn is a special case and those
  just won't exist.

  We're going to add device exists checks around that code to be safe
  and get that into newton-rc2 and move forward unless someone more that
  knows more about OVN can tell us otherwise.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1624383/+subscriptions
References

[Bug 1624383] [NEW] vif plugging fails with ovn when trying to set mtu on qvb device that does not exist
From: Matt Riedemann, 2016-09-16