yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1794126] [NEW] typo in IVS related privsep method, VM creation failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Aditya Vaja <wolverine.av@xxxxxxxxx>
Date: Mon, 24 Sep 2018 15:07:35 -0000
Reply-to: Bug 1794126 <1794126@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

As part of the change here [1], a few calls to edit files as root were moved to a privsep class that encompasses all those edits.
A typo creeped in, causing a failure when creating a VM with VIF type IVS:

log excerpt for failure:
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [req-e2560c77-8950-4dc6-8f4f-44254b184bbb 17bbd148d3f34a42bede6bdb8fcc84aa 62daca4c6dff406685f3fcf754cb82c4 - default default] [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] Instance failed to spawn: IOError: [Errno 2] No such file or directory
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] Traceback (most recent call last):
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     yield resources
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     block_device_info=block_device_info)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3100, in spawn
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     destroy_disks_on_failure=True)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5590, in _create_domain_and_network
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     destroy_disks_on_failure)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.force_reraise()
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     six.reraise(self.type_, self.value, self.tb)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5549, in _create_domain_and_network
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.plug_vifs(instance, network_info)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 829, in plug_vifs
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.vif_driver.plug(instance, vif)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 896, in plug
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     func(instance, vif)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 722, in plug_ivs
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.plug_ivs_hybrid(instance, vif)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 703, in plug_ivs_hybrid
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     nova.privsep.libvirt.disable_ipv6(br_name)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     return self.channel.remote_call(name, args, kwargs)
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     raise exc_type(*result[2])
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] IOError: [Errno 2] No such file or directory
2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]

I'll open a fix PR as well. And add a check that file exists before
editing - apparently that's not true for this particular case and wasn't
carried over from existing code.

Note: only affects stable/queens branch. Later releases has proper
calling to os-vif-bigswitch and this code has been cleaned up from Nova.

[1] https://review.openstack.org/#/c/489438/

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1794126

Title:
  typo in IVS related privsep method, VM creation failure

Status in OpenStack Compute (nova):
  New

Bug description:
  As part of the change here [1], a few calls to edit files as root were moved to a privsep class that encompasses all those edits.
  A typo creeped in, causing a failure when creating a VM with VIF type IVS:

  log excerpt for failure:
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [req-e2560c77-8950-4dc6-8f4f-44254b184bbb 17bbd148d3f34a42bede6bdb8fcc84aa 62daca4c6dff406685f3fcf754cb82c4 - default default] [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] Instance failed to spawn: IOError: [Errno 2] No such file or directory
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] Traceback (most recent call last):
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     yield resources
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     block_device_info=block_device_info)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3100, in spawn
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     destroy_disks_on_failure=True)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5590, in _create_domain_and_network
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     destroy_disks_on_failure)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.force_reraise()
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     six.reraise(self.type_, self.value, self.tb)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5549, in _create_domain_and_network
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.plug_vifs(instance, network_info)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 829, in plug_vifs
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.vif_driver.plug(instance, vif)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 896, in plug
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     func(instance, vif)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 722, in plug_ivs
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     self.plug_ivs_hybrid(instance, vif)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 703, in plug_ivs_hybrid
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     nova.privsep.libvirt.disable_ipv6(br_name)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     return self.channel.remote_call(name, args, kwargs)
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]     raise exc_type(*result[2])
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d] IOError: [Errno 2] No such file or directory
  2018-09-24 14:21:47.167 1 ERROR nova.compute.manager [instance: a631eff3-9d68-4e06-a7ce-4338e29e8e2d]

  I'll open a fix PR as well. And add a check that file exists before
  editing - apparently that's not true for this particular case and
  wasn't carried over from existing code.

  Note: only affects stable/queens branch. Later releases has proper
  calling to os-vif-bigswitch and this code has been cleaned up from
  Nova.

  [1] https://review.openstack.org/#/c/489438/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1794126/+subscriptions
Follow ups

[Bug 1794126] Re: typo in IVS related privsep method, VM creation failure
From: Matt Riedemann, 2018-10-03