yahoo-eng-team team mailing list archive

[Edward Hope-Morley <1939604@xxxxxxxxxxxxxxxxxx>]

** Changed in: cloud-archive/ussuri
       Status: Triaged => Fix Released

** Changed in: cloud-archive/victoria
       Status: Triaged => Fix Released

** Changed in: cloud-archive/wallaby
       Status: Triaged => Fix Released

** Changed in: nova (Ubuntu Focal)
       Status: Triaged => Fix Released

** Changed in: nova (Ubuntu Hirsute)
       Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1939604

Title:
  [SRU] Cannot create 1vcpu instance with multiqueue image, vif_type=tap
  (calico)

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in Ubuntu Cloud Archive wallaby series:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ussuri series:
  Fix Released
Status in OpenStack Compute (nova) victoria series:
  Fix Released
Status in OpenStack Compute (nova) wallaby series:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Focal:
  Fix Released
Status in nova source package in Hirsute:
  Fix Released

Bug description:
  Tested on stable/wallaby

  Fix for bug #1893263, in which it enabled vif_type=tap (calico use
  case) devices to support multiqueue in nova, also caused a regression
  where now when creating the instances with multiqueue, if using a
  flavor with only VCPU, it fails with the error below in the logs.

  This problem can easily be avoided by not using 1VCPUs flavors with
  multiqueue images (because they wouldn't make sense anyway), and
  therefore using non-multiqueue images when the flavor is 1VCPU, but
  provides a bad user experience: Users shouldn't need to be concerned
  about flavor+image combinations

  Steps to reproduce are the same as #1893263 but using a 1VCPU flavor +
  multiqueue metadata on images

  21-08-11 17:36:44.317 376565 ERROR nova.compute.manager [req-99e80890-6c99-4015-91b6-ef99e6be3fa7 ea7dfe225d48428c860321498e184739 8833157a5d244727a74017e5f8729312 - 0373963ccb0042da8306b35775521d60 0373963ccb0042da8306b35775521d60] [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] Failed to build and run instance: libvirt.libvirtError: Unable to create tap device tap73a105b8-82: Invalid argument
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] Traceback (most recent call last):
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2366, in _build_and_run_instance
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     self.driver.spawn(context, instance, image_meta,
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3885, in spawn
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     self._create_guest_with_network(
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6961, in _create_guest_with_network
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     self._cleanup_failed_start(
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     self.force_reraise()
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     raise self.value
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6930, in _create_guest_with_network
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     guest = self._create_guest(
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6863, in _create_guest
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     guest.launch(pause=pause)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 158, in launch
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     LOG.exception('Error launching a defined domain with XML: %s',
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     self.force_reraise()
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     raise self.value
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 155, in launch
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     return self._domain.createWithFlags(flags)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     rv = execute(f, *args, **kwargs)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     six.reraise(c, e, tb)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     raise value
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     rv = meth(*args, **kwargs)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] libvirt.libvirtError: Unable to create tap device tap73a105b8-82: Invalid argument
  2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]

  =====================================================================================================

  SRU template below:

  [Impact]

  Impact only for very specific use case of calico plugin + 1vcpu flavor
  + multiqueue-enabled images. Workaround can avoid the issue but user
  experience becomes suboptimal.

  [Test case]

  1. Setting up env
  1a. Deploy environment
  1b. Install calico plugin as per [0]
  1c. Setup SSH

  ssh-keygen

  1d. Create keypair for testing

  openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub

  1e. Create test flavors

  openstack flavor create --vcpu 1 --ram 1024 --disk 10 --public --id 10
  test_flavor1

  openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10
  test_flavor2

  1f. Download an example image

  wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-
  cloudimg-amd64.img

  1g. Create image in glance with multiqueue metadata

  openstack image create bionic-mq --file bionic-server-cloudimg-
  amd64.img --property hw_vif_multiqueue_enabled=True

  1h. Create instance with multiqueue + test_flavor2. Make sure instance
  creation and connectivity succeeds.

  openstack server create --network calico --flavor test_flavor2 --image
  bionic-mq --key-name key1 mq2vcpu

  2. Reproducing the bug

  2a. Create instance with multiqueue + test_flavor1

  openstack server create --network calico --flavor test_flavor1 --image
  bionic-mq --key-name key1 mq1vcpu

  Instance creation will fail

  2b. Check logs for error

  egrep "libvirt.libvirtError: Unable to create tap device .*: Invalid
  argument" /var/log/nova/nova-compute.log

  3. Cleanup

  3a. Delete instances "mq2vcpu" and "mq1vcpu"

  4. Install package that contains the fixed code

  5. Repeat step 2a. 2a should now succeed.

  [Regression Potential]

  Looking at the code which is just 1 line, the behavior previous to fix
  of #1893263 is restored in case vcpus=1. The only change in behavior
  introduced by this change is when vcpus=1 and affect only calico
  users.

  [Other Info]

  None

  [0] https://docs.projectcalico.org/getting-
  started/openstack/installation/

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1939604/+subscriptions