yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88092
[Bug 1939604] Re: [SRU] Cannot create 1vcpu instance with multiqueue image, vif_type=tap (calico)
** Changed in: cloud-archive/ussuri
Status: Triaged => Fix Released
** Changed in: cloud-archive/victoria
Status: Triaged => Fix Released
** Changed in: cloud-archive/wallaby
Status: Triaged => Fix Released
** Changed in: nova (Ubuntu Focal)
Status: Triaged => Fix Released
** Changed in: nova (Ubuntu Hirsute)
Status: Triaged => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1939604
Title:
[SRU] Cannot create 1vcpu instance with multiqueue image, vif_type=tap
(calico)
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive ussuri series:
Fix Released
Status in Ubuntu Cloud Archive victoria series:
Fix Released
Status in Ubuntu Cloud Archive wallaby series:
Fix Released
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ussuri series:
Fix Released
Status in OpenStack Compute (nova) victoria series:
Fix Released
Status in OpenStack Compute (nova) wallaby series:
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Focal:
Fix Released
Status in nova source package in Hirsute:
Fix Released
Bug description:
Tested on stable/wallaby
Fix for bug #1893263, in which it enabled vif_type=tap (calico use
case) devices to support multiqueue in nova, also caused a regression
where now when creating the instances with multiqueue, if using a
flavor with only VCPU, it fails with the error below in the logs.
This problem can easily be avoided by not using 1VCPUs flavors with
multiqueue images (because they wouldn't make sense anyway), and
therefore using non-multiqueue images when the flavor is 1VCPU, but
provides a bad user experience: Users shouldn't need to be concerned
about flavor+image combinations
Steps to reproduce are the same as #1893263 but using a 1VCPU flavor +
multiqueue metadata on images
21-08-11 17:36:44.317 376565 ERROR nova.compute.manager [req-99e80890-6c99-4015-91b6-ef99e6be3fa7 ea7dfe225d48428c860321498e184739 8833157a5d244727a74017e5f8729312 - 0373963ccb0042da8306b35775521d60 0373963ccb0042da8306b35775521d60] [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] Failed to build and run instance: libvirt.libvirtError: Unable to create tap device tap73a105b8-82: Invalid argument
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] Traceback (most recent call last):
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2366, in _build_and_run_instance
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.driver.spawn(context, instance, image_meta,
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3885, in spawn
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] self._create_guest_with_network(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6961, in _create_guest_with_network
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] self._cleanup_failed_start(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.force_reraise()
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] raise self.value
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6930, in _create_guest_with_network
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] guest = self._create_guest(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6863, in _create_guest
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] guest.launch(pause=pause)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 158, in launch
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] LOG.exception('Error launching a defined domain with XML: %s',
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.force_reraise()
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] raise self.value
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 155, in launch
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] return self._domain.createWithFlags(flags)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] rv = execute(f, *args, **kwargs)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] six.reraise(c, e, tb)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] raise value
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] rv = meth(*args, **kwargs)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b] libvirt.libvirtError: Unable to create tap device tap73a105b8-82: Invalid argument
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance: 505b68b4-498c-4ea9-85ce-8be0c305ec4b]
=====================================================================================================
SRU template below:
[Impact]
Impact only for very specific use case of calico plugin + 1vcpu flavor
+ multiqueue-enabled images. Workaround can avoid the issue but user
experience becomes suboptimal.
[Test case]
1. Setting up env
1a. Deploy environment
1b. Install calico plugin as per [0]
1c. Setup SSH
ssh-keygen
1d. Create keypair for testing
openstack keypair create key1 --public-key ~/.ssh/id_rsa.pub
1e. Create test flavors
openstack flavor create --vcpu 1 --ram 1024 --disk 10 --public --id 10
test_flavor1
openstack flavor create --vcpu 2 --ram 1024 --disk 10 --public --id 10
test_flavor2
1f. Download an example image
wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-
cloudimg-amd64.img
1g. Create image in glance with multiqueue metadata
openstack image create bionic-mq --file bionic-server-cloudimg-
amd64.img --property hw_vif_multiqueue_enabled=True
1h. Create instance with multiqueue + test_flavor2. Make sure instance
creation and connectivity succeeds.
openstack server create --network calico --flavor test_flavor2 --image
bionic-mq --key-name key1 mq2vcpu
2. Reproducing the bug
2a. Create instance with multiqueue + test_flavor1
openstack server create --network calico --flavor test_flavor1 --image
bionic-mq --key-name key1 mq1vcpu
Instance creation will fail
2b. Check logs for error
egrep "libvirt.libvirtError: Unable to create tap device .*: Invalid
argument" /var/log/nova/nova-compute.log
3. Cleanup
3a. Delete instances "mq2vcpu" and "mq1vcpu"
4. Install package that contains the fixed code
5. Repeat step 2a. 2a should now succeed.
[Regression Potential]
Looking at the code which is just 1 line, the behavior previous to fix
of #1893263 is restored in case vcpus=1. The only change in behavior
introduced by this change is when vcpus=1 and affect only calico
users.
[Other Info]
None
[0] https://docs.projectcalico.org/getting-
started/openstack/installation/
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1939604/+subscriptions
References