yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #90081
[Bug 1994913] [NEW] Unit tests can fail if python guestfs is installed in the test environment
Public bug reported:
Ran into this downstream [1] where a unit test failed because it
actually tried to call libvirt, which of course failed:
libguestfs: error: could not create appliance through libvirt.
Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct
Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-10-22T13:20:42.059709Z qemu-kvm: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}: Image is not in qcow2 format [code=1 int1=-1]
{5} nova.tests.unit.virt.disk.test_api.APITestCase.test_can_resize_need_fs_type_specified [191.116248s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/tests/unit/virt/disk/test_api.py", line 47, in test_can_resize_need_fs_type_specified
self.assertFalse(api.is_image_extendable(image))
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/api.py", line 190, in is_image_extendable
fs.setup(mount=False)
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/vfs/guestfs.py", line 235, in setup
self.handle.launch()
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 121, in execute
rv = e.wait()
File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
result = hub.switch()
File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
return self.greenlet.switch()
File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
presult = self.do_poll(seconds)
File "/usr/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
return self.poll.poll(seconds)
File "/usr/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
That ^ was with libguestfs 1.46.1.
Repro steps (not identical traceback but close):
1. Make sure you have libguestfs header files
$ sudo apt install libguestfs-dev
2. Enter the unit test venv
$ source .tox/py38/bin/activate
3. Install the libguestfs python bindings into the venv. This has to be
done by tarball [2]. Note that the bindings were not built for python 3
until version 1.40 [3]
$ pip install
http://libguestfs.org/download/python/guestfs-1.40.2.tar.gz
4. Run the unit test
$ tox -epy38 test_can_resize_need_fs_type_specified
5. The test should fail
The code in nova will call libvirt if guestfs is installed [4] and if it
is not installed, it will emit a warning and move on [5]. In my repro
environment it didn't get as far as a call to libvirt though.
Upstream the test doesn't fail because neither libvirt nor guestfs are
installed in the CI environment for unit test jobs.
A unit test shouldn't be able to call out to real guestfs or libvirt and
it wasn't mocked out properly.
We can address it by adding mocking to the unit test and create a
fixture that will poison future imports by unit or func tests.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2135856
[2] https://libguestfs.org/guestfs-python.3.html#using-python-bindings-in-a-virtualenv
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1627964#c15
[4] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/vfs/guestfs.py#L66-L70
[5] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/api.py#L189-L201
** Affects: nova
Importance: Undecided
Assignee: melanie witt (melwitt)
Status: New
** Tags: testing
** Description changed:
Ran into this downstream [1] where a unit test failed because it
actually tried to call libvirt, which of course failed:
libguestfs: error: could not create appliance through libvirt.
Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct
Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-10-22T13:20:42.059709Z qemu-kvm: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}: Image is not in qcow2 format [code=1 int1=-1]
{5} nova.tests.unit.virt.disk.test_api.APITestCase.test_can_resize_need_fs_type_specified [191.116248s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
- Traceback (most recent call last):
- File "/home/zuul/src/code.engineering.redhat.com/nova/nova/tests/unit/virt/disk/test_api.py", line 47, in test_can_resize_need_fs_type_specified
- self.assertFalse(api.is_image_extendable(image))
- File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/api.py", line 190, in is_image_extendable
- fs.setup(mount=False)
- File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/vfs/guestfs.py", line 235, in setup
- self.handle.launch()
- File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
- result = proxy_call(self._autowrap, f, *args, **kwargs)
- File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
- rv = execute(f, *args, **kwargs)
- File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 121, in execute
- rv = e.wait()
- File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
- result = hub.switch()
- File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
- return self.greenlet.switch()
- File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 365, in run
- self.wait(sleep_time)
- File "/usr/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
- presult = self.do_poll(seconds)
- File "/usr/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
- return self.poll.poll(seconds)
- File "/usr/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
- raise TimeoutException()
- fixtures._fixtures.timeout.TimeoutException
+ Traceback (most recent call last):
+ File "/home/zuul/src/code.engineering.redhat.com/nova/nova/tests/unit/virt/disk/test_api.py", line 47, in test_can_resize_need_fs_type_specified
+ self.assertFalse(api.is_image_extendable(image))
+ File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/api.py", line 190, in is_image_extendable
+ fs.setup(mount=False)
+ File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/vfs/guestfs.py", line 235, in setup
+ self.handle.launch()
+ File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
+ result = proxy_call(self._autowrap, f, *args, **kwargs)
+ File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
+ rv = execute(f, *args, **kwargs)
+ File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 121, in execute
+ rv = e.wait()
+ File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
+ result = hub.switch()
+ File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
+ return self.greenlet.switch()
+ File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 365, in run
+ self.wait(sleep_time)
+ File "/usr/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
+ presult = self.do_poll(seconds)
+ File "/usr/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
+ return self.poll.poll(seconds)
+ File "/usr/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
+ raise TimeoutException()
+ fixtures._fixtures.timeout.TimeoutException
That ^ was with libguestfs 1.46.1.
-
Repro steps (not identical traceback but close):
1. Make sure you have libguestfs header files
$ sudo apt install libguestfs-dev
2. Enter the unit test venv
$ source .tox/py38/bin/activate
3. Install the libguestfs python bindings into the venv. This has to be
- done by tarball [4]. Note that the bindings were not built for python 3
- until version 1.40 [5]
+ done by tarball [2]. Note that the bindings were not built for python 3
+ until version 1.40 [3]
$ pip install
http://libguestfs.org/download/python/guestfs-1.40.2.tar.gz
4. Run the unit test
$ tox -epy38 test_can_resize_need_fs_type_specified
5. The test should fail
-
- The code in nova will call libvirt if guestfs is installed [2] and if it is not installed, it will emit a warning and move on [3]. In my repro environment it didn't get as far as a call to libvirt though.
+ The code in nova will call libvirt if guestfs is installed [4] and if it
+ is not installed, it will emit a warning and move on [5]. In my repro
+ environment it didn't get as far as a call to libvirt though.
Upstream the test doesn't fail because neither libvirt nor guestfs are
installed in the CI environment for unit test jobs.
A unit test shouldn't be able to call out to real guestfs or libvirt and
it wasn't mocked out properly.
We can address it by adding mocking to the unit test and create a
fixture that will poison future imports by unit or func tests.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2135856
- [2] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/vfs/guestfs.py#L66-L70
- [3] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/api.py#L189-L201
- [4] https://libguestfs.org/guestfs-python.3.html#using-python-bindings-in-a-virtualenv
- [5] https://bugzilla.redhat.com/show_bug.cgi?id=1627964#c15
+ [2] https://libguestfs.org/guestfs-python.3.html#using-python-bindings-in-a-virtualenv
+ [3] https://bugzilla.redhat.com/show_bug.cgi?id=1627964#c15
+ [4] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/vfs/guestfs.py#L66-L70
+ [5] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/api.py#L189-L201
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1994913
Title:
Unit tests can fail if python guestfs is installed in the test
environment
Status in OpenStack Compute (nova):
New
Bug description:
Ran into this downstream [1] where a unit test failed because it
actually tried to call libvirt, which of course failed:
libguestfs: error: could not create appliance through libvirt.
Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct
Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-10-22T13:20:42.059709Z qemu-kvm: -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}: Image is not in qcow2 format [code=1 int1=-1]
{5} nova.tests.unit.virt.disk.test_api.APITestCase.test_can_resize_need_fs_type_specified [191.116248s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/tests/unit/virt/disk/test_api.py", line 47, in test_can_resize_need_fs_type_specified
self.assertFalse(api.is_image_extendable(image))
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/api.py", line 190, in is_image_extendable
fs.setup(mount=False)
File "/home/zuul/src/code.engineering.redhat.com/nova/nova/virt/disk/vfs/guestfs.py", line 235, in setup
self.handle.launch()
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 121, in execute
rv = e.wait()
File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
result = hub.switch()
File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
return self.greenlet.switch()
File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 365, in run
self.wait(sleep_time)
File "/usr/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
presult = self.do_poll(seconds)
File "/usr/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
return self.poll.poll(seconds)
File "/usr/lib/python3.9/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
That ^ was with libguestfs 1.46.1.
Repro steps (not identical traceback but close):
1. Make sure you have libguestfs header files
$ sudo apt install libguestfs-dev
2. Enter the unit test venv
$ source .tox/py38/bin/activate
3. Install the libguestfs python bindings into the venv. This has to
be done by tarball [2]. Note that the bindings were not built for
python 3 until version 1.40 [3]
$ pip install
http://libguestfs.org/download/python/guestfs-1.40.2.tar.gz
4. Run the unit test
$ tox -epy38 test_can_resize_need_fs_type_specified
5. The test should fail
The code in nova will call libvirt if guestfs is installed [4] and if
it is not installed, it will emit a warning and move on [5]. In my
repro environment it didn't get as far as a call to libvirt though.
Upstream the test doesn't fail because neither libvirt nor guestfs are
installed in the CI environment for unit test jobs.
A unit test shouldn't be able to call out to real guestfs or libvirt
and it wasn't mocked out properly.
We can address it by adding mocking to the unit test and create a
fixture that will poison future imports by unit or func tests.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2135856
[2] https://libguestfs.org/guestfs-python.3.html#using-python-bindings-in-a-virtualenv
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1627964#c15
[4] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/vfs/guestfs.py#L66-L70
[5] https://github.com/openstack/nova/blob/b1958b7cfa6b8aca5b76b3f133627bb733d29f00/nova/virt/disk/api.py#L189-L201
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1994913/+subscriptions
Follow ups
