yahoo-eng-team team mailing list archive

[Alexey Stupnikov <2007860@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume
failed for one of my patches. Failure itself looks sporadic, but may be
a sign of underlying problems.

Complete log:
https://3597fc80eb5e70afafd6-8fe1ce567603c46a794844f7885416e9.ssl.cf1.rackcdn.com/873388/2/check/nova-
tox-functional-py310/6921a72/job-output.txt

My understanding:
- "Original exception" [1] is logged when nova-conductor is started
- This exception triggers calls, which in turn cause libvirt import and trigger ImportError exception:

2023-02-17 11:43:02.536761 | ubuntu-jammy | 2023-02-17 11:26:59,887
WARNING [nova.tests.fixtures.nova] !!!---!!! TestCase ID
nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume
hit the import poison while importing libvirt. If you see this in a
failed functional test then please let #openstack-nova on IRC know about
it. !!!---!!!

It looks like original issue is caused by some RPC call related to live
migration.

This problem seem to be sporadic and is not a blocker, but it looks
quite interesting and could affect CI in general, so I am reporting this
bug.

[1]
2023-02-17 11:43:02.529191 | ubuntu-jammy | 2023-02-17 11:26:59,782 ERROR [root] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send\n    reply, failure = reply_q.get(timeout=timeout)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n    return waiter.wait()\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n    return get_hub().switch()\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n    return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 9306, in _post_live_migration\n    self._update_scheduler_instance_info(ctxt, instance)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2225, in _update_scheduler_instance_info\n    self.query_client.update_instance_info(context, self.host,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 69, in update_instance_info\n    self.scheduler_rpcapi.update_instance_info(context, host_name,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 174, in update_instance_info\n    cctxt.cast(ctxt, \'update_instance_info\', host_name=host_name,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/fixtures/_fixtures/monkeypatch.py", line 86, in avoid_get\n    return captured_method(*args, **kwargs)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in c
2023-02-17 11:43:02.529623 | ubuntu-jammy |     all\n    result = self.transport._send(\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n    return self._driver.send(target, ctxt, message,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send\n    raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: No reply on topic scheduler\n']
2023-02-17 11:43:02.529678 | ubuntu-jammy | 2023-02-17 11:26:59,881 WARNING [nova.virt.libvirt.driver] Error monitoring migration: (sqlite3.OperationalError) no such table: compute_nodes

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2007860

Title:
  master nova-tox-functional-py310 fails because of
  VDPAServersTest.test_suspend_and_resume

Status in OpenStack Compute (nova):
  New

Bug description:
  nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume
  failed for one of my patches. Failure itself looks sporadic, but may
  be a sign of underlying problems.

  Complete log:
  https://3597fc80eb5e70afafd6-8fe1ce567603c46a794844f7885416e9.ssl.cf1.rackcdn.com/873388/2/check/nova-
  tox-functional-py310/6921a72/job-output.txt

  My understanding:
  - "Original exception" [1] is logged when nova-conductor is started
  - This exception triggers calls, which in turn cause libvirt import and trigger ImportError exception:

  2023-02-17 11:43:02.536761 | ubuntu-jammy | 2023-02-17 11:26:59,887
  WARNING [nova.tests.fixtures.nova] !!!---!!! TestCase ID
  nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume
  hit the import poison while importing libvirt. If you see this in a
  failed functional test then please let #openstack-nova on IRC know
  about it. !!!---!!!

  It looks like original issue is caused by some RPC call related to
  live migration.

  This problem seem to be sporadic and is not a blocker, but it looks
  quite interesting and could affect CI in general, so I am reporting
  this bug.

  [1]
  2023-02-17 11:43:02.529191 | ubuntu-jammy | 2023-02-17 11:26:59,782 ERROR [root] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send\n    reply, failure = reply_q.get(timeout=timeout)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n    return waiter.wait()\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n    return get_hub().switch()\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n    return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 9306, in _post_live_migration\n    self._update_scheduler_instance_info(ctxt, instance)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2225, in _update_scheduler_instance_info\n    self.query_client.update_instance_info(context, self.host,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 69, in update_instance_info\n    self.scheduler_rpcapi.update_instance_info(context, host_name,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 174, in update_instance_info\n    cctxt.cast(ctxt, \'update_instance_info\', host_name=host_name,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/fixtures/_fixtures/monkeypatch.py", line 86, in avoid_get\n    return captured_method(*args, **kwargs)\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in c
  2023-02-17 11:43:02.529623 | ubuntu-jammy |     all\n    result = self.transport._send(\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n    return self._driver.send(target, ctxt, message,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout,\n', '  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send\n    raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: No reply on topic scheduler\n']
  2023-02-17 11:43:02.529678 | ubuntu-jammy | 2023-02-17 11:26:59,881 WARNING [nova.virt.libvirt.driver] Error monitoring migration: (sqlite3.OperationalError) no such table: compute_nodes

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2007860/+subscriptions