yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2065689] [NEW] live_block_migration tests randomly fails with libvirt.libvirtError: unable to connect to server at '<host>': Connection refused

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2065689@xxxxxxxxxxxxxxxxxx>
Date: Tue, 14 May 2024 13:20:13 -0000
Reply-to: Bug 2065689 <2065689@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Test fails like:-
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 150, in test_live_block_migration
    self._test_live_migration()
  File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 135, in _test_live_migration
    self._live_migrate(server_id, destination_host, state, volume_backed)
  File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 94, in _live_migrate
    self.assertEqual(target_host, self.get_host_for_server(server_id),
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 395, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 482, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'np0037423811' != 'np0037423819': Live Migration failed. Migrations list for Instance 490cfc79-aa11-481d-82e3-99ca38fd2a07: [
{'id': 1, 'source_compute': 'np0037423819', 'dest_compute': 'np0037423811', 'source_node': 'np0037423819', 'dest_node': 'np0037423811', 'dest_host': None, 'old_instance_type_id': 11, 'new_instance_type_id': 11, 'instance_uuid': '490cfc79-aa11-481d-82e3-99ca38fd2a07', 'status': 'failed', 'migration_type': 'live-migration', 'created_at': '2024-05-02T10:11:10.000000', 'updated_at': '2024-05-02T10:11:17.000000'}]

sample failures:-
- http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d01/860283/32/check/nova-live-migration/d01ef23/testr_results.html
- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_397/916112/6/gate/neutron-ovs-tempest-multinode-full/3978b33/testr_results.html
- https://d64b7553b36edb75caae-206dd41ab15ee6b59e703e19aac8d4d8.ssl.cf1.rackcdn.com/918195/8/check/nova-grenade-multinode/506fec9/testr_results.html

Nova compute have Traces:-
May 02 10:11:16.775995 np0037423819 nova-compute[30136]: ERROR nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Live Migration failure: unable to connect to server at 'np0037423811:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused
May 02 10:11:16.776306 np0037423819 nova-compute[30136]: DEBUG nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Migration operation thread notification {{(pid=30136) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:11059}}
May 02 10:11:16.781772 np0037423819 nova-compute[30136]: Traceback (most recent call last):
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     timer()
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     cb(*args, **kw)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     waiter.switch(result)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     result = function(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     return func(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10713, in _live_migration_operation
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     with excutils.save_and_reraise_exception():
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     self.force_reraise()
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise self.value
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10702, in _live_migration_operation
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     guest.migrate(self._live_migration_uri(dest),
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 647, in migrate
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     self._domain.migrateToURI3(
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     rv = execute(f, *args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise e.with_traceback(tb)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     rv = meth(*args, **kwargs)
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise libvirtError('virDomainMigrateToURI3() failed')
May 02 10:11:16.781922 np0037423819 nova-compute[30136]: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2065689

Title:
   live_block_migration tests randomly fails with  libvirt.libvirtError:
  unable to connect to server at '<host>': Connection refused

Status in OpenStack Compute (nova):
  New

Bug description:
  Test fails like:-
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 150, in test_live_block_migration
      self._test_live_migration()
    File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 135, in _test_live_migration
      self._live_migrate(server_id, destination_host, state, volume_backed)
    File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 94, in _live_migrate
      self.assertEqual(target_host, self.get_host_for_server(server_id),
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 395, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/testtools/testcase.py", line 482, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: 'np0037423811' != 'np0037423819': Live Migration failed. Migrations list for Instance 490cfc79-aa11-481d-82e3-99ca38fd2a07: [
  {'id': 1, 'source_compute': 'np0037423819', 'dest_compute': 'np0037423811', 'source_node': 'np0037423819', 'dest_node': 'np0037423811', 'dest_host': None, 'old_instance_type_id': 11, 'new_instance_type_id': 11, 'instance_uuid': '490cfc79-aa11-481d-82e3-99ca38fd2a07', 'status': 'failed', 'migration_type': 'live-migration', 'created_at': '2024-05-02T10:11:10.000000', 'updated_at': '2024-05-02T10:11:17.000000'}]

  sample failures:-
  - http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d01/860283/32/check/nova-live-migration/d01ef23/testr_results.html
  - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_397/916112/6/gate/neutron-ovs-tempest-multinode-full/3978b33/testr_results.html
  - https://d64b7553b36edb75caae-206dd41ab15ee6b59e703e19aac8d4d8.ssl.cf1.rackcdn.com/918195/8/check/nova-grenade-multinode/506fec9/testr_results.html

  Nova compute have Traces:-
  May 02 10:11:16.775995 np0037423819 nova-compute[30136]: ERROR nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Live Migration failure: unable to connect to server at 'np0037423811:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused
  May 02 10:11:16.776306 np0037423819 nova-compute[30136]: DEBUG nova.virt.libvirt.driver [None req-0ef45e21-38a4-44ae-aa70-6d59daf17b5b tempest-LiveAutoBlockMigrationV225Test-2095721713 tempest-LiveAutoBlockMigrationV225Test-2095721713-project-admin] [instance: 490cfc79-aa11-481d-82e3-99ca38fd2a07] Migration operation thread notification {{(pid=30136) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:11059}}
  May 02 10:11:16.781772 np0037423819 nova-compute[30136]: Traceback (most recent call last):
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     timer()
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     cb(*args, **kw)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     waiter.switch(result)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     result = function(*args, **kwargs)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     return func(*args, **kwargs)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10713, in _live_migration_operation
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     with excutils.save_and_reraise_exception():
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     self.force_reraise()
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise self.value
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10702, in _live_migration_operation
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     guest.migrate(self._live_migration_uri(dest),
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 647, in migrate
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     self._domain.migrateToURI3(
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     rv = execute(f, *args, **kwargs)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise e.with_traceback(tb)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     rv = meth(*args, **kwargs)
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:   File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]:     raise libvirtError('virDomainMigrateToURI3() failed')
  May 02 10:11:16.781922 np0037423819 nova-compute[30136]: libvirt.libvirtError: unable to connect to server at 'np0037423811:49152': Connection refused

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2065689/+subscriptions