yahoo-eng-team team mailing list archive

[Alexey Stupnikov <2009156@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

stable/victoria merge process is effectively blocked because
test_live_block_migration_with_attached_volume tempest test (called as
tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume
and
tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_with_attached_volume)
is failing sporadically. As a result, nova-grenade-multinode or nova-
live-migration are failing sporadically: one of these jobs fails almost
every time.

I have used failed tests for review
https://review.opendev.org/c/openstack/nova/+/833436 to analyze the
problem.

Picture is always the same:

1. Tempest deletes volume attachment [1]
2. nova-compute fails to detach volume because of error [2]

It looks like a communication problem between nova-compute and libvirt.

[1]
2023-02-28 12:43:25.610324 | controller | 2023-02-28 12:40:08,991 6532 INFO     [tempest.lib.common.rest_client] Request (LiveAutoBlockMigrationV225Test:_run_cleanups): 202 DELETE http://213.32.76.87/compute/v2.1/servers/e49ccfac-97fc-4b8d-a15a-6a319d8c9a9b/os-volume_attachments/e0d103c8-cc39-4ccc-a91e-a415ff68bdee 0.217s

[2]
Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 472, in _do_wait_and_retry_detach
Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server     device=alternative_device_name, reason=reason)
Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2009156

Title:
  stable/victoria tests are very unstable because of tempest
  test_live_block_migration_with_attached_volume

Status in OpenStack Compute (nova):
  New

Bug description:
  stable/victoria merge process is effectively blocked because
  test_live_block_migration_with_attached_volume tempest test (called as
  tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume
  and
  tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_with_attached_volume)
  is failing sporadically. As a result, nova-grenade-multinode or nova-
  live-migration are failing sporadically: one of these jobs fails
  almost every time.

  I have used failed tests for review
  https://review.opendev.org/c/openstack/nova/+/833436 to analyze the
  problem.

  Picture is always the same:

  1. Tempest deletes volume attachment [1]
  2. nova-compute fails to detach volume because of error [2]

  It looks like a communication problem between nova-compute and
  libvirt.

  [1]
  2023-02-28 12:43:25.610324 | controller | 2023-02-28 12:40:08,991 6532 INFO     [tempest.lib.common.rest_client] Request (LiveAutoBlockMigrationV225Test:_run_cleanups): 202 DELETE http://213.32.76.87/compute/v2.1/servers/e49ccfac-97fc-4b8d-a15a-6a319d8c9a9b/os-volume_attachments/e0d103c8-cc39-4ccc-a91e-a415ff68bdee 0.217s

  [2]
  Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 472, in _do_wait_and_retry_detach
  Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server     device=alternative_device_name, reason=reason)
  Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2009156/+subscriptions