← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #73582

[Bug 1779270] [NEW] live-migration tempest test failed because the instance didn't seem to migrate to its target host

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
test_live_block_migration[0][1][2] tempest tests failed frequently[3].

[0] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration?duration=P1M
[1] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration?duration=P1M
[2] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test.test_live_block_migration?duration=P1M
[3] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Live%20Migration%20failed.%20Migrations%20list%20for%20Instance%20%5C%22

Steps to reproduce
==================
* run nova-live-migration, neutron-tempest-multinode-full or some other job which contains the test_live_block_migration test.

Expected result
===============
* The test doesn't fail. The instance is migrated.

Actual result
=============
* The instance doesn't seem to be migrated, and then, the test fails.

Environment
===========
* In the gate, master branch. Not sure about the stable branches.

Logs & Configs
==============
Quote from http://logs.openstack.org/34/575734/8/gate/nova-live-migration/cb90e1d/job-output.txt.gz

------------
2018-06-25 19:47:23.645249 | primary | 2018-06-25 19:47:23.643 | {1} tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration [25.911015s] ... FAILED
2018-06-25 19:47:23.647327 | primary | 2018-06-25 19:47:23.646 |
2018-06-25 19:47:23.653059 | primary | 2018-06-25 19:47:23.648 | Captured traceback:
2018-06-25 19:47:23.655235 | primary | 2018-06-25 19:47:23.654 | ~~~~~~~~~~~~~~~~~~~
2018-06-25 19:47:23.657181 | primary | 2018-06-25 19:47:23.656 |     Traceback (most recent call last):
2018-06-25 19:47:23.658951 | primary | 2018-06-25 19:47:23.658 |       File "tempest/api/compute/admin/test_live_migration.py", line 127, in test_live_block_migration
2018-06-25 19:47:23.664233 | primary | 2018-06-25 19:47:23.663 |         self._test_live_migration()
2018-06-25 19:47:23.669330 | primary | 2018-06-25 19:47:23.668 |       File "tempest/api/compute/admin/test_live_migration.py", line 117, in _test_live_migration
2018-06-25 19:47:23.671351 | primary | 2018-06-25 19:47:23.670 |         self._live_migrate(server_id, destination_host, state, volume_backed)
2018-06-25 19:47:23.673324 | primary | 2018-06-25 19:47:23.672 |       File "tempest/api/compute/admin/test_live_migration.py", line 91, in _live_migrate
2018-06-25 19:47:23.676421 | primary | 2018-06-25 19:47:23.675 |         msg)
2018-06-25 19:47:23.678566 | primary | 2018-06-25 19:47:23.678 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
2018-06-25 19:47:23.680280 | primary | 2018-06-25 19:47:23.679 |         self.assertThat(observed, matcher, message)
2018-06-25 19:47:23.682290 | primary | 2018-06-25 19:47:23.681 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
2018-06-25 19:47:23.685235 | primary | 2018-06-25 19:47:23.683 |         raise mismatch_error
2018-06-25 19:47:23.688145 | primary | 2018-06-25 19:47:23.687 |     testtools.matchers._impl.MismatchError: !=:
2018-06-25 19:47:23.690041 | primary | 2018-06-25 19:47:23.689 |     reference = u'ubuntu-xenial-ovh-gra1-0000332527'
2018-06-25 19:47:23.691611 | primary | 2018-06-25 19:47:23.691 |     actual    = u'ubuntu-xenial-ovh-gra1-0000332530'
2018-06-25 19:47:23.693251 | primary | 2018-06-25 19:47:23.692 |     : Live Migration failed. Migrations list for Instance 607cb757-f381-44b4-aff9-5cafe6d2cdfe: [
2018-06-25 19:47:23.695192 | primary | 2018-06-25 19:47:23.694 |     {u'id': 9, u'instance_uuid': u'607cb757-f381-44b4-aff9-5cafe6d2cdfe', u'status': u'error', u'updated_at': u'2018-06-25T19:47:23.000000', u'new_instance_type_id': 11, u'old_instance_type_id': 11, u'source_node': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_host': None, u'source_compute': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_node': u'ubuntu-xenial-ovh-gra1-0000332527', u'created_at': u'2018-06-25T19:47:11.000000', u'dest_compute': u'ubuntu-xenial-ovh-gra1-0000332527'}]
------------

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1779270

Title:
  live-migration tempest test failed because the instance didn't seem to
  migrate to its target host

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  test_live_block_migration[0][1][2] tempest tests failed frequently[3].

  [0] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration?duration=P1M
  [1] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration?duration=P1M
  [2] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test.test_live_block_migration?duration=P1M
  [3] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Live%20Migration%20failed.%20Migrations%20list%20for%20Instance%20%5C%22

  Steps to reproduce
  ==================
  * run nova-live-migration, neutron-tempest-multinode-full or some other job which contains the test_live_block_migration test.

  Expected result
  ===============
  * The test doesn't fail. The instance is migrated.

  Actual result
  =============
  * The instance doesn't seem to be migrated, and then, the test fails.

  Environment
  ===========
  * In the gate, master branch. Not sure about the stable branches.

  Logs & Configs
  ==============
  Quote from http://logs.openstack.org/34/575734/8/gate/nova-live-migration/cb90e1d/job-output.txt.gz

  ------------
  2018-06-25 19:47:23.645249 | primary | 2018-06-25 19:47:23.643 | {1} tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration [25.911015s] ... FAILED
  2018-06-25 19:47:23.647327 | primary | 2018-06-25 19:47:23.646 |
  2018-06-25 19:47:23.653059 | primary | 2018-06-25 19:47:23.648 | Captured traceback:
  2018-06-25 19:47:23.655235 | primary | 2018-06-25 19:47:23.654 | ~~~~~~~~~~~~~~~~~~~
  2018-06-25 19:47:23.657181 | primary | 2018-06-25 19:47:23.656 |     Traceback (most recent call last):
  2018-06-25 19:47:23.658951 | primary | 2018-06-25 19:47:23.658 |       File "tempest/api/compute/admin/test_live_migration.py", line 127, in test_live_block_migration
  2018-06-25 19:47:23.664233 | primary | 2018-06-25 19:47:23.663 |         self._test_live_migration()
  2018-06-25 19:47:23.669330 | primary | 2018-06-25 19:47:23.668 |       File "tempest/api/compute/admin/test_live_migration.py", line 117, in _test_live_migration
  2018-06-25 19:47:23.671351 | primary | 2018-06-25 19:47:23.670 |         self._live_migrate(server_id, destination_host, state, volume_backed)
  2018-06-25 19:47:23.673324 | primary | 2018-06-25 19:47:23.672 |       File "tempest/api/compute/admin/test_live_migration.py", line 91, in _live_migrate
  2018-06-25 19:47:23.676421 | primary | 2018-06-25 19:47:23.675 |         msg)
  2018-06-25 19:47:23.678566 | primary | 2018-06-25 19:47:23.678 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
  2018-06-25 19:47:23.680280 | primary | 2018-06-25 19:47:23.679 |         self.assertThat(observed, matcher, message)
  2018-06-25 19:47:23.682290 | primary | 2018-06-25 19:47:23.681 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
  2018-06-25 19:47:23.685235 | primary | 2018-06-25 19:47:23.683 |         raise mismatch_error
  2018-06-25 19:47:23.688145 | primary | 2018-06-25 19:47:23.687 |     testtools.matchers._impl.MismatchError: !=:
  2018-06-25 19:47:23.690041 | primary | 2018-06-25 19:47:23.689 |     reference = u'ubuntu-xenial-ovh-gra1-0000332527'
  2018-06-25 19:47:23.691611 | primary | 2018-06-25 19:47:23.691 |     actual    = u'ubuntu-xenial-ovh-gra1-0000332530'
  2018-06-25 19:47:23.693251 | primary | 2018-06-25 19:47:23.692 |     : Live Migration failed. Migrations list for Instance 607cb757-f381-44b4-aff9-5cafe6d2cdfe: [
  2018-06-25 19:47:23.695192 | primary | 2018-06-25 19:47:23.694 |     {u'id': 9, u'instance_uuid': u'607cb757-f381-44b4-aff9-5cafe6d2cdfe', u'status': u'error', u'updated_at': u'2018-06-25T19:47:23.000000', u'new_instance_type_id': 11, u'old_instance_type_id': 11, u'source_node': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_host': None, u'source_compute': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_node': u'ubuntu-xenial-ovh-gra1-0000332527', u'created_at': u'2018-06-25T19:47:11.000000', u'dest_compute': u'ubuntu-xenial-ovh-gra1-0000332527'}]
  ------------

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1779270/+subscriptions
  Thread Previous • Date Previous • • Thread Next