yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #89827
[Bug 1992328] [NEW] volume timeouts in nova gate
Public bug reported:
I'm trying to track here a bug I have seen in nova gate appearing
randomly through rechecks.
Typical stack traces:
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_volume_swap.py", line 110, in test_volume_swap
volume1['id'], 'available')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume a19743a3-4651-4c7f-a9a1-823735ea84a0 failed to reach available status (current in-use) within the required time (196 s).
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 190, in test_live_block_migration_with_attached_volume
self.attach_volume(server, volume, device='/dev/xvdb')
File "/opt/stack/tempest/tempest/api/compute/base.py", line 581, in attach_volume
volume['id'], 'in-use')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 92685b8f-4db0-4110-a1ac-016ea7c51d1f failed to reach in-use status (current available) within the required time (196 s).
Typical jobs and tests:
nova-multi-cell
test_volume_swap[id-1769f00d-a693-4d67-a631-6a3496773813]
nova-live-migration
test_live_block_migration_with_attached_volume[id-e19c0cc6-6720-4ed8-be83-b6603ed5c812]
Example hits with (affecting multiple branches):
$ logsearch log --project openstack/nova --job nova-live-migration --result FAILURE --limit 50 "test_live_block_migration_with_attached_volume .* ... FAILED"
...
Builds with matching logs 10/50:
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| uuid | finished | pipeline | review | branch |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| 36b367b0d0bb46d2a7fc6af4eb7739ca | 2022-10-07T19:39:47 | check | https://review.opendev.org/860736 | stable/victoria |
| d02ed047fcfd4180902dc0bec0334c38 | 2022-10-03T10:37:00 | check | https://review.opendev.org/854980 | stable/victoria |
| 0df9b00df16c4bbc9e49baf853fe0cf5 | 2022-09-19T09:47:02 | check | https://review.opendev.org/854980 | stable/victoria |
| 0db0e8d510d04443a172cc43e537f973 | 2022-09-16T14:14:31 | check | https://review.opendev.org/857877 | stable/train |
| 6ca30836a1b34be58728dc5d69c44c21 | 2022-09-16T10:33:55 | check | https://review.opendev.org/858051 | stable/victoria |
| 684e7c37c61745829908495ba249afb7 | 2022-09-16T10:14:07 | check | https://review.opendev.org/854980 | stable/victoria |
| 6bcf4105d0fc476faf9ee56e7f0ed41f | 2022-09-15T14:22:01 | check | https://review.opendev.org/857877 | stable/train |
| 0ea47624757c48a8bcfa9fd5c35b6465 | 2022-09-13T10:33:52 | check | https://review.opendev.org/854980 | stable/victoria |
| ca0d5f750b3040ed99c1e6ec3414d154 | 2022-09-06T17:28:41 | check | https://review.opendev.org/836830 | master |
| 2ce6d7aa67404587b050a6b56f4d15e6 | 2022-08-29T11:58:59 | check | https://review.opendev.org/833090 | master |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
** Affects: nova
Importance: Undecided
Status: New
** Tags: gate-failure
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1992328
Title:
volume timeouts in nova gate
Status in OpenStack Compute (nova):
New
Bug description:
I'm trying to track here a bug I have seen in nova gate appearing
randomly through rechecks.
Typical stack traces:
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_volume_swap.py", line 110, in test_volume_swap
volume1['id'], 'available')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume a19743a3-4651-4c7f-a9a1-823735ea84a0 failed to reach available status (current in-use) within the required time (196 s).
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 190, in test_live_block_migration_with_attached_volume
self.attach_volume(server, volume, device='/dev/xvdb')
File "/opt/stack/tempest/tempest/api/compute/base.py", line 581, in attach_volume
volume['id'], 'in-use')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 92685b8f-4db0-4110-a1ac-016ea7c51d1f failed to reach in-use status (current available) within the required time (196 s).
Typical jobs and tests:
nova-multi-cell
test_volume_swap[id-1769f00d-a693-4d67-a631-6a3496773813]
nova-live-migration
test_live_block_migration_with_attached_volume[id-e19c0cc6-6720-4ed8-be83-b6603ed5c812]
Example hits with (affecting multiple branches):
$ logsearch log --project openstack/nova --job nova-live-migration --result FAILURE --limit 50 "test_live_block_migration_with_attached_volume .* ... FAILED"
...
Builds with matching logs 10/50:
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| uuid | finished | pipeline | review | branch |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| 36b367b0d0bb46d2a7fc6af4eb7739ca | 2022-10-07T19:39:47 | check | https://review.opendev.org/860736 | stable/victoria |
| d02ed047fcfd4180902dc0bec0334c38 | 2022-10-03T10:37:00 | check | https://review.opendev.org/854980 | stable/victoria |
| 0df9b00df16c4bbc9e49baf853fe0cf5 | 2022-09-19T09:47:02 | check | https://review.opendev.org/854980 | stable/victoria |
| 0db0e8d510d04443a172cc43e537f973 | 2022-09-16T14:14:31 | check | https://review.opendev.org/857877 | stable/train |
| 6ca30836a1b34be58728dc5d69c44c21 | 2022-09-16T10:33:55 | check | https://review.opendev.org/858051 | stable/victoria |
| 684e7c37c61745829908495ba249afb7 | 2022-09-16T10:14:07 | check | https://review.opendev.org/854980 | stable/victoria |
| 6bcf4105d0fc476faf9ee56e7f0ed41f | 2022-09-15T14:22:01 | check | https://review.opendev.org/857877 | stable/train |
| 0ea47624757c48a8bcfa9fd5c35b6465 | 2022-09-13T10:33:52 | check | https://review.opendev.org/854980 | stable/victoria |
| ca0d5f750b3040ed99c1e6ec3414d154 | 2022-09-06T17:28:41 | check | https://review.opendev.org/836830 | master |
| 2ce6d7aa67404587b050a6b56f4d15e6 | 2022-08-29T11:58:59 | check | https://review.opendev.org/833090 | master |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1992328/+subscriptions
