← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76823

[Bug 1814177] [NEW] nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSample.test_volume_swap_server_with_error intermittently fails with IndexError

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Seen here:

http://logs.openstack.org/86/633086/1/gate/nova-tox-functional/8a2c597
/job-output.txt.gz#_2019-01-31_20_37_18_577225

2019-01-31 20:37:18.641058 | ubuntu-xenial |     2019-01-31 20:37:17,911 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb
2019-01-31 20:37:18.641191 | ubuntu-xenial |     Traceback (most recent call last):
2019-01-31 20:37:18.641370 | ubuntu-xenial |       File "nova/compute/manager.py", line 5710, in _swap_volume
2019-01-31 20:37:18.641508 | ubuntu-xenial |         bdm, new_attachment_id, mountpoint)
2019-01-31 20:37:18.641707 | ubuntu-xenial |       File "nova/compute/manager.py", line 5697, in _init_volume_connection
2019-01-31 20:37:18.641850 | ubuntu-xenial |         new_cinfo['serial'] = new_volume_id
2019-01-31 20:37:18.642025 | ubuntu-xenial |     TypeError: 'tuple' object does not support item assignment
2019-01-31 20:37:18.642275 | ubuntu-xenial |     2019-01-31 20:37:18,369 ERROR [oslo_messaging.rpc.server] Exception during message handling
2019-01-31 20:37:18.642402 | ubuntu-xenial |     Traceback (most recent call last):
2019-01-31 20:37:18.642783 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-01-31 20:37:18.642931 | ubuntu-xenial |         res = self.dispatcher.dispatch(message)
2019-01-31 20:37:18.643307 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-01-31 20:37:18.643490 | ubuntu-xenial |         return self._do_dispatch(endpoint, method, ctxt, args)
2019-01-31 20:37:18.643874 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-01-31 20:37:18.644007 | ubuntu-xenial |         result = func(ctxt, **new_args)
2019-01-31 20:37:18.644175 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 79, in wrapped
2019-01-31 20:37:18.644314 | ubuntu-xenial |         function_name, call_dict, binary, tb)
2019-01-31 20:37:18.644739 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-31 20:37:18.644850 | ubuntu-xenial |         self.force_reraise()
2019-01-31 20:37:18.645209 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-31 20:37:18.645365 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2019-01-31 20:37:18.645532 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 69, in wrapped
2019-01-31 20:37:18.645668 | ubuntu-xenial |         return f(self, context, *args, **kw)
2019-01-31 20:37:18.645864 | ubuntu-xenial |       File "nova/compute/utils.py", line 1141, in decorated_function
2019-01-31 20:37:18.646051 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
2019-01-31 20:37:18.646263 | ubuntu-xenial |       File "nova/compute/manager.py", line 216, in decorated_function
2019-01-31 20:37:18.646406 | ubuntu-xenial |         kwargs['instance'], e, sys.exc_info())
2019-01-31 20:37:18.646755 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-31 20:37:18.646864 | ubuntu-xenial |         self.force_reraise()
2019-01-31 20:37:18.647224 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-31 20:37:18.647379 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2019-01-31 20:37:18.647564 | ubuntu-xenial |       File "nova/compute/manager.py", line 204, in decorated_function
2019-01-31 20:37:18.647721 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
2019-01-31 20:37:18.647894 | ubuntu-xenial |       File "nova/compute/manager.py", line 5868, in swap_volume
2019-01-31 20:37:18.647997 | ubuntu-xenial |         is_cinder_migration)
2019-01-31 20:37:18.648170 | ubuntu-xenial |       File "nova/compute/manager.py", line 5766, in _swap_volume
2019-01-31 20:37:18.648288 | ubuntu-xenial |         context, new_attachment_id)
2019-01-31 20:37:18.648676 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-31 20:37:18.648784 | ubuntu-xenial |         self.force_reraise()
2019-01-31 20:37:18.649144 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-31 20:37:18.649298 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
2019-01-31 20:37:18.649473 | ubuntu-xenial |       File "nova/compute/manager.py", line 5710, in _swap_volume
2019-01-31 20:37:18.649606 | ubuntu-xenial |         bdm, new_attachment_id, mountpoint)
2019-01-31 20:37:18.649809 | ubuntu-xenial |       File "nova/compute/manager.py", line 5697, in _init_volume_connection
2019-01-31 20:37:18.649966 | ubuntu-xenial |         new_cinfo['serial'] = new_volume_id
2019-01-31 20:37:18.650149 | ubuntu-xenial |     TypeError: 'tuple' object does not support item assignment
2019-01-31 20:37:18.650208 | ubuntu-xenial |
2019-01-31 20:37:18.650254 | ubuntu-xenial |
2019-01-31 20:37:18.650337 | ubuntu-xenial | Captured traceback:
2019-01-31 20:37:18.650423 | ubuntu-xenial | ~~~~~~~~~~~~~~~~~~~
2019-01-31 20:37:18.650545 | ubuntu-xenial |     Traceback (most recent call last):
2019-01-31 20:37:18.650846 | ubuntu-xenial |       File "nova/tests/functional/notification_sample_tests/test_instance.py", line 1443, in test_volume_swap_server_with_error
2019-01-31 20:37:18.651028 | ubuntu-xenial |         tb2 = fake_notifier.VERSIONED_NOTIFICATIONS[7]['payload'][
2019-01-31 20:37:18.651153 | ubuntu-xenial |     IndexError: list index out of range

The TypeError is expected, it's part of the swap.volume.error
notification test. The bug is the IndexError in the expected
notifications. We should probably debug that with a patch to check the
length and dump the notifications if the length isn't what we expect.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tb2%20%3D%20fake_notifier.VERSIONED_NOTIFICATIONS%5B7%5D%5B'payload'%5D%5B%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=10d

6 hits in 10 days, check and gate, all failures.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1814177

Title:
  nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSample.test_volume_swap_server_with_error
  intermittently fails with IndexError

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/86/633086/1/gate/nova-tox-functional/8a2c597
  /job-output.txt.gz#_2019-01-31_20_37_18_577225

  2019-01-31 20:37:18.641058 | ubuntu-xenial |     2019-01-31 20:37:17,911 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb
  2019-01-31 20:37:18.641191 | ubuntu-xenial |     Traceback (most recent call last):
  2019-01-31 20:37:18.641370 | ubuntu-xenial |       File "nova/compute/manager.py", line 5710, in _swap_volume
  2019-01-31 20:37:18.641508 | ubuntu-xenial |         bdm, new_attachment_id, mountpoint)
  2019-01-31 20:37:18.641707 | ubuntu-xenial |       File "nova/compute/manager.py", line 5697, in _init_volume_connection
  2019-01-31 20:37:18.641850 | ubuntu-xenial |         new_cinfo['serial'] = new_volume_id
  2019-01-31 20:37:18.642025 | ubuntu-xenial |     TypeError: 'tuple' object does not support item assignment
  2019-01-31 20:37:18.642275 | ubuntu-xenial |     2019-01-31 20:37:18,369 ERROR [oslo_messaging.rpc.server] Exception during message handling
  2019-01-31 20:37:18.642402 | ubuntu-xenial |     Traceback (most recent call last):
  2019-01-31 20:37:18.642783 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2019-01-31 20:37:18.642931 | ubuntu-xenial |         res = self.dispatcher.dispatch(message)
  2019-01-31 20:37:18.643307 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2019-01-31 20:37:18.643490 | ubuntu-xenial |         return self._do_dispatch(endpoint, method, ctxt, args)
  2019-01-31 20:37:18.643874 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2019-01-31 20:37:18.644007 | ubuntu-xenial |         result = func(ctxt, **new_args)
  2019-01-31 20:37:18.644175 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 79, in wrapped
  2019-01-31 20:37:18.644314 | ubuntu-xenial |         function_name, call_dict, binary, tb)
  2019-01-31 20:37:18.644739 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-01-31 20:37:18.644850 | ubuntu-xenial |         self.force_reraise()
  2019-01-31 20:37:18.645209 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-01-31 20:37:18.645365 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2019-01-31 20:37:18.645532 | ubuntu-xenial |       File "nova/exception_wrapper.py", line 69, in wrapped
  2019-01-31 20:37:18.645668 | ubuntu-xenial |         return f(self, context, *args, **kw)
  2019-01-31 20:37:18.645864 | ubuntu-xenial |       File "nova/compute/utils.py", line 1141, in decorated_function
  2019-01-31 20:37:18.646051 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
  2019-01-31 20:37:18.646263 | ubuntu-xenial |       File "nova/compute/manager.py", line 216, in decorated_function
  2019-01-31 20:37:18.646406 | ubuntu-xenial |         kwargs['instance'], e, sys.exc_info())
  2019-01-31 20:37:18.646755 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-01-31 20:37:18.646864 | ubuntu-xenial |         self.force_reraise()
  2019-01-31 20:37:18.647224 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-01-31 20:37:18.647379 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2019-01-31 20:37:18.647564 | ubuntu-xenial |       File "nova/compute/manager.py", line 204, in decorated_function
  2019-01-31 20:37:18.647721 | ubuntu-xenial |         return function(self, context, *args, **kwargs)
  2019-01-31 20:37:18.647894 | ubuntu-xenial |       File "nova/compute/manager.py", line 5868, in swap_volume
  2019-01-31 20:37:18.647997 | ubuntu-xenial |         is_cinder_migration)
  2019-01-31 20:37:18.648170 | ubuntu-xenial |       File "nova/compute/manager.py", line 5766, in _swap_volume
  2019-01-31 20:37:18.648288 | ubuntu-xenial |         context, new_attachment_id)
  2019-01-31 20:37:18.648676 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-01-31 20:37:18.648784 | ubuntu-xenial |         self.force_reraise()
  2019-01-31 20:37:18.649144 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-01-31 20:37:18.649298 | ubuntu-xenial |         six.reraise(self.type_, self.value, self.tb)
  2019-01-31 20:37:18.649473 | ubuntu-xenial |       File "nova/compute/manager.py", line 5710, in _swap_volume
  2019-01-31 20:37:18.649606 | ubuntu-xenial |         bdm, new_attachment_id, mountpoint)
  2019-01-31 20:37:18.649809 | ubuntu-xenial |       File "nova/compute/manager.py", line 5697, in _init_volume_connection
  2019-01-31 20:37:18.649966 | ubuntu-xenial |         new_cinfo['serial'] = new_volume_id
  2019-01-31 20:37:18.650149 | ubuntu-xenial |     TypeError: 'tuple' object does not support item assignment
  2019-01-31 20:37:18.650208 | ubuntu-xenial |
  2019-01-31 20:37:18.650254 | ubuntu-xenial |
  2019-01-31 20:37:18.650337 | ubuntu-xenial | Captured traceback:
  2019-01-31 20:37:18.650423 | ubuntu-xenial | ~~~~~~~~~~~~~~~~~~~
  2019-01-31 20:37:18.650545 | ubuntu-xenial |     Traceback (most recent call last):
  2019-01-31 20:37:18.650846 | ubuntu-xenial |       File "nova/tests/functional/notification_sample_tests/test_instance.py", line 1443, in test_volume_swap_server_with_error
  2019-01-31 20:37:18.651028 | ubuntu-xenial |         tb2 = fake_notifier.VERSIONED_NOTIFICATIONS[7]['payload'][
  2019-01-31 20:37:18.651153 | ubuntu-xenial |     IndexError: list index out of range

  The TypeError is expected, it's part of the swap.volume.error
  notification test. The bug is the IndexError in the expected
  notifications. We should probably debug that with a patch to check the
  length and dump the notifications if the length isn't what we expect.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tb2%20%3D%20fake_notifier.VERSIONED_NOTIFICATIONS%5B7%5D%5B'payload'%5D%5B%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=10d

  6 hits in 10 days, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1814177/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next