yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1921399] Re: check_instance_shared_storage RPC call is broken

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1921399@xxxxxxxxxxxxxxxxxx>
Date: Fri, 26 Mar 2021 06:14:18 -0000
Reply-to: Bug 1921399 <1921399@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Fix merged to master, and will be part of Wallaby RC1.

** Changed in: nova
       Status: In Progress => Fix Released

** Changed in: nova
    Milestone: xena-rc1 => wallaby-rc1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1921399

Title:
  check_instance_shared_storage RPC call is broken

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  We broke check_instance_shared_storage() in this change:

  https://review.opendev.org/c/openstack/nova/+/761452/13..15/nova/compute/rpcapi.py

  Where we re-ordered the rpcapi client signature without adjusting the
  caller. This leads to this failure:

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef] Traceback (most recent call last):

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/new/nova/nova/compute/manager.py", line 797, in
  _is_instance_storage_shared

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     instance, data, host=host))

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/new/nova/nova/compute/rpcapi.py", line 618, in
  check_instance_shared_storage

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return cctxt.call(ctxt,
  'check_instance_shared_storage', **msg_args)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/rpc/client.py", line 179, in call

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]
  transport_options=self.transport_options)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/transport.py", line 128, in _send

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     transport_options=transport_options)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     transport_options=transport_options)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     raise result

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef] AttributeError: 'Instance' object has no
  attribute 'filename'

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef] Traceback (most recent call last):

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     res =
  self.dispatcher.dispatch(message)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return self._do_dispatch(endpoint,
  method, ctxt, args)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     result = func(ctxt, **new_args)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/old/nova/nova/exception_wrapper.py", line 78, in wrapped

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     function_name, call_dict, binary)

  Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_utils/excutils.py", line 220, in __exit__

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     self.force_reraise()

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_utils/excutils.py", line 196, in force_reraise

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     six.reraise(self.type_, self.value,
  self.tb)

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/six.py", line 703, in reraise

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     raise value

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/old/nova/nova/exception_wrapper.py", line 69, in wrapped

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return f(self, context, *args, **kw)

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/old/nova/nova/compute/manager.py", line 7792, in
  check_instance_shared_storage

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return
  self.driver.check_instance_shared_storage_remote(ctxt, data)

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File
  "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 8550, in
  check_instance_shared_storage_remote

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return
  os.path.exists(data['filename'])

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]   File "/usr/local/lib/python3.6/dist-
  packages/oslo_versionedobjects/base.py", line 755, in __getitem__

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]     return getattr(self, name)

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef] AttributeError: 'Instance' object has no
  attribute 'filename'

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

  Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006
  nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76
  -f93c-4b3c-90a8-cd7f654b28ef]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1921399/+subscriptions
References

[Bug 1921399] [NEW] check_instance_shared_storage RPC call is broken
From: Dan Smith, 2021-03-25