yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1777448] [NEW] lvm-backed libvirt ephemeral disk removal fails with TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Mon, 18 Jun 2018 13:22:17 -0000
Reply-to: Bug 1777448 <1777448@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://logs.openstack.org/10/515210/39/check/barbican-simple-crypto-
devstack-
tempest/7b7f049/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_15_18_35_04_327368

Jun 15 18:35:04.310640 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server [None req-830f8142-1f52-4eea-a72c-0a1da266fde6 tempest-ImageSigningSnapshotTest-1736125101 tempest-ImageSigningSnapshotTest-1736125101] Exception during message handling: TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)
Jun 15 18:35:04.310859 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 15 18:35:04.311069 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
Jun 15 18:35:04.311238 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jun 15 18:35:04.311429 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Jun 15 18:35:04.311591 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jun 15 18:35:04.311760 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Jun 15 18:35:04.311921 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jun 15 18:35:04.312088 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 76, in wrapped
Jun 15 18:35:04.312248 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
Jun 15 18:35:04.312408 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 18:35:04.312656 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jun 15 18:35:04.312913 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 18:35:04.313117 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jun 15 18:35:04.313280 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 67, in wrapped
Jun 15 18:35:04.313448 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Jun 15 18:35:04.313622 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 185, in decorated_function
Jun 15 18:35:04.313780 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
Jun 15 18:35:04.313954 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 18:35:04.314115 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jun 15 18:35:04.314281 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 18:35:04.314450 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jun 15 18:35:04.314623 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 155, in decorated_function
Jun 15 18:35:04.314778 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Jun 15 18:35:04.314941 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 1077, in decorated_function
Jun 15 18:35:04.315094 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Jun 15 18:35:04.315244 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 213, in decorated_function
Jun 15 18:35:04.315396 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
Jun 15 18:35:04.315556 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 18:35:04.315707 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jun 15 18:35:04.315857 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 18:35:04.316009 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jun 15 18:35:04.316159 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 201, in decorated_function
Jun 15 18:35:04.316311 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Jun 15 18:35:04.317554 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2538, in terminate_instance
Jun 15 18:35:04.317730 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     do_terminate_instance(instance, bdms)
Jun 15 18:35:04.317929 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Jun 15 18:35:04.318108 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
Jun 15 18:35:04.318271 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2536, in do_terminate_instance
Jun 15 18:35:04.318432 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
Jun 15 18:35:04.318591 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 18:35:04.318806 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jun 15 18:35:04.318979 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 18:35:04.319141 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jun 15 18:35:04.319310 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2526, in do_terminate_instance
Jun 15 18:35:04.319477 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._delete_instance(context, instance, bdms)
Jun 15 18:35:04.319636 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/hooks.py", line 154, in inner
Jun 15 18:35:04.319795 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     rv = f(*args, **kwargs)
Jun 15 18:35:04.319960 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance
Jun 15 18:35:04.320120 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._shutdown_instance(context, instance, bdms)
Jun 15 18:35:04.320287 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2361, in _shutdown_instance
Jun 15 18:35:04.320488 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     requested_networks)
Jun 15 18:35:04.321706 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 18:35:04.321949 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jun 15 18:35:04.322197 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 18:35:04.322450 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jun 15 18:35:04.322724 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2348, in _shutdown_instance
Jun 15 18:35:04.322992 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     block_device_info)
Jun 15 18:35:04.323246 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 935, in destroy
Jun 15 18:35:04.323507 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     destroy_disks)
Jun 15 18:35:04.323761 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1025, in cleanup
Jun 15 18:35:04.325962 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._cleanup_lvm(instance, block_device_info)
Jun 15 18:35:04.326238 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1110, in _cleanup_lvm
Jun 15 18:35:04.326523 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     lvm.remove_volumes(disks)
Jun 15 18:35:04.326815 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/storage/lvm.py", line 190, in remove_volumes
Jun 15 18:35:04.327095 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     brick_lvm.LVM.delete(path)
Jun 15 18:35:04.327368 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)
Jun 15 18:35:04.327643 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server 

Looks like it's due to this:

https://review.openstack.org/#/c/565703/

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20unbound%20method%20delete()%20must%20be%20called%20with%20LVM%20instance%20as%20first%20argument%20(got%20str%20instance%20instead)%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

** Affects: nova
     Importance: High
     Assignee: Matt Riedemann (mriedem)
         Status: In Progress


** Tags: libvirt lvm

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1777448

Title:
  lvm-backed libvirt ephemeral disk removal fails with TypeError:
  unbound method delete() must be called with LVM instance as first
  argument (got str instance instead)

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Seen here:

  http://logs.openstack.org/10/515210/39/check/barbican-simple-crypto-
  devstack-
  tempest/7b7f049/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_15_18_35_04_327368

  Jun 15 18:35:04.310640 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server [None req-830f8142-1f52-4eea-a72c-0a1da266fde6 tempest-ImageSigningSnapshotTest-1736125101 tempest-ImageSigningSnapshotTest-1736125101] Exception during message handling: TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)
  Jun 15 18:35:04.310859 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 15 18:35:04.311069 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jun 15 18:35:04.311238 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 15 18:35:04.311429 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  Jun 15 18:35:04.311591 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 15 18:35:04.311760 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  Jun 15 18:35:04.311921 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 15 18:35:04.312088 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 76, in wrapped
  Jun 15 18:35:04.312248 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jun 15 18:35:04.312408 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 15 18:35:04.312656 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 15 18:35:04.312913 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 15 18:35:04.313117 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 15 18:35:04.313280 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 67, in wrapped
  Jun 15 18:35:04.313448 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jun 15 18:35:04.313622 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 185, in decorated_function
  Jun 15 18:35:04.313780 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  Jun 15 18:35:04.313954 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 15 18:35:04.314115 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 15 18:35:04.314281 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 15 18:35:04.314450 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 15 18:35:04.314623 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 155, in decorated_function
  Jun 15 18:35:04.314778 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jun 15 18:35:04.314941 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 1077, in decorated_function
  Jun 15 18:35:04.315094 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jun 15 18:35:04.315244 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 213, in decorated_function
  Jun 15 18:35:04.315396 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jun 15 18:35:04.315556 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 15 18:35:04.315707 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 15 18:35:04.315857 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 15 18:35:04.316009 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 15 18:35:04.316159 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 201, in decorated_function
  Jun 15 18:35:04.316311 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jun 15 18:35:04.317554 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2538, in terminate_instance
  Jun 15 18:35:04.317730 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     do_terminate_instance(instance, bdms)
  Jun 15 18:35:04.317929 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
  Jun 15 18:35:04.318108 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Jun 15 18:35:04.318271 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2536, in do_terminate_instance
  Jun 15 18:35:04.318432 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
  Jun 15 18:35:04.318591 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 15 18:35:04.318806 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 15 18:35:04.318979 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 15 18:35:04.319141 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 15 18:35:04.319310 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2526, in do_terminate_instance
  Jun 15 18:35:04.319477 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._delete_instance(context, instance, bdms)
  Jun 15 18:35:04.319636 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/hooks.py", line 154, in inner
  Jun 15 18:35:04.319795 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     rv = f(*args, **kwargs)
  Jun 15 18:35:04.319960 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance
  Jun 15 18:35:04.320120 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._shutdown_instance(context, instance, bdms)
  Jun 15 18:35:04.320287 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2361, in _shutdown_instance
  Jun 15 18:35:04.320488 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     requested_networks)
  Jun 15 18:35:04.321706 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 15 18:35:04.321949 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 15 18:35:04.322197 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 15 18:35:04.322450 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 15 18:35:04.322724 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 2348, in _shutdown_instance
  Jun 15 18:35:04.322992 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     block_device_info)
  Jun 15 18:35:04.323246 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 935, in destroy
  Jun 15 18:35:04.323507 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     destroy_disks)
  Jun 15 18:35:04.323761 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1025, in cleanup
  Jun 15 18:35:04.325962 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     self._cleanup_lvm(instance, block_device_info)
  Jun 15 18:35:04.326238 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1110, in _cleanup_lvm
  Jun 15 18:35:04.326523 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     lvm.remove_volumes(disks)
  Jun 15 18:35:04.326815 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/storage/lvm.py", line 190, in remove_volumes
  Jun 15 18:35:04.327095 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server     brick_lvm.LVM.delete(path)
  Jun 15 18:35:04.327368 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)
  Jun 15 18:35:04.327643 ubuntu-xenial-inap-mtl01-0000165445 nova-compute[1555]: ERROR oslo_messaging.rpc.server 

  Looks like it's due to this:

  https://review.openstack.org/#/c/565703/

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20unbound%20method%20delete()%20must%20be%20called%20with%20LVM%20instance%20as%20first%20argument%20(got%20str%20instance%20instead)%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1777448/+subscriptions
Follow ups

[Bug 1777448] Re: lvm-backed libvirt ephemeral disk removal fails with TypeError: unbound method delete() must be called with LVM instance as first argument (got str instance instead)
From: OpenStack Infra, 2018-06-19