yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1755266] [NEW] Instance resize with swap on cinder volume fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Dr. Clemens Hardewig" <1755266@xxxxxxxxxxxxxxxxxx>
Date: Mon, 12 Mar 2018 19:13:13 -0000
Reply-to: Bug 1755266 <1755266@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Environment:

Nova on Pike and Queens with kvm/libvirt compute driver

Versions tested:
ii  nova-common                          2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - common files
ii  nova-compute                         2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node base
ii  nova-compute-kvm                     2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                 2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node libvirt support
ii  python-nova                          2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute Python libraries
ii  python-novaclient                    2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - Python 2.7
ii  python3-novaclient                   2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - 3.x

Storage on cinder and lvm backend

ii  cinder-common                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder storage service - common files
ii  cinder-volume                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder storage service - Volume server
ii  python-cinder                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder Python libraries
ii  python-cinderclient                  1:3.5.0-0ubuntu1~cloud0                     all          Python bindings to the OpenStack Volume API - Python 2.x
ii  python3-cinderclient                 1:3.5.0-0ubuntu1~cloud0                     all          Python bindings to the OpenStack Volume API - Python 3.x

Network with Neutron
ii  neutron-common                       2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - common
rc  neutron-linuxbridge-agent            2:11.0.2-0ubuntu1.1~cloud0                  all          Neutron is a virtual network service for Openstack - linuxbridge agent
ii  neutron-openvswitch-agent            2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii  python-neutron                       2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - Python library
ii  python-neutron-fwaas                 1:12.0.0-0ubuntu1~cloud0                    all          Firewall-as-a-Service driver for OpenStack Neutron
ii  python-neutron-lib                   1.13.0-0ubuntu1~cloud0                      all          Neutron shared routines and utilities - Python 2.7
ii  python-neutronclient                 1:6.7.0-0ubuntu1~cloud0                     all          client API library for Neutron - Python 2.7
ii  python3-neutronclient                1:6.7.0-0ubuntu1~cloud0                     all          client API library for Neutron - Python 3.x

Steps to reproduce:
1.) Start a nova instance with a flavor which has a swap volume
Root Volume is created as /dev/cinder-volume/volume-UUID
The swap volume is created as /dev/cinder-volume/<UUID>_disk.swap

2.) Resize the instance to a different flavor with swap storage via
horizon or CLI

Expected Result:

1.) The swap volume on Node 1 is dropped
2.) A new swap volume is created on Node 2 and the instance is migrated/restarted to node 2

Actual result:
1.) The swap volume is tried to be COPIED via scp from Node 1 to Node 2
2.) The source address is assumed to be /var/lib/nova/instances/<uuid>_disk.swap
3.) This fails with 'No such file or directory'(SIC) as swap volume is /dev/cinder-volume/<UUID>_disk.swap
4.) Resizing leaves instance in error mode 

Logfile:
 (...)
2018-03-12 17:08:51.771 15427 DEBUG os_brick.initiator.connectors.iscsi [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] <== disconnect_volume: return (678ms) None trace_logging_wrapper /usr/lib/python2.7/dist-packages/os_brick/utils.py:170
2018-03-12 17:08:51.772 15427 DEBUG nova.virt.libvirt.volume.iscsi [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Disconnected iSCSI Volume disconnect_volume /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:78
2018-03-12 17:08:51.774 15427 DEBUG nova.virt.libvirt.driver [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] skipping disk /dev/sdo (vda) as it is a volume _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7722
2018-03-12 17:08:51.774 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: request[139660391538352]: (3, 'nova.privsep.fs.blockdev_size', ('/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap',), {}) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:443
2018-03-12 17:08:51.775 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): blockdev --getsize64 /dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:51.781 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "blockdev --getsize64 /dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap" returned: 0 in 0.007s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:51.782 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: reply[139660391538352]: (4, ('2097152000\n', '')) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:456
2018-03-12 17:08:51.783 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:51.798 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize" returned: 0 in 0.015s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:51.799 15427 DEBUG nova.virt.libvirt.volume.remotefs [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Copying file /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap to 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap copy_file /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py:107
2018-03-12 17:08:51.799 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:51.965 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap" returned: 1 in 0.166s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:51.966 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] u'scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap' failed. Not Retrying. execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:457
2018-03-12 17:08:51.967 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:51.983 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.016s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:51.984 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:51.999 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.015s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:52.000 15427 DEBUG nova.virt.libvirt.volume.remotefs [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Removing directory /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 on remote host 10.0.88.31 remove_dir /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py:101
2018-03-12 17:08:52.000 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): ssh -o BatchMode=yes 10.0.88.31 rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:52.132 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "ssh -o BatchMode=yes 10.0.88.31 rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.132s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command.
Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
Exit code: 1
Stdout: u''
Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Traceback (most recent call last):
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7395, in _error_out_instance_on_exception
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     yield
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4247, in resize_instance
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     timeout, retry_interval)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8039, in migrate_disk_and_power_off
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     shared_storage)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     self.force_reraise()
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     six.reraise(self.type_, self.value, self.tb)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8024, in migrate_disk_and_power_off
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 226, in copy_image
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     on_execute=on_execute, on_completion=on_completion)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 231, in execute
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     return processutils.execute(*cmd, **kwargs)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     cmd=sanitized_cmd)
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] ProcessExecutionError: Unexpected error while running command.
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Exit code: 1
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Stdout: u''
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]
2018-03-12 17:08:52.696 15427 DEBUG oslo_concurrency.lockutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
2018-03-12 17:08:52.758 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: request[139660391538352]: (3, 'nova.privsep.fs.vginfo', ('cinder-volumes',), {}) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:443
2018-03-12 17:08:52.758 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free cinder-volumes execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
2018-03-12 17:08:52.783 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free cinder-volumes" returned: 0 in 0.024s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
2018-03-12 17:08:52.783 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: reply[139660391538352]: (4, ('  3000454086656|2075542945792\n', '')) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:456
2018-03-12 17:08:52.799 15427 DEBUG nova.compute.provider_tree [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Updating resource provider 1c1ad99d-3976-4aa8-bf32-3394f6c2040f generation from 53 to 54 _update_generation /usr/lib/python2.7/dist-packages/nova/compute/provider_tree.py:153
2018-03-12 17:08:52.800 15427 DEBUG oslo_concurrency.lockutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.104s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
2018-03-12 17:08:52.800 15427 INFO nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Successfully reverted task state from resize_migrating on failure for instance.
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
Exit code: 1
Stdout: u''
Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 186, in decorated_function
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 156, in decorated_function
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 976, in decorated_function
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 214, in decorated_function
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4247, in resize_instance
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     timeout, retry_interval)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8039, in migrate_disk_and_power_off
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     shared_storage)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8024, in migrate_disk_and_power_off
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 226, in copy_image
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 231, in execute
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Exit code: 1
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Stdout: u''
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server
2018-03-12 17:09:06.091 15427 DEBUG nova.virt.driver [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Emitting event <LifecycleEvent: 1520870931.09, 1ae5b6ee-aa77-4661-8542-58bd9da5ef82 => Stopped> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1514
2018-03-12 17:09:06.092 15427 INFO nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] VM Stopped (Lifecycle Event)

Instance need then to be destroyed and removed

Br br

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1755266

Title:
  Instance resize with swap on cinder volume fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Environment:

  Nova on Pike and Queens with kvm/libvirt compute driver

  Versions tested:
  ii  nova-common                          2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - common files
  ii  nova-compute                         2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          2:17.0.0-0ubuntu1~cloud0                    all          OpenStack Compute Python libraries
  ii  python-novaclient                    2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - Python 2.7
  ii  python3-novaclient                   2:9.1.1-0ubuntu1~cloud0                     all          client library for OpenStack Compute API - 3.x

  Storage on cinder and lvm backend

  ii  cinder-common                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder storage service - common files
  ii  cinder-volume                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder storage service - Volume server
  ii  python-cinder                        2:12.0.0-0ubuntu1~cloud0                    all          Cinder Python libraries
  ii  python-cinderclient                  1:3.5.0-0ubuntu1~cloud0                     all          Python bindings to the OpenStack Volume API - Python 2.x
  ii  python3-cinderclient                 1:3.5.0-0ubuntu1~cloud0                     all          Python bindings to the OpenStack Volume API - Python 3.x

  Network with Neutron
  ii  neutron-common                       2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - common
  rc  neutron-linuxbridge-agent            2:11.0.2-0ubuntu1.1~cloud0                  all          Neutron is a virtual network service for Openstack - linuxbridge agent
  ii  neutron-openvswitch-agent            2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
  ii  python-neutron                       2:12.0.0-0ubuntu1~cloud0                    all          Neutron is a virtual network service for Openstack - Python library
  ii  python-neutron-fwaas                 1:12.0.0-0ubuntu1~cloud0                    all          Firewall-as-a-Service driver for OpenStack Neutron
  ii  python-neutron-lib                   1.13.0-0ubuntu1~cloud0                      all          Neutron shared routines and utilities - Python 2.7
  ii  python-neutronclient                 1:6.7.0-0ubuntu1~cloud0                     all          client API library for Neutron - Python 2.7
  ii  python3-neutronclient                1:6.7.0-0ubuntu1~cloud0                     all          client API library for Neutron - Python 3.x

  Steps to reproduce:
  1.) Start a nova instance with a flavor which has a swap volume
  Root Volume is created as /dev/cinder-volume/volume-UUID
  The swap volume is created as /dev/cinder-volume/<UUID>_disk.swap

  2.) Resize the instance to a different flavor with swap storage via
  horizon or CLI

  Expected Result:

  1.) The swap volume on Node 1 is dropped
  2.) A new swap volume is created on Node 2 and the instance is migrated/restarted to node 2

  Actual result:
  1.) The swap volume is tried to be COPIED via scp from Node 1 to Node 2
  2.) The source address is assumed to be /var/lib/nova/instances/<uuid>_disk.swap
  3.) This fails with 'No such file or directory'(SIC) as swap volume is /dev/cinder-volume/<UUID>_disk.swap
  4.) Resizing leaves instance in error mode 

  Logfile:
   (...)
  2018-03-12 17:08:51.771 15427 DEBUG os_brick.initiator.connectors.iscsi [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] <== disconnect_volume: return (678ms) None trace_logging_wrapper /usr/lib/python2.7/dist-packages/os_brick/utils.py:170
  2018-03-12 17:08:51.772 15427 DEBUG nova.virt.libvirt.volume.iscsi [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Disconnected iSCSI Volume disconnect_volume /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:78
  2018-03-12 17:08:51.774 15427 DEBUG nova.virt.libvirt.driver [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] skipping disk /dev/sdo (vda) as it is a volume _get_instance_disk_info_from_config /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:7722
  2018-03-12 17:08:51.774 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: request[139660391538352]: (3, 'nova.privsep.fs.blockdev_size', ('/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap',), {}) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:443
  2018-03-12 17:08:51.775 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): blockdev --getsize64 /dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:51.781 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "blockdev --getsize64 /dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap" returned: 0 in 0.007s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:51.782 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: reply[139660391538352]: (4, ('2097152000\n', '')) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:456
  2018-03-12 17:08:51.783 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:51.798 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize" returned: 0 in 0.015s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:51.799 15427 DEBUG nova.virt.libvirt.volume.remotefs [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Copying file /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap to 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap copy_file /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py:107
  2018-03-12 17:08:51.799 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:51.965 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap" returned: 1 in 0.166s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:51.966 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] u'scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap' failed. Not Retrying. execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:457
  2018-03-12 17:08:51.967 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:51.983 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.016s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:51.984 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:51.999 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "mv /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.015s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:52.000 15427 DEBUG nova.virt.libvirt.volume.remotefs [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Removing directory /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 on remote host 10.0.88.31 remove_dir /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py:101
  2018-03-12 17:08:52.000 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): ssh -o BatchMode=yes 10.0.88.31 rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82 execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:52.132 15427 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "ssh -o BatchMode=yes 10.0.88.31 rm -rf /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82" returned: 0 in 0.132s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command.
  Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
  Exit code: 1
  Stdout: u''
  Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Traceback (most recent call last):
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7395, in _error_out_instance_on_exception
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     yield
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4247, in resize_instance
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     timeout, retry_interval)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8039, in migrate_disk_and_power_off
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     shared_storage)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     self.force_reraise()
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     six.reraise(self.type_, self.value, self.tb)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8024, in migrate_disk_and_power_off
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 226, in copy_image
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     compression=compression)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     on_execute=on_execute, on_completion=on_completion)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 231, in execute
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     return processutils.execute(*cmd, **kwargs)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]     cmd=sanitized_cmd)
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] ProcessExecutionError: Unexpected error while running command.
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Exit code: 1
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Stdout: u''
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
  2018-03-12 17:08:52.191 15427 ERROR nova.compute.manager [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82]
  2018-03-12 17:08:52.696 15427 DEBUG oslo_concurrency.lockutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
  2018-03-12 17:08:52.758 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: request[139660391538352]: (3, 'nova.privsep.fs.vginfo', ('cinder-volumes',), {}) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:443
  2018-03-12 17:08:52.758 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Running cmd (subprocess): vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free cinder-volumes execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:372
  2018-03-12 17:08:52.783 15594 DEBUG oslo_concurrency.processutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] CMD "vgs --noheadings --nosuffix --separator | --units b -o vg_size,vg_free cinder-volumes" returned: 0 in 0.024s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409
  2018-03-12 17:08:52.783 15594 DEBUG oslo.privsep.daemon [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] privsep: reply[139660391538352]: (4, ('  3000454086656|2075542945792\n', '')) loop /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:456
  2018-03-12 17:08:52.799 15427 DEBUG nova.compute.provider_tree [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Updating resource provider 1c1ad99d-3976-4aa8-bf32-3394f6c2040f generation from 53 to 54 _update_generation /usr/lib/python2.7/dist-packages/nova/compute/provider_tree.py:153
  2018-03-12 17:08:52.800 15427 DEBUG oslo_concurrency.lockutils [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.104s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
  2018-03-12 17:08:52.800 15427 INFO nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] Successfully reverted task state from resize_migrating on failure for instance.
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
  Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
  Exit code: 1
  Stdout: u''
  Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 186, in decorated_function
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 156, in decorated_function
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 976, in decorated_function
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 214, in decorated_function
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4247, in resize_instance
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     timeout, retry_interval)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8039, in migrate_disk_and_power_off
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     shared_storage)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8024, in migrate_disk_and_power_off
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 226, in copy_image
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     compression=compression)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 231, in execute
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap 10.0.88.31:/dev/cinder-volumes/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Exit code: 1
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Stdout: u''
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server Stderr: u'/var/lib/nova/instances/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_resize/1ae5b6ee-aa77-4661-8542-58bd9da5ef82_disk.swap: No such file or directory\n'
  2018-03-12 17:08:52.829 15427 ERROR oslo_messaging.rpc.server
  2018-03-12 17:09:06.091 15427 DEBUG nova.virt.driver [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] Emitting event <LifecycleEvent: 1520870931.09, 1ae5b6ee-aa77-4661-8542-58bd9da5ef82 => Stopped> emit_event /usr/lib/python2.7/dist-packages/nova/virt/driver.py:1514
  2018-03-12 17:09:06.092 15427 INFO nova.compute.manager [req-c85ef642-2708-4395-bcf4-3a3f8820a39e 9fce36209f42437bb9d4e5d4423586ae 87cbad1ec81143b5bbc557a40d81c81a - default default] [instance: 1ae5b6ee-aa77-4661-8542-58bd9da5ef82] VM Stopped (Lifecycle Event)

  Instance need then to be destroyed and removed

  Br br

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1755266/+subscriptions