← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #85138

[Bug 1915441] [NEW] Octavia amphora VMs fail to migrate (scp fails for disk.config file)

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Octavia Amphora VM migration fails on newly deployed Octavia install

Non amphora VMs seem to migrate with any issues.

The relevant error reported from nova-compute is

2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Command: scp -r HostOne.nonprod.maas:/var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3/disk.config /var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3
2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Exit code: 1
2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stdout: ''
2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stderr: 'Host key verification failed.\r\n'

Steps to reproduce.
  One an Openstack Stein cloud with Octavia installed. Simply deploy a load balancer and try and migrate that instance to another node.

VERSIONS Used

dpkg -l | grep nova
ii  nova-common                           2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - common files
ii  nova-compute                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node base
ii  nova-compute-kvm                      2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                  2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node libvirt support
ii  python3-nova                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute Python 3 libraries
ii  python3-novaclient                    2:13.0.0-0ubuntu1~cloud0                        all          client library for OpenStack Compute API - 3.x

Work around

I can login to the target machine and do this to get it to work
ubuntu@HostTwo:~$ sudo -u nova scp -r HostOne.pnp.maas:/var/lib/nova/instances/7c92fefb-61a7-41e0-a224-eb52a6a24f12/disk.config  test.config
The authenticity of host 'cmooschstupOne.pnp.maas (10.55.33.148)' can't be established.
ECDSA key fingerprint is SHA256:XXXXXXXXX
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostOne.pnp.maas,10.55.33.148' (ECDSA) to the list of known hosts.
 
    and now it works fine

There is a somewhat unique execution path here. That disk.config is the
meta data that would otherwise probably be served to other "regular" vms
via the meta data service

We do not see that scp action for vms that are not amphora on our cloud,
nor do we see that file under /var/lib/nova/instances/INSTANCE for
instances other then amphora.

Thanks,
  Steven

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915441

Title:
  Octavia amphora VMs fail to migrate (scp fails for disk.config file)

Status in OpenStack Compute (nova):
  New

Bug description:
  Octavia Amphora VM migration fails on newly deployed Octavia install

  Non amphora VMs seem to migrate with any issues.

  The relevant error reported from nova-compute is

  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Command: scp -r HostOne.nonprod.maas:/var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3/disk.config /var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Exit code: 1
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stdout: ''
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stderr: 'Host key verification failed.\r\n'

  Steps to reproduce.
    One an Openstack Stein cloud with Octavia installed. Simply deploy a load balancer and try and migrate that instance to another node.

  VERSIONS Used

  dpkg -l | grep nova
  ii  nova-common                           2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - common files
  ii  nova-compute                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                      2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                  2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node libvirt support
  ii  python3-nova                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                    2:13.0.0-0ubuntu1~cloud0                        all          client library for OpenStack Compute API - 3.x

  Work around

  I can login to the target machine and do this to get it to work
  ubuntu@HostTwo:~$ sudo -u nova scp -r HostOne.pnp.maas:/var/lib/nova/instances/7c92fefb-61a7-41e0-a224-eb52a6a24f12/disk.config  test.config
  The authenticity of host 'cmooschstupOne.pnp.maas (10.55.33.148)' can't be established.
  ECDSA key fingerprint is SHA256:XXXXXXXXX
  Are you sure you want to continue connecting (yes/no)? yes
  Warning: Permanently added 'hostOne.pnp.maas,10.55.33.148' (ECDSA) to the list of known hosts.
   
      and now it works fine

  There is a somewhat unique execution path here. That disk.config is
  the meta data that would otherwise probably be served to other
  "regular" vms via the meta data service

  We do not see that scp action for vms that are not amphora on our
  cloud, nor do we see that file under /var/lib/nova/instances/INSTANCE
  for instances other then amphora.

  Thanks,
    Steven

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915441/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next