yahoo-eng-team team mailing list archive

[Balazs Gibizer <1915441@xxxxxxxxxxxxxxxxxx>]

I think this is an deployment issue. It seem that the ssh keys are
properly distributed to the compute hosts but the .ssh/known_hosts files
are not populated so the ssh connection between compute hosts (for scp)
fails.

when you manually connect the host you also get the warning message from
the ssh

The authenticity of host 'cmooschstupOne.pnp.maas (10.55.33.148)' can't be established.
ECDSA key fingerprint is SHA256:XXXXXXXXX
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostOne.pnp.maas,10.55.33.148' (ECDSA) to the list of known hosts.

This is the same condition that prevented nova to scp to this host. But
when you run ssh in an interactive session the ssh client can ask you to
confirm the host identity. You answered yes and therefore you was able
to log in. Also when you answered yes your ssh client updated the
known_hosts file resolving the original issue.

If I understand correctly you used maas to deploy openstack. I think
updating the known_hosts files are the responsibility of the deployment
tool.  So I'm marking this as Invalid please report the bug to your
deployment tool.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915441

Title:
  Octavia amphora VMs fail to migrate (scp fails for disk.config file)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Octavia Amphora VM migration fails on newly deployed Octavia install

  Non amphora VMs seem to migrate with any issues.

  The relevant error reported from nova-compute is

  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Command: scp -r HostOne.nonprod.maas:/var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3/disk.config /var/lib/nova/instances/e3967a91-3f5d-45b3-9300-28f8ee936fa3
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Exit code: 1
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stdout: ''
  2021-02-11 18:46:16.194 898529 ERROR nova.compute.manager [instance: e3967a91-3f5d-45b3-9300-28f8ee936fa3] Stderr: 'Host key verification failed.\r\n'

  Steps to reproduce.
    One an Openstack Stein cloud with Octavia installed. Simply deploy a load balancer and try and migrate that instance to another node.

  VERSIONS Used

  dpkg -l | grep nova
  ii  nova-common                           2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - common files
  ii  nova-compute                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                      2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                  2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute - compute node libvirt support
  ii  python3-nova                          2:19.1.0-0ubuntu1~cloud0                        all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                    2:13.0.0-0ubuntu1~cloud0                        all          client library for OpenStack Compute API - 3.x

  Work around

  I can login to the target machine and do this to get it to work
  ubuntu@HostTwo:~$ sudo -u nova scp -r HostOne.pnp.maas:/var/lib/nova/instances/7c92fefb-61a7-41e0-a224-eb52a6a24f12/disk.config  test.config
  The authenticity of host 'cmooschstupOne.pnp.maas (10.55.33.148)' can't be established.
  ECDSA key fingerprint is SHA256:XXXXXXXXX
  Are you sure you want to continue connecting (yes/no)? yes
  Warning: Permanently added 'hostOne.pnp.maas,10.55.33.148' (ECDSA) to the list of known hosts.
   
      and now it works fine

  There is a somewhat unique execution path here. That disk.config is
  the meta data that would otherwise probably be served to other
  "regular" vms via the meta data service

  We do not see that scp action for vms that are not amphora on our
  cloud, nor do we see that file under /var/lib/nova/instances/INSTANCE
  for instances other then amphora.

  Thanks,
    Steven

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915441/+subscriptions