yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1926894] [NEW] Nova set-password error for Ubuntu VMs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: khawar <1926894@xxxxxxxxxxxxxxxxxx>
Date: Sun, 02 May 2021 15:36:21 -0000
Reply-to: Bug 1926894 <1926894@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Hi folks, i ran into a problem in OpenStack Ussuri release:
When i was trying to use `set-password` subcommand of nova CLI to reset
root password for my VM, it failed and my VM became error. This happens on Ubuntu VMs whereas Centos works fine.

I searched launchpad for similar issues, but got nothing particularly for Ubuntu. I believe this
problem may also exist in latest OpenStack distro.

Steps to reproduce
==================
* Upload any image(without QGA inside), e.g: Ubuntu18/20
* Update the image with property: hw_qemu_guest_agent=yes
  $ glance image-update --property hw_qemu_guest_agent=yes <UUID of Ubuntu>
* Boot new instance (e.g: QGA) with image Ubuntu and ensure instance is active/running.
* Use cli `nova set-password <UUID of QGA>` to reset password for the instance.

Expected result
===============
state should be CONNECTED in Nova_libvirt

e.g. virsh dumpxml <instance-number> | grep qemu.guest
<source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000000a5.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>


Actual result
=============
root@server01:~/openstack# nova set-password ubuntu
New password:
Again:
ERROR (HTTPNotImplemented): Unable to set password on instance (HTTP 501) (Request-ID: req-52278c9a-4cde-45d7-95cf-a87282a0b909)

In nova_libvirt, state is disconnected
e.g. 
<source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000000a6.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

Environment
===========
1. version: OpenStack Ussuri + Ubuntu20
2. hypervisor: Libvirt + KVM
4. networking Neutron with OpenVSwitch

Logs & Configs
==============
cat /etc/kolla/config/nova.conf
[libvirt]
inject_password=true

#### Nova CLI error #####
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 189, in decorated_function
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 159, in decorated_function
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1447, in decorated_function
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 217, in decorated_function
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 4075, in set_admin_password
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise NotImplementedError(_('set_admin_password is not '
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server NotImplementedError: set_admin_password is not implemented by this driver or guest instance.
2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1926894

Title:
  Nova set-password error for Ubuntu VMs

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Hi folks, i ran into a problem in OpenStack Ussuri release:
  When i was trying to use `set-password` subcommand of nova CLI to reset
  root password for my VM, it failed and my VM became error. This happens on Ubuntu VMs whereas Centos works fine.

  I searched launchpad for similar issues, but got nothing particularly for Ubuntu. I believe this
  problem may also exist in latest OpenStack distro.

  Steps to reproduce
  ==================
  * Upload any image(without QGA inside), e.g: Ubuntu18/20
  * Update the image with property: hw_qemu_guest_agent=yes
    $ glance image-update --property hw_qemu_guest_agent=yes <UUID of Ubuntu>
  * Boot new instance (e.g: QGA) with image Ubuntu and ensure instance is active/running.
  * Use cli `nova set-password <UUID of QGA>` to reset password for the instance.

  Expected result
  ===============
  state should be CONNECTED in Nova_libvirt

  e.g. virsh dumpxml <instance-number> | grep qemu.guest
  <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000000a5.sock'/>
        <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>

  
  Actual result
  =============
  root@server01:~/openstack# nova set-password ubuntu
  New password:
  Again:
  ERROR (HTTPNotImplemented): Unable to set password on instance (HTTP 501) (Request-ID: req-52278c9a-4cde-45d7-95cf-a87282a0b909)

  In nova_libvirt, state is disconnected
  e.g. 
  <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000000a6.sock'/>
        <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

  Environment
  ===========
  1. version: OpenStack Ussuri + Ubuntu20
  2. hypervisor: Libvirt + KVM
  4. networking Neutron with OpenVSwitch

  Logs & Configs
  ==============
  cat /etc/kolla/config/nova.conf
  [libvirt]
  inject_password=true

  #### Nova CLI error #####
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 189, in decorated_function
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 159, in decorated_function
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/utils.py", line 1447, in decorated_function
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 217, in decorated_function
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise value
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 4075, in set_admin_password
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server     raise NotImplementedError(_('set_admin_password is not '
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server NotImplementedError: set_admin_password is not implemented by this driver or guest instance.
  2021-05-02 15:31:32.680 8 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1926894/+subscriptions
Follow ups

[Bug 1926894] Re: Nova set-password error for Ubuntu VMs
From: Launchpad Bug Tracker, 2021-07-11