← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #66535

[Bug 1709550] [NEW] nova-compute doesn't start if there is difference between current compute driver and driver which was used to create instance

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Steps to reproduce
==================
1. Create instance with (for example) qemu as nova-compute backend.
2. Change nova-compute backend to (for example) lxd.
3. Restart nova-compute service

Expected result
===============
I expected to see an error with something like this: "You have to delete all your instances, which were created with old nova-compute driver. Use 'openstack server delete instance-name' on your controller node."

Actual result
=============
nova-compute service doesn't start and there is no clear explanation in nova-compute.log (see log below).

Environment
===========
1. Version of OpenStack is Ocata:
user@compute ~> dpkg -l | grep nova
rc  nova-api                                   2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - API frontend
ii  nova-common                                2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - common files
ii  nova-compute                               2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node base
rc  nova-compute-kvm                           2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                       2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node libvirt support
ii  nova-compute-lxd                           15.0.2-0ubuntu1~cloud0                       all          Openstack Compute - LXD container hypervisor support
rc  nova-conductor                             2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - conductor service
rc  nova-consoleauth                           2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - Console Authenticator
rc  nova-novncproxy                            2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - NoVNC proxy
rc  nova-placement-api                         2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - placement API frontend
rc  nova-scheduler                             2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - virtual machine scheduler
ii  python-nova                                2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute Python libraries
ii  python-nova-lxd                            15.0.2-0ubuntu1~cloud0                       all          OpenStack Compute Python libraries - LXD driver
ii  python-novaclient                          2:7.1.0-0ubuntu1~cloud0                      all          client library for OpenStack Compute API - Python 2.7


2. Hypervisors: qemu and lxd

3. Storage: lvm

4. Networking type: Neutron with OpenVSwitch

Log
==============
nova-compute.log:
2017-08-08 16:18:51.112 29592 INFO nova.service [-] Starting compute node (version 15.0.5)
2017-08-08 16:18:51.882 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpGK77AD/privsep.sock']
2017-08-08 16:18:53.810 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Spawned new privsep daemon via rootwrap
2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep daemon starting
2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29634
2017-08-08 16:18:53.957 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:ae:ba:06,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=4a1e4eaa-17d3-4baa-a2c5-9f6a7369272c,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap4a1e4eaa-17')
2017-08-08 16:18:54.310 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:7e:a4:a1,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=ae4b3102-ad58-474c-bfbf-da6b083eda9b,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapae4b3102-ad')
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Error starting thread.
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service Traceback (most recent call last):
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     service.start()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self.manager.init_host()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1152, in init_host
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self._init_instance(context, instance)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 862, in _init_instance
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     current_power_state = self._get_power_state(context, instance)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1175, in _get_power_state
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     return self.driver.get_info(instance).state
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 306, in get_info
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     container = self.client.containers.get(instance.name)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 104, in get
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     response = client.api.containers[name].get()
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 103, in get
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self._assert_response(response, stream=kwargs.get('stream', False))
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 68, in _assert_response
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     raise exceptions.NotFound(response)
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service NotFound: not found
2017-08-08 16:18:54.408 29592 ERROR oslo_service.service

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1709550

Title:
  nova-compute doesn't start if there is difference between current
  compute driver and driver which was used to create instance

Status in OpenStack Compute (nova):
  New

Bug description:
  Steps to reproduce
  ==================
  1. Create instance with (for example) qemu as nova-compute backend.
  2. Change nova-compute backend to (for example) lxd.
  3. Restart nova-compute service

  Expected result
  ===============
  I expected to see an error with something like this: "You have to delete all your instances, which were created with old nova-compute driver. Use 'openstack server delete instance-name' on your controller node."

  Actual result
  =============
  nova-compute service doesn't start and there is no clear explanation in nova-compute.log (see log below).

  Environment
  ===========
  1. Version of OpenStack is Ocata:
  user@compute ~> dpkg -l | grep nova
  rc  nova-api                                   2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - API frontend
  ii  nova-common                                2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - common files
  ii  nova-compute                               2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node base
  rc  nova-compute-kvm                           2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                       2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute - compute node libvirt support
  ii  nova-compute-lxd                           15.0.2-0ubuntu1~cloud0                       all          Openstack Compute - LXD container hypervisor support
  rc  nova-conductor                             2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - conductor service
  rc  nova-consoleauth                           2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - Console Authenticator
  rc  nova-novncproxy                            2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - NoVNC proxy
  rc  nova-placement-api                         2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - placement API frontend
  rc  nova-scheduler                             2:15.0.2-0ubuntu1~cloud0                     all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                                2:15.0.5-0ubuntu1~cloud0                     all          OpenStack Compute Python libraries
  ii  python-nova-lxd                            15.0.2-0ubuntu1~cloud0                       all          OpenStack Compute Python libraries - LXD driver
  ii  python-novaclient                          2:7.1.0-0ubuntu1~cloud0                      all          client library for OpenStack Compute API - Python 2.7

  
  2. Hypervisors: qemu and lxd

  3. Storage: lvm

  4. Networking type: Neutron with OpenVSwitch

  Log
  ==============
  nova-compute.log:
  2017-08-08 16:18:51.112 29592 INFO nova.service [-] Starting compute node (version 15.0.5)
  2017-08-08 16:18:51.882 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpGK77AD/privsep.sock']
  2017-08-08 16:18:53.810 29592 INFO oslo.privsep.daemon [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Spawned new privsep daemon via rootwrap
  2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep daemon starting
  2017-08-08 16:18:53.812 29592 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
  2017-08-08 16:18:53.813 29592 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29634
  2017-08-08 16:18:53.957 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:ae:ba:06,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=4a1e4eaa-17d3-4baa-a2c5-9f6a7369272c,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap4a1e4eaa-17')
  2017-08-08 16:18:54.310 29592 INFO os_vif [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:7e:a4:a1,bridge_name='brq3b3f3c75-8f',has_traffic_filtering=True,id=ae4b3102-ad58-474c-bfbf-da6b083eda9b,network=Network(3b3f3c75-8f2b-4fe0-b91e-9c3dd53fe9ec),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapae4b3102-ad')
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service [req-0777ab5e-8b64-4631-8690-3dcf94ab2118 - - - - -] Error starting thread.
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service Traceback (most recent call last):
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     service.start()
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 144, in start
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self.manager.init_host()
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1152, in init_host
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self._init_instance(context, instance)
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 862, in _init_instance
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     current_power_state = self._get_power_state(context, instance)
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1175, in _get_power_state
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     return self.driver.get_info(instance).state
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 306, in get_info
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     container = self.client.containers.get(instance.name)
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 104, in get
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     response = client.api.containers[name].get()
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 103, in get
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     self._assert_response(response, stream=kwargs.get('stream', False))
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 68, in _assert_response
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service     raise exceptions.NotFound(response)
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service NotFound: not found
  2017-08-08 16:18:54.408 29592 ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1709550/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next