group.of.nepali.translators team mailing list archive

Thread
Date
[Bug 1739253] Re: Removing nova-compute unit with scheduled but stopped VM breaks hypervisor-list api call

To: group.of.nepali.translators@xxxxxxxxxxxxxxxxxxx
From: Corey Bryant <corey.bryant@xxxxxxxxxxxxx>
Date: Thu, 11 Jan 2018 19:53:34 -0000
Reply-to: Bug 1739253 <1739253@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I took a look at the code for Mitaka and while the patch won't cherry-
pick cleanly a backport of the patch should be doable with some
adjustments.

** Also affects: nova (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Changed in: nova (Ubuntu Xenial)
       Status: New => Triaged

** Changed in: nova (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: nova (Ubuntu)
       Status: Triaged => Fix Released

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/mitaka
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/mitaka
       Status: New => Triaged

** Changed in: cloud-archive/mitaka
   Importance: Undecided => Medium

** Changed in: cloud-archive
       Status: New => Fix Released

** Changed in: cloud-archive
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1739253

Title:
  Removing nova-compute unit with scheduled but stopped VM breaks
  hypervisor-list api call

Status in OpenStack nova-cloud-controller charm:
  Invalid
Status in OpenStack nova-compute charm:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Triaged

Bug description:
  After removing nova-compute unit on node mycloud-cs-003, nova
  hypervisor-list stopped working and started to return below error to
  user and nova-api-os-compute.log on the nova-api server.

  While this may be an upstream issue, I believe the charm should
  probably handle this edge case.

  When querying the database, I find that the service and the
  compute_node entry for the host are both in deleted status, but I see
  that there is a scheduled vm on the node mycloud-cs-003.  I went in
  and did a nova delete <instanceid> on the instance that was scheduled
  on that node, and that succeeded, but the "running_vms" total in
  compute_nodes table did not decrease, so I updated that row to
  running_vms = 0, and I'm still experiencing the below traceback in
  nova-api-os-compute.log.

  
  2017-12-19 17:59:35.733 218705 DEBUG nova.api.openstack.wsgi [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] Calling method '<bound method HypervisorsController.index of <nova.api.openstack.compute.hypervisors.HypervisorsController object at 0x7fc679ba5d10>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] Unexpected exception in API method
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py\", line 478, in wrapped
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return f(*args, **kwargs)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/hypervisors.py\", line 88, in index
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ for hyp in compute_nodes])
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/compute/api.py\", line 3743, in service_get_by_compute_host
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return objects.Service.get_by_compute_host(context, host_name)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py\", line 181, in wrapper
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ result = fn(cls, context, *args, **kwargs)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/objects/service.py\", line 243, in get_by_compute_host
  --
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/objects/service.py\", line 238, in _db_service_get_by_compute_host
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return db.service_get_by_compute_host(context, host)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/db/api.py\", line 163, in service_get_by_compute_host
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions return IMPL.service_get_by_compute_host(context, host)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py\", line 330, in wrapped
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py\", line 585, in service_get_by_compute_host
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ raise exception.ComputeHostNotFound(host=host)
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions ComputeHostNotFound: Compute host mycloud-cs-003 could not be found.
  2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions 
  2017-12-19 17:59:36.046 218705 INFO nova.api.openstack.wsgi [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.ComputeHostNotFound'>

  Steps to recreate:

  1. deploy nova-cloud-controller and nova-compute with proper relations to keystone/mysql/etc
  2. deploy a vm to the nova-compute environment
  3. stop the instance
  4. juju remove-unit <nova-compute/X> for the unit that the VM was scheduled on
  5. nova hypervisor-list should exhibit this error.

  Please let me know if this does not work.

  Notes: this environment was previously upgraded from either icehouse
  or liberty to mitaka.  (guessing liberty since the service deleted and
  compute_node deleted columns are ordered, incrementing numbers, and
  not just 0 or 1)

  Running openstack 17.02 charms, I believe on trusty/mitaka cloud.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1739253/+subscriptions