yahoo-eng-team team mailing list archive

[melanie witt <1742508@xxxxxxxxxxxxxxxxxx>]

I looked into this with David today and it looked like a one-off problem
that resulted from some problems they encountered during a recent
upgrade from Newton to Ocata. Some database migrations were failing
during the 'db sync' and they were missing a table column, etc.

Once they made it past the upgrade, they ended up with these instances
in the nova_cell0 database that were missing entries in
nova_api.instance_mappings and were thus orphaned from the Nova API. The
nova_api.instance_mappings table stores the location (cell) of every
instance that exists. The orphaned instances are able to be shown in a
'nova list' because that API call will simply list all instances in
cell0 and cell1 for the project_id and isn't targeting a specific
instance. The commands to act upon a specific instance fail because
there is no nova_api.instance_mappings entry for the instance and when
the API tries to look up the mapping, it fails because there's no entry.

To recover, we found that we could manually insert
nova_api.instance_mappings records for the orphaned instances cell_id,
instance_uuid and project_id through mysql and then delete the instances
normally via the Nova API. This ensures that Nova takes care of cleaning
up any related records for the instance (as opposed to manually deleting
the various records of the orphaned instances). So,

Because this seems like a one-off issue and I'm not able to reproduce
the problem on a stable/ocata devstack setup (create too-large instance,
ERROR state instance goes to cell0, delete ERROR instance, all works),
I'm going to close this bug. Please re-open it if you determine that
there is a bug in the database migrations that led to the orphaned cell0
instances or otherwise have additional data we can use to investigate
this.


** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1742508

Title:
  Deleted instances are still shown in openstack server list and Horizon

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  Deleted instances are still being listed when using openstack server list or in Horizon instances dashboard.
  Any operation on the instance (delete, show) will fail with 'No server with a name or ID'
  We have more than 70 instances in this situation.
  The instance is no longer present in nova.instances and it is only referenced in 
  nova.instance_id_mappings            1 time
  nova_api.request_specs               1 time
  nova_cell0.instance_extra            1 time
  nova_cell0.instance_faults           1 time
  nova_cell0.instance_id_mappings      1 time
  nova_cell0.instance_info_caches      1 time
  nova_cell0.instance_metadata         1 time
  nova_cell0.instance_system_metadata  1 time
  nova_cell0.instances                 1 time

  Steps to reproduce
  ==================
  Not sure.
  Most of the issues might be related to operation while upgrading the overcloud and led instances to inconsistent status or due to database or migration/upgrade tasks.

  
  Expected result
  ===============
  Deleted instances to be gone

  
  Actual result
  =============
  Deleted instance is still being listed/shown. No operation can be done.
  $ openstack server list --all --format json | grep f91980fb-40e9-4f64-a90e-8701575edac1
      "ID": "f91980fb-40e9-4f64-a90e-8701575edac1", 

  $ openstack server show f91980fb-40e9-4f64-a90e-8701575edac1
  No server with a name or ID of 'f91980fb-40e9-4f64-a90e-8701575edac1' exists.

  
  Environment
  ===========
  This is a TripleO / CentOS / RDO setup.
  1. Exact version of OpenStack you are running.
  openstack-nova-scheduler-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  python2-novaclient-7.1.2-1.el7.noarch
  openstack-nova-novncproxy-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-cert-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-console-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-conductor-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-common-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-compute-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-placement-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  puppet-nova-10.4.2-0.20180102233330.f4bc1f0.el7.centos.noarch
  openstack-nova-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  python-nova-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch

  2. Which hypervisor did you use?
  Libvirt + KVM
  libvirt-daemon-driver-qemu-3.2.0-14.el7_4.7.x86_64
  libvirt-daemon-kvm-3.2.0-14.el7_4.7.x86_64
  ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
  centos-release-qemu-ev-1.0-2.el7.noarch
  qemu-kvm-ev-2.9.0-16.el7_4.13.1.x86_64
  qemu-kvm-common-ev-2.9.0-16.el7_4.13.1.x86_64
  qemu-img-ev-2.9.0-16.el7_4.13.1.x86_64

  
  2. Which storage type did you use?
  Ceph 10.2.7

  3. Which networking type did you use?
  Neutron Openvswitch

  Logs & Configs
  ==============

  nova-api.log:2018-01-10 12:34:52.252 6799 INFO nova.api.openstack.wsgi [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 12:34:52.253 6799 INFO nova.osapi_compute.wsgi.server [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.2764058
  nova-api.log:2018-01-10 12:35:01.367 6803 INFO nova.api.openstack.wsgi [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 12:35:01.368 6803 INFO nova.osapi_compute.wsgi.server [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3496580
  nova-api.log:2018-01-10 12:35:01.626 6803 INFO nova.osapi_compute.wsgi.server [req-39dd7466-584f-41df-bc8a-569a4f9851f5 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers?name=f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 200 len: 323 time: 0.0913641
  nova-api.log:2018-01-10 13:56:45.206 6810 INFO nova.api.openstack.wsgi [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 13:56:45.210 6810 INFO nova.osapi_compute.wsgi.server [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.4883139
  nova-api.log:2018-01-10 18:00:08.397 6806 INFO nova.api.openstack.wsgi [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 18:00:08.398 6806 INFO nova.osapi_compute.wsgi.server [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3506091

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1742508/+subscriptions