yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1742508] [NEW] Deleted instances are still shown in openstack server list and Horizon

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: David Manchado <dmanchad@xxxxxxxxxx>
Date: Wed, 10 Jan 2018 18:10:51 -0000
Reply-to: Bug 1742508 <1742508@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Deleted instances are still being listed when using openstack server list or in Horizon instances dashboard.
Any operation on the instance (delete, show) will fail with 'No server with a name or ID'
We have more than 70 instances in this situation.
The instance is no longer present in nova.instances and it is only referenced in 
nova.instance_id_mappings            1 time
nova_api.request_specs               1 time
nova_cell0.instance_extra            1 time
nova_cell0.instance_faults           1 time
nova_cell0.instance_id_mappings      1 time
nova_cell0.instance_info_caches      1 time
nova_cell0.instance_metadata         1 time
nova_cell0.instance_system_metadata  1 time
nova_cell0.instances                 1 time

Steps to reproduce
==================
Not sure.
Most of the issues might be related to operation while upgrading the overcloud and led instances to inconsistent status or due to database or migration/upgrade tasks.


Expected result
===============
Deleted instances to be gone


Actual result
=============
Deleted instance is still being listed/shown. No operation can be done.
$ openstack server list --all --format json | grep f91980fb-40e9-4f64-a90e-8701575edac1
    "ID": "f91980fb-40e9-4f64-a90e-8701575edac1", 

$ openstack server show f91980fb-40e9-4f64-a90e-8701575edac1
No server with a name or ID of 'f91980fb-40e9-4f64-a90e-8701575edac1' exists.


Environment
===========
This is a TripleO / CentOS / RDO setup.
1. Exact version of OpenStack you are running.
openstack-nova-scheduler-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
python2-novaclient-7.1.2-1.el7.noarch
openstack-nova-novncproxy-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-cert-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-console-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-conductor-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-common-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-compute-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
openstack-nova-placement-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
puppet-nova-10.4.2-0.20180102233330.f4bc1f0.el7.centos.noarch
openstack-nova-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
python-nova-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch

2. Which hypervisor did you use?
Libvirt + KVM
libvirt-daemon-driver-qemu-3.2.0-14.el7_4.7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7_4.7.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
centos-release-qemu-ev-1.0-2.el7.noarch
qemu-kvm-ev-2.9.0-16.el7_4.13.1.x86_64
qemu-kvm-common-ev-2.9.0-16.el7_4.13.1.x86_64
qemu-img-ev-2.9.0-16.el7_4.13.1.x86_64


2. Which storage type did you use?
Ceph 10.2.7

3. Which networking type did you use?
Neutron Openvswitch

Logs & Configs
==============

nova-api.log:2018-01-10 12:34:52.252 6799 INFO nova.api.openstack.wsgi [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
nova-api.log:2018-01-10 12:34:52.253 6799 INFO nova.osapi_compute.wsgi.server [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.2764058
nova-api.log:2018-01-10 12:35:01.367 6803 INFO nova.api.openstack.wsgi [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
nova-api.log:2018-01-10 12:35:01.368 6803 INFO nova.osapi_compute.wsgi.server [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3496580
nova-api.log:2018-01-10 12:35:01.626 6803 INFO nova.osapi_compute.wsgi.server [req-39dd7466-584f-41df-bc8a-569a4f9851f5 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers?name=f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 200 len: 323 time: 0.0913641
nova-api.log:2018-01-10 13:56:45.206 6810 INFO nova.api.openstack.wsgi [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
nova-api.log:2018-01-10 13:56:45.210 6810 INFO nova.osapi_compute.wsgi.server [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.4883139
nova-api.log:2018-01-10 18:00:08.397 6806 INFO nova.api.openstack.wsgi [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
nova-api.log:2018-01-10 18:00:08.398 6806 INFO nova.osapi_compute.wsgi.server [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3506091

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1742508

Title:
  Deleted instances are still shown in openstack server list and Horizon

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Deleted instances are still being listed when using openstack server list or in Horizon instances dashboard.
  Any operation on the instance (delete, show) will fail with 'No server with a name or ID'
  We have more than 70 instances in this situation.
  The instance is no longer present in nova.instances and it is only referenced in 
  nova.instance_id_mappings            1 time
  nova_api.request_specs               1 time
  nova_cell0.instance_extra            1 time
  nova_cell0.instance_faults           1 time
  nova_cell0.instance_id_mappings      1 time
  nova_cell0.instance_info_caches      1 time
  nova_cell0.instance_metadata         1 time
  nova_cell0.instance_system_metadata  1 time
  nova_cell0.instances                 1 time

  Steps to reproduce
  ==================
  Not sure.
  Most of the issues might be related to operation while upgrading the overcloud and led instances to inconsistent status or due to database or migration/upgrade tasks.

  
  Expected result
  ===============
  Deleted instances to be gone

  
  Actual result
  =============
  Deleted instance is still being listed/shown. No operation can be done.
  $ openstack server list --all --format json | grep f91980fb-40e9-4f64-a90e-8701575edac1
      "ID": "f91980fb-40e9-4f64-a90e-8701575edac1", 

  $ openstack server show f91980fb-40e9-4f64-a90e-8701575edac1
  No server with a name or ID of 'f91980fb-40e9-4f64-a90e-8701575edac1' exists.

  
  Environment
  ===========
  This is a TripleO / CentOS / RDO setup.
  1. Exact version of OpenStack you are running.
  openstack-nova-scheduler-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  python2-novaclient-7.1.2-1.el7.noarch
  openstack-nova-novncproxy-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-cert-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-console-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-conductor-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-common-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-compute-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  openstack-nova-placement-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  puppet-nova-10.4.2-0.20180102233330.f4bc1f0.el7.centos.noarch
  openstack-nova-api-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch
  python-nova-15.1.1-0.20180103153502.ff2231f.el7.centos.noarch

  2. Which hypervisor did you use?
  Libvirt + KVM
  libvirt-daemon-driver-qemu-3.2.0-14.el7_4.7.x86_64
  libvirt-daemon-kvm-3.2.0-14.el7_4.7.x86_64
  ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
  centos-release-qemu-ev-1.0-2.el7.noarch
  qemu-kvm-ev-2.9.0-16.el7_4.13.1.x86_64
  qemu-kvm-common-ev-2.9.0-16.el7_4.13.1.x86_64
  qemu-img-ev-2.9.0-16.el7_4.13.1.x86_64

  
  2. Which storage type did you use?
  Ceph 10.2.7

  3. Which networking type did you use?
  Neutron Openvswitch

  Logs & Configs
  ==============

  nova-api.log:2018-01-10 12:34:52.252 6799 INFO nova.api.openstack.wsgi [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 12:34:52.253 6799 INFO nova.osapi_compute.wsgi.server [req-3a17767a-6f3b-43c9-b6bc-4672fc19a9c3 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.2764058
  nova-api.log:2018-01-10 12:35:01.367 6803 INFO nova.api.openstack.wsgi [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 12:35:01.368 6803 INFO nova.osapi_compute.wsgi.server [req-c5c7d5cd-4b9c-416a-83c8-069e2c92f188 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3496580
  nova-api.log:2018-01-10 12:35:01.626 6803 INFO nova.osapi_compute.wsgi.server [req-39dd7466-584f-41df-bc8a-569a4f9851f5 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers?name=f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 200 len: 323 time: 0.0913641
  nova-api.log:2018-01-10 13:56:45.206 6810 INFO nova.api.openstack.wsgi [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 13:56:45.210 6810 INFO nova.osapi_compute.wsgi.server [req-0667f674-653d-43f1-9712-fa8c5b20343c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.4883139
  nova-api.log:2018-01-10 18:00:08.397 6806 INFO nova.api.openstack.wsgi [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] HTTP exception thrown: Instance f91980fb-40e9-4f64-a90e-8701575edac1 could not be found.
  nova-api.log:2018-01-10 18:00:08.398 6806 INFO nova.osapi_compute.wsgi.server [req-ec17a76d-51d8-4e41-b300-633244033c6c 1ded2d1b92794bf5b362d76fa2fcee69 8dbc460fe5fd4fcab0096c2c0aad3ece - default default] XXX.XXX.XXX.XXX "GET /v2.1/servers/f91980fb-40e9-4f64-a90e-8701575edac1 HTTP/1.1" status: 404 len: 442 time: 0.3506091

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1742508/+subscriptions
Follow ups

[Bug 1742508] Re: Deleted instances are still shown in openstack server list and Horizon
From: Launchpad Bug Tracker, 2023-10-21
[Bug 1742508] Re: Deleted instances are still shown in openstack server list and Horizon
From: melanie witt, 2018-01-16