yahoo-eng-team team mailing list archive

[Lars Kellogg-Stedman <lars@xxxxxxxxxx>]

Public bug reported:

We found a server that shows up in the output of "openstack server
list":

  $ openstack server list | grep 705e3bb6-27a1-4b15-a37b-62f914fc221c
  | 705e3bb6-27a1-4b15-a37b-62f914fc221c | 1mg89chr-bc4a8-sh8xn-worker-slpnh            | ERROR             |                                                                                      | 1mg89chr-bc4a8-sh8xn-rhcos                             | m1.s2.xlarge            |

But we were otherwise unable to access it:

  $ openstack server show 705e3bb6-27a1-4b15-a37b-62f914fc221c
  No server with a name or ID of '705e3bb6-27a1-4b15-a37b-62f914fc221c' exists.

Examining the nova database, there is an entry in the instances table:

  MariaDB [nova_api]> select id, uuid, vm_state, task_state from nova.instances where uuid = "705e3bb6-27a1-4b15-a37b-62f914fc221c";
  +--------+--------------------------------------+----------+------------+
  | id     | uuid                                 | vm_state | task_state |
  +--------+--------------------------------------+----------+------------+
  | 198335 | 705e3bb6-27a1-4b15-a37b-62f914fc221c | error    | NULL       |
  +--------+--------------------------------------+----------+------------+
  1 row in set (0.00 sec)

There is also an entry in the nova_api.instance_mapping table, but the
data appears to be invalid:

  MariaDB [nova_api]> select * from instance_mappings where instance_uuid="705e3bb6-27a1-4b15-a37b-62f914fc221c";
  +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
  | created_at          | updated_at | id     | instance_uuid                        | cell_id | project_id                       |
  +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
  | 2020-02-22 08:01:53 | NULL       | 211514 | 705e3bb6-27a1-4b15-a37b-62f914fc221c |    NULL | 593227d1d5d04cba8847d5b6b742e0a7 |
  +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
  
Updating the entry so that cell_id is non-null allowed things to work as expected:

  MariaDB [nova_api]> update instance_mappings set cell_id=5 where
id=211514;

And now:

  $ openstack --os-cloud kaizen-admin server show 705e3bb6-27a1-4b15-a37b-62f914fc221c -c id -c status
  +--------+--------------------------------------+
  | Field  | Value                                |
  +--------+--------------------------------------+
  | id     | 705e3bb6-27a1-4b15-a37b-62f914fc221c |
  | status | ERROR                                |
  +--------+--------------------------------------+

This is a queens environment. We are currently running nova 17.0.13, but
this issue probably cropped up before we updated to that version.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1871925

Title:
  Server appears in "openstack server list" but "openstack server
  (show|delete|etc)" insists it doesn't exist

Status in OpenStack Compute (nova):
  New

Bug description:
  We found a server that shows up in the output of "openstack server
  list":

    $ openstack server list | grep 705e3bb6-27a1-4b15-a37b-62f914fc221c
    | 705e3bb6-27a1-4b15-a37b-62f914fc221c | 1mg89chr-bc4a8-sh8xn-worker-slpnh            | ERROR             |                                                                                      | 1mg89chr-bc4a8-sh8xn-rhcos                             | m1.s2.xlarge            |

  But we were otherwise unable to access it:

    $ openstack server show 705e3bb6-27a1-4b15-a37b-62f914fc221c
    No server with a name or ID of '705e3bb6-27a1-4b15-a37b-62f914fc221c' exists.

  Examining the nova database, there is an entry in the instances table:

    MariaDB [nova_api]> select id, uuid, vm_state, task_state from nova.instances where uuid = "705e3bb6-27a1-4b15-a37b-62f914fc221c";
    +--------+--------------------------------------+----------+------------+
    | id     | uuid                                 | vm_state | task_state |
    +--------+--------------------------------------+----------+------------+
    | 198335 | 705e3bb6-27a1-4b15-a37b-62f914fc221c | error    | NULL       |
    +--------+--------------------------------------+----------+------------+
    1 row in set (0.00 sec)

  There is also an entry in the nova_api.instance_mapping table, but the
  data appears to be invalid:

    MariaDB [nova_api]> select * from instance_mappings where instance_uuid="705e3bb6-27a1-4b15-a37b-62f914fc221c";
    +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
    | created_at          | updated_at | id     | instance_uuid                        | cell_id | project_id                       |
    +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
    | 2020-02-22 08:01:53 | NULL       | 211514 | 705e3bb6-27a1-4b15-a37b-62f914fc221c |    NULL | 593227d1d5d04cba8847d5b6b742e0a7 |
    +---------------------+------------+--------+--------------------------------------+---------+----------------------------------+
    
  Updating the entry so that cell_id is non-null allowed things to work as expected:

    MariaDB [nova_api]> update instance_mappings set cell_id=5 where
  id=211514;

  And now:

    $ openstack --os-cloud kaizen-admin server show 705e3bb6-27a1-4b15-a37b-62f914fc221c -c id -c status
    +--------+--------------------------------------+
    | Field  | Value                                |
    +--------+--------------------------------------+
    | id     | 705e3bb6-27a1-4b15-a37b-62f914fc221c |
    | status | ERROR                                |
    +--------+--------------------------------------+

  This is a queens environment. We are currently running nova 17.0.13,
  but this issue probably cropped up before we updated to that version.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1871925/+subscriptions