yahoo-eng-team team mailing list archive

[OpenStack Infra <1768094@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/565264
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=b0ba58d5360bb42ed9dffa347c44dd8530ade3e0
Submitter: Zuul
Branch:    master

commit b0ba58d5360bb42ed9dffa347c44dd8530ade3e0
Author: Jorge Niedbalski <jorge.niedbalski@xxxxxxxxxx>
Date:   Mon Apr 30 14:27:29 2018 -0300

    List and map compute hosts by service instead of by compute node.
    
    This commit adds the --by-service flag in the discover_hosts
    command of the nova cell subsystem, using this flag,
    hosts are mapped based on the existing nova-compute services instead
    of using the compute node reference.
    
    In this way, we can ensure that the host mappings exists
    for this new host (and create it if not) preventing the
    HostMappingNotFound exception to loop over and over.
    
    Bug: #1768094
    Change-Id: I948004031bc3e125f929ed3e6c670aea11e73a31
    Signed-off-by: Jorge Niedbalski <jorge.niedbalski@xxxxxxxxxx>


** Changed in: kolla-ansible
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1768094

Title:
  nova_api nova-manage cell_v2 list_hosts not displaying all hypervisors

Status in kolla-ansible:
  Fix Released
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  [Environment]

  Kernel 4.14
  Queens release

  [Description]

  While doing a multinode deployment [0] ,not all the hypervisors are
  registered on the default nova cell right after the deployment has
  completed.

  The machines that fails to be enlisted are coincidentally part of
  different services (controllers/compute) groups, however, the machines
  that are exclusive part of the compute group, are enlisted correctly.

  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:45:28.516 7 INFO nova.scheduler.host_manager [req-c719e3c6-ceb0-4a39-9997-59799f785a72 - - - - -] Host mapping not found for host xxx-09. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:45:51.388 7 INFO nova.scheduler.host_manager [req-5ba217bb-f449-4a65-a875-8251191df434 - - - - -] Host mapping not found for host xxx-04. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:45:56.353 7 INFO nova.scheduler.host_manager [req-6e44a9e0-9b90-41aa-826d-65ead111a1ed - - - - -] Host mapping not found for host xxx-10. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:46:16.451 7 INFO nova.scheduler.host_manager [req-b9755302-3eca-47a8-90c3-7e2b72537956 - - - - -] Host mapping not found for host xxx-02. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:46:37.436 7 INFO nova.scheduler.host_manager [req-86348de2-ee5c-4a38-9851-caecdfd5374e - - - - -] Host mapping not found for host xxx-08. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:46:37.779 7 INFO nova.scheduler.host_manager [req-93183d20-8ac7-4c02-9d45-288939679a09 - - - - -] Host mapping not found for host xxx-03. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:47:10.784 7 INFO nova.scheduler.host_manager [req-0c008505-c0b5-4f44-8a04-569d2013d63b - - - - -] Host mapping not found for host xxx-01. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:47:18.598 7 INFO nova.scheduler.host_manager [req-7afb8034-470a-408e-b9c8-b1611df1b5de - - - - -] Host mapping not found for host xxx-07. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:47:19.196 7 INFO nova.scheduler.host_manager [req-7dfb3e0c-a734-49b3-865e-1856a7939e55 - - - - -] Host mapping not found for host xxx-05. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log:2018-04-30 14:47:21.354 7 INFO nova.scheduler.host_manager [req-6fce72dc-8716-4ac4-b2bc-cd13af812dd8 - - - - -] Host mapping not found for host xxx-06. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:17.212 7 INFO nova.scheduler.host_manager [req-9795b7c4-9ce2-4299-b198-5697eef8b3bf - - - - -] Host mapping not found for host xxx-05. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:17.356 7 INFO nova.scheduler.host_manager [req-6f01945d-9405-4364-9597-d7444a58bd91 - - - - -] Host mapping not found for host xxx-06. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:17.385 7 INFO nova.scheduler.host_manager [req-f5de3e7c-5034-4980-9897-59b56a9257ee - - - - -] Host mapping not found for host xxx-04. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:17.600 7 INFO nova.scheduler.host_manager [req-c506d111-d309-422a-b4fb-afe6620dd838 - - - - -] Host mapping not found for host xxx-07. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:17.744 7 INFO nova.scheduler.host_manager [req-7d136730-e3f4-49db-bd3d-260516eae96a - - - - -] Host mapping not found for host xxx-03. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:32.841 7 INFO nova.scheduler.host_manager [req-d227c40d-9a4b-4fbc-84ed-87e00cb01f4e - - - - -] Host mapping not found for host uk-dc-moonshot-cartridge-02. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:37.443 7 INFO nova.scheduler.host_manager [req-11730392-be04-4d32-8aef-9384da570cdf - - - - -] Host mapping not found for host xxx-08. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:37.462 7 INFO nova.scheduler.host_manager [req-11730392-be04-4d32-8aef-9384da570cdf - - - - -] Host mapping not found for host xxx-02. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:37.517 7 INFO nova.scheduler.host_manager [req-318845ef-bc1f-4479-b43e-dd7eb6068c19 - - - - -] Host mapping not found for host xxx-09. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:37.787 7 INFO nova.scheduler.host_manager [req-494c4cf8-4769-49a9-9e89-37c4ee33da93 - - - - -] Host mapping not found for host xxx-01. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:52.770 7 INFO nova.scheduler.host_manager [req-90c650a5-85d8-44a1-868c-12e6980504d4 - - - - -] Host mapping not found for host xxx-01. Not tracking instance info for this host.
  /var/log/kolla/nova/nova-scheduler.log.1:2018-04-28 04:22:52.791 7 INFO nova.scheduler.host_manager [req-90c650a5-85d8-44a1-868c-12e6980504d4 - - - - -] Host mapping not found for host xxx-04. Not tracking instance info for this host.

  For the not enlisted machines, these are the entries at the nova-api
  logs, with a reference to
  http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/host_manager.py#n745

  "It's possible to hit this when the compute service first starts
   up and casts to update_instance_info with an empty list but
   before the host is mapped in the API database."

  The problem is that even considering that theoretically this should self-heal once new instances
  arrive into this compute host, the cells doesn't return the correct information after deployment, so any automation validation/check  will fail.

  As a side point, the machines are listed when nova service-list is invoked, and
  the nova-compute service is indeed up.

  [Possible solution]

  Implement the by-service parameter
  https://github.com/openstack/nova/blob/37b4f77b5ed2c3606ebeae42ae2b1aedb0eda775/nova/objects/host_mapping.py

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1768094/+subscriptions