yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1793747] [NEW] Fails to boot instance using Blazar flavor if compute host names are in uppercase

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Neha Alhat <1793747@xxxxxxxxxxxxxxxxxx>
Date: Fri, 21 Sep 2018 12:11:59 -0000
Reply-to: Bug 1793747 <1793747@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Bug Description:
================

Steps to reproduce:
====================

$ nova hypervisor-list
+--------------------------------------+----------------------+--------------------+----------------------+
| ID                                   | Hypervisor hostname  | State              | Status               |
+--------------------------------------+----------------------+--------------------+----------------------+
| e517e75b-d57c-45b2-af41-6be2fed536c6 | Openstack-VirtualBox | up                 | enabled              |
+--------------------------------------+----------------------+--------------------+----------------------+

Step 1: Add host into freepool
$ blazar host-create Openstack-VirtualBox

$ blazar host-list
+----+---------------------+-------+-----------+----------+
| id | hypervisor_hostname | vcpus | memory_mb | local_gb |          
+----+---------------------+-------+-----------+----------+
| 1  | Openstack-VirtualBox|     4 |     11941 |       91 |             
+----+---------------------+-------+-----------+----------+


Step 2: Create a lease
$ blazar lease-create --reservation resource_type=virtual:instance,vcpus=1,memory_mb=1024,disk_gb=20,amount=1,affinity=False --start-date "2018-08-27 12:59" --end-date "2018-08-27 13:55" lease-1

nova_api database entries related to aggregates:

Blazar creates aggregate with id=18
mysql> select * from aggregates;
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| created_at          | updated_at | id | uuid                                 | name                                 |
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| 2018-08-13 06:49:37 | NULL       |  1 | 2a7d838f-4e42-48af-a9a9-faf3f29e3c96 | freepool                             |
| 2018-08-29 13:43:15 | NULL       | 18 | 88c37cc5-373a-4da5-820f-508d25f00903 | 8c85522a-cc39-4a0d-a5ca-4de92e4d2c1f |
+---------------------+------------+----+--------------------------------------+--------------------------------------+

Blazar adds aggregate with aggregate_id=18, to host at the time of
lease-start event.

mysql> select * from aggregate_hosts;
+---------------------+------------+----+-----------------------+--------------+
| created_at          | updated_at | id | host                  | aggregate_id |
+---------------------+------------+----+-----------------------+--------------+
| 2018-08-29 13:34:46 | NULL       | 32 | Openstack-VirtualBox  |            1 |
| 2018-08-29 13:39:05 | NULL       | 34 | Openstack-VirtualBox  |           18 |
+---------------------+------------+----+-----------------------+--------------+

Step 3: Create a server: Please specify the flavor of the reservation and group_id as a scheduler hint.
$ openstack server create --flavor 03067174-2a5e-43f7-baf7-037aac23b4ef --image cirros-0.4.0-x86_64-disk --network 42d6f419-b445-40a6-b542-e8a502c6ae64 --hint group=09389292-6639-48d6-9709-045061f42ebf instance-1

For more details regarding instance reservation please refer:
https://docs.openstack.org/blazar/latest/cli/instance-reservation.html

Logs
=====

Service logs of n-sch:
=======================
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: DEBUG nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: [('RetryFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AvailabilityZoneFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeCapabilitiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ImagePropertiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAntiAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('SameHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('DifferentHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AggregateInstanceExtraSpecsFilter', None)] {{(pid=22198) get_filtered_objects /opt/stack/nova/nova/filters.py:129}}
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'SameHostFilter: (start: 1, end: 1)', 'DifferentHostFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']

Service logs of n-super-cond:
===============================
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: Traceback (most recent call last):
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     return func(*args, **kwargs)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 169, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 256, in _schedule
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     claimed_instance_uuids)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 293, in _ensure_sufficient_hosts
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     raise exception.NoValidHost(reason=reason)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: NoValidHost: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] [instance: 7e5aef57-fec6-4e25-8219-0e01aab4b9f3] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.


'AggregateInstanceExtraSpecsFilter' returns 0 hosts because at[1] host_state.aggregates is not returning aggregate with metadata created by Blazar. Lately, this patch [2] was merged in which HostManager._get_aggregates_info method, the host is converted into lower case and then it finds aggregates for that host in host_aggregates_map. But if the compute host name is in uppercase, it doesn’t add the host in host_aggregates_map in lowercase in _update_aggregates method hence _get_aggregates_info method doesn’t return the aggregate associated with the requested host.


Expected Result: Instance should be booted on the host "Openstack-VirtualBox"
Actual Result: NoValidHost_Remote: No valid host was found


[1] : https://github.com/openstack/nova/blob/master/nova/scheduler/filters/utils.py#L38
[2] : https://review.openstack.org/#/c/498334

** Affects: nova
     Importance: Undecided
     Assignee: Neha Alhat (nehaalhat)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Neha Alhat (nehaalhat)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1793747

Title:
  Fails to boot instance using Blazar flavor if compute host names are
  in uppercase

Status in OpenStack Compute (nova):
  New

Bug description:
  Bug Description:
  ================

  Steps to reproduce:
  ====================

  $ nova hypervisor-list
  +--------------------------------------+----------------------+--------------------+----------------------+
  | ID                                   | Hypervisor hostname  | State              | Status               |
  +--------------------------------------+----------------------+--------------------+----------------------+
  | e517e75b-d57c-45b2-af41-6be2fed536c6 | Openstack-VirtualBox | up                 | enabled              |
  +--------------------------------------+----------------------+--------------------+----------------------+

  Step 1: Add host into freepool
  $ blazar host-create Openstack-VirtualBox

  $ blazar host-list
  +----+---------------------+-------+-----------+----------+
  | id | hypervisor_hostname | vcpus | memory_mb | local_gb |          
  +----+---------------------+-------+-----------+----------+
  | 1  | Openstack-VirtualBox|     4 |     11941 |       91 |             
  +----+---------------------+-------+-----------+----------+

  
  Step 2: Create a lease
  $ blazar lease-create --reservation resource_type=virtual:instance,vcpus=1,memory_mb=1024,disk_gb=20,amount=1,affinity=False --start-date "2018-08-27 12:59" --end-date "2018-08-27 13:55" lease-1

  nova_api database entries related to aggregates:

  Blazar creates aggregate with id=18
  mysql> select * from aggregates;
  +---------------------+------------+----+--------------------------------------+--------------------------------------+
  | created_at          | updated_at | id | uuid                                 | name                                 |
  +---------------------+------------+----+--------------------------------------+--------------------------------------+
  | 2018-08-13 06:49:37 | NULL       |  1 | 2a7d838f-4e42-48af-a9a9-faf3f29e3c96 | freepool                             |
  | 2018-08-29 13:43:15 | NULL       | 18 | 88c37cc5-373a-4da5-820f-508d25f00903 | 8c85522a-cc39-4a0d-a5ca-4de92e4d2c1f |
  +---------------------+------------+----+--------------------------------------+--------------------------------------+

  Blazar adds aggregate with aggregate_id=18, to host at the time of
  lease-start event.

  mysql> select * from aggregate_hosts;
  +---------------------+------------+----+-----------------------+--------------+
  | created_at          | updated_at | id | host                  | aggregate_id |
  +---------------------+------------+----+-----------------------+--------------+
  | 2018-08-29 13:34:46 | NULL       | 32 | Openstack-VirtualBox  |            1 |
  | 2018-08-29 13:39:05 | NULL       | 34 | Openstack-VirtualBox  |           18 |
  +---------------------+------------+----+-----------------------+--------------+

  Step 3: Create a server: Please specify the flavor of the reservation and group_id as a scheduler hint.
  $ openstack server create --flavor 03067174-2a5e-43f7-baf7-037aac23b4ef --image cirros-0.4.0-x86_64-disk --network 42d6f419-b445-40a6-b542-e8a502c6ae64 --hint group=09389292-6639-48d6-9709-045061f42ebf instance-1

  For more details regarding instance reservation please refer:
  https://docs.openstack.org/blazar/latest/cli/instance-reservation.html

  Logs
  =====

  Service logs of n-sch:
  =======================
  Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: DEBUG nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: [('RetryFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AvailabilityZoneFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeCapabilitiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ImagePropertiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAntiAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('SameHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('DifferentHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AggregateInstanceExtraSpecsFilter', None)] {{(pid=22198) get_filtered_objects /opt/stack/nova/nova/filters.py:129}}
  Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'SameHostFilter: (start: 1, end: 1)', 'DifferentHostFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']

  Service logs of n-super-cond:
  ===============================
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: Traceback (most recent call last):
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     return func(*args, **kwargs)
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 169, in select_destinations
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     allocation_request_version, return_alternates)
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     allocation_request_version, return_alternates)
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 256, in _schedule
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     claimed_instance_uuids)
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 293, in _ensure_sufficient_hosts
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]:     raise exception.NoValidHost(reason=reason)
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: NoValidHost: No valid host was found. There are not enough hosts available.
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
  Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] [instance: 7e5aef57-fec6-4e25-8219-0e01aab4b9f3] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

  
  'AggregateInstanceExtraSpecsFilter' returns 0 hosts because at[1] host_state.aggregates is not returning aggregate with metadata created by Blazar. Lately, this patch [2] was merged in which HostManager._get_aggregates_info method, the host is converted into lower case and then it finds aggregates for that host in host_aggregates_map. But if the compute host name is in uppercase, it doesn’t add the host in host_aggregates_map in lowercase in _update_aggregates method hence _get_aggregates_info method doesn’t return the aggregate associated with the requested host.

  
  Expected Result: Instance should be booted on the host "Openstack-VirtualBox"
  Actual Result: NoValidHost_Remote: No valid host was found

  
  [1] : https://github.com/openstack/nova/blob/master/nova/scheduler/filters/utils.py#L38
  [2] : https://review.openstack.org/#/c/498334

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1793747/+subscriptions
Follow ups

[Bug 1793747] Re: Fails to boot instance using Blazar flavor if compute host names are in uppercase
From: OpenStack Infra, 2018-09-27
[Bug 1793747] Re: Fails to boot instance using Blazar flavor if compute host names are in uppercase
From: Matt Riedemann, 2018-09-25
[Bug 1793747] Re: Fails to boot instance using Blazar flavor if compute host names are in uppercase
From: Matt Riedemann, 2018-09-24