openstack team mailing list archive

[Sam Morrison <sorrison@xxxxxxxxx>]

Hi Joe,

On 26/02/2013, at 11:19 AM, Joe Gordon <jogo@xxxxxxxxxxxxxxxx> wrote:

> On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison <sorrison@xxxxxxxxx> wrote:
> I have been playing with the AggregateInstanceExtraSpecs filter and can't get it to work.
> 
> In our staging environment it works fine with 4 compute nodes, I have 2 aggregates to split them into 2.
> 
> When I try to do the same in our production environment which has 80 compute nodes (splitting them again into 2 aggregates) it doesn't work.
> 
> nova-scheduler starts to go very slow,  I scheduled an instance and gave up after 5 minutes, it seemed to be taking ages and the host was at 100% cpu. Also got about 500 messages in rabbit that were unacknowledged.
> 
> 
> what does the nova-scheduler log say?  Where is the unacknowledged rabbitmq messages sent from?

Logs are below. Note the large time gap between selecting a host, this is pretty much instantaneous without this filter.

Can't figure out how to see an unacknowledged message in rabbit but my guess is it is the compute service updates from all the compute nodes. These aren't happening and I think this is the reason that the attempts to schedule further down are rejected with "is disabled or has not been heard from in a while"

Do you see anything that could be an issue? Flags we use for scheduler are below also:

Thanks for your help,
Sam


# Scheduler Flags
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
ram_allocation_ratio=1.0
cpu_allocation_ratio=0.92
reserved_host_memory_mb=1024
reserved_host_disk_mb=0
scheduler_default_filters=RetryFilter,AggregateInstanceExtraSpecsFilter,RamFilter,CoreFilter,ComputeFilter
compute_fill_first_cost_fn_weight=1.0



2013-02-25 10:01:35 DEBUG nova.scheduler.filter_scheduler [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Attempting to build 1 instance(s) schedule_run_instance /usr/lib/python2.7/dist-packages/nova/sc
heduler/filter_scheduler.py:66
2013-02-25 10:01:35 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc27) host_passes /usr/lib/python2.7/dist-packages/n
ova/scheduler/filters/retry_filter.py:39
2013-02-25 10:02:13 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter passes for qh2-rcc27 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:178
2013-02-25 10:02:13 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc26) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:02:51 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method CoreFilter.host_passes of <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed for qh2-rcc26 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:02:51 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc25) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:03:28 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc25': free_ram_mb:71086 free_disk_mb:3035136 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:03:28 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc25 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:03:28 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc24) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:04:05 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc24': free_ram_mb:99758 free_disk_mb:3296256 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:04:05 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc24 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:04:05 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc23) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:04:43 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc23': free_ram_mb:71086 free_disk_mb:3012608 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:04:43 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc23 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:04:43 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc22) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:05:22 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc22': free_ram_mb:91566 free_disk_mb:3196928 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:05:22 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc22 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:05:22 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc21) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:06:01 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc21': free_ram_mb:91566 free_disk_mb:3216384 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:06:01 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc21 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:06:01 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc20) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:06:37 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method CoreFilter.host_passes of <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed for qh2-rcc20 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:06:37 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc77) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:07:14 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc77': free_ram_mb:75182 free_disk_mb:3059712 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
2013-02-25 10:07:14 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc77 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
2013-02-25 10:07:14 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: [].  (host=qh2-rcc73) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
2013-02-25 10:07:52 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc73': free_ram_mb:87470 free_disk_mb:3148800 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37