openstack team mailing list archive

[Joe Gordon <jogo@xxxxxxxxxxxxxxxx>]

On Mon, Feb 25, 2013 at 6:14 PM, Sam Morrison <sorrison@xxxxxxxxx> wrote:

> Hi Joe,
>
> On 26/02/2013, at 11:19 AM, Joe Gordon <jogo@xxxxxxxxxxxxxxxx> wrote:
>
> On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison <sorrison@xxxxxxxxx> wrote:
>
>> I have been playing with the AggregateInstanceExtraSpecs filter and can't
>> get it to work.
>>
>> In our staging environment it works fine with 4 compute nodes, I have 2
>> aggregates to split them into 2.
>>
>> When I try to do the same in our production environment which has 80
>> compute nodes (splitting them again into 2 aggregates) it doesn't work.
>>
>> nova-scheduler starts to go very slow,  I scheduled an instance and gave
>> up after 5 minutes, it seemed to be taking ages and the host was at 100%
>> cpu. Also got about 500 messages in rabbit that were unacknowledged.
>>
>>
> what does the nova-scheduler log say?  Where is the unacknowledged
> rabbitmq messages sent from?
>
>
> Logs are below. Note the large time gap between selecting a host, this is
> pretty much instantaneous without this filter.
>
> Can't figure out how to see an unacknowledged message in rabbit but my
> guess is it is the compute service updates from all the compute nodes.
> These aren't happening and I think this is the reason that the attempts to
> schedule further down are rejected with "is disabled or has not been heard
> from in a while"
>
> Do you see anything that could be an issue? Flags we use for scheduler are
> below also:
>
> Thanks for your help,
> Sam
>
>
> # Scheduler Flags
> compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
> ram_allocation_ratio=1.0
> cpu_allocation_ratio=0.92
> reserved_host_memory_mb=1024
> reserved_host_disk_mb=0
>
> scheduler_default_filters=RetryFilter,AggregateInstanceExtraSpecsFilter,RamFilter,CoreFilter,ComputeFilter
> compute_fill_first_cost_fn_weight=1.0
>
>
>
> 2013-02-25 10:01:35 DEBUG nova.scheduler.filter_scheduler
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Attempting to build 1
> instance(s) schedule_run_instance /usr/lib/python2.7/dist-packages/nova/sc
> heduler/filter_scheduler.py:66
> 2013-02-25 10:01:35 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc27) host_passes /usr/lib/python2.7/dist-packages/n
> ova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:02:13 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter passes for
> qh2-rcc27 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:178
> 2013-02-25 10:02:13 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc26) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:02:51 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method CoreFilter.host_passes of
> <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed
> for qh2-rcc26 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:02:51 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc25) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:03:28 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc25':
> free_ram_mb:71086 free_disk_mb:3035136 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
> 2013-02-25 10:03:28 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc25 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:03:28 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc24) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:04:05 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc24':
> free_ram_mb:99758 free_disk_mb:3296256 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
> 2013-02-25 10:04:05 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc24 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:04:05 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc23) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:04:43 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc23':
> free_ram_mb:71086 free_disk_mb:3012608 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
> 2013-02-25 10:04:43 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc23 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:04:43 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc22) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:05:22 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc22':
> free_ram_mb:91566 free_disk_mb:3196928 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
> 2013-02-25 10:05:22 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc22 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:05:22 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc21) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:06:01 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc21':
> free_ram_mb:91566 free_disk_mb:3216384 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
> 2013-02-25 10:06:01 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc21 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:06:01 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc20) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:06:37 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method CoreFilter.host_passes of
> <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed
> for qh2-rcc20 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:06:37 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc77) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:07:14 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc77':
> free_ram_mb:75182 free_disk_mb:3059712 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
>


It looks like the scheduler issues are related to the rabbitmq issues.
"host 'qh2-rcc77' ... is disabled or has not been heard from in a while"

What does 'nova host-list' say?   the clocks must all be synced up?


> 2013-02-25 10:07:14 DEBUG nova.scheduler.host_manager
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function
> <bound method ComputeFilter.host_passes of
> <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>>
> failed for qh2-rcc77 passes_filters
> /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175
> 2013-02-25 10:07:14 DEBUG nova.scheduler.filters.retry_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts:
> [].  (host=qh2-rcc73) host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39
> 2013-02-25 10:07:52 DEBUG nova.scheduler.filters.compute_filter
> [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc73':
> free_ram_mb:87470 free_disk_mb:3148800 is disabled or has not been heard
> from in a while host_passes
> /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37
>
>
>