openstack team mailing list archive

[Heng Xu <shouhengzhang.xu@xxxxxxxxxxxxxxxx>]

Hi, attached is the json_filter file I was used, but I it just came with devstack script installation, I did not even modify it.
Heng
________________________________________
From: Pengjun Pan [panpengjun@xxxxxxxxx]
Sent: Thursday, August 02, 2012 6:07 PM
To: Heng Xu
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack] Cannot pass hint to Nova Scheduler

Post your filter file. Might be a typo.

PJ

On Thu, Aug 2, 2012 at 1:02 PM, Heng Xu
<shouhengzhang.xu@xxxxxxxxxxxxxxxx> wrote:
> Hi, I recorded the error message, below
>
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp   File "/opt/stack/nova/nova/scheduler/filters/json_filter.py", line 141, in host_passes
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp     result = self._process_filter(jsonutils.loads(query), host_state)
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp   File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 123, in loads
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp     return json.loads(s)
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp     return _default_decoder.decode(s)
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp     raise ValueError("No JSON object could be decoded")
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp ValueError: No JSON object could be decoded
> 2012-08-02 13:51:02 TRACE nova.rpc.amqp
>
> it seems that the filter cannot find my json file, so although I was using the --hint functionality, whatever typed after the hint did not went to the filter host_passed function, so it could not locate the json object, any thoughts?
> Thanks. Heng
>
> ________________________________________
> From: openstack-bounces+shouhengzhang.xu=mail.utoronto.ca@xxxxxxxxxxxxxxxxxxx [openstack-bounces+shouhengzhang.xu=mail.utoronto.ca@xxxxxxxxxxxxxxxxxxx] on behalf of Heng Xu [shouhengzhang.xu@xxxxxxxxxxxxxxxx]
> Sent: Thursday, August 02, 2012 4:47 PM
> To: Pengjun Pan
> Cc: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Openstack] Cannot pass hint to Nova Scheduler
>
> Hi PJ
>
> I don't know what happen, I could not find the file in my Ubuntu filesystem, I searched for it, no result, but I just used ./stack.sh to install it, I it is just me could not find the file? Any thoughts?
> thank you
>
> Heng
> ________________________________________
> From: Pengjun Pan [panpengjun@xxxxxxxxx]
> Sent: Thursday, August 02, 2012 4:42 PM
> To: Heng Xu
> Cc: Joseph Suh; openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Openstack] Cannot pass hint to Nova Scheduler
>
> Hi Heng,
>
> The log should be in /var/log/nova/nova-scheduler.log.
>
> PJ
>
> On Thu, Aug 2, 2012 at 10:44 AM, Heng Xu
> <shouhengzhang.xu@xxxxxxxxxxxxxxxx> wrote:
>> Hello Joseph:
>> I am not sure where to find the log, so I just used the screen to n-sch,
>> and one of the error is
>> TRACE nova.rpc.amqp ValueError: No JSON object could be decoded
>> and I have no idea why this happened?
>> Thank you.
>> Heng
>>
>> ________________________________________
>> From: Joseph Suh [jsuh@xxxxxxx]
>> Sent: Thursday, August 02, 2012 3:28 PM
>> To: Heng Xu
>> Cc: openstack@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: [Openstack] Cannot pass hint to Nova Scheduler
>>
>> Heng,
>>
>> Does scheduler log show any error message or complaints?
>>
>> Thanks,
>>
>> Joseph
>>
>> ----
>> (w) 703-248-6160
>> (f) 703-812-3712
>> http://www.east.isi.edu/~jsuh
>>
>> Information Sciences Institute
>> University of Southern California
>> 3811 N. Fairfax Drive Suite 200
>> Arlington, VA, 22203, USA
>>
>>
>> ----- Original Message -----
>> From: "Heng Xu" <shouhengzhang.xu@xxxxxxxxxxxxxxxx>
>> To: openstack@xxxxxxxxxxxxxxxxxxx
>> Sent: Thursday, August 2, 2012 10:57:53 AM
>> Subject: [Openstack] Cannot pass hint to Nova Scheduler
>>
>>
>>
>> Hi folks:
>> I am new to openstack, I am current trying to test the json filter, I changed my /etc/nova/nova.conf as follow
>>
>> scheduler_driver=nova.
>> scheduler.multi.MultiScheduler
>> compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
>> volume_scheduler_driver=nova.scheduler.chance.ChanceScheduler
>> scheduler_available_filters=nova.scheduler.filters.standard_filters
>> scheduler_default_filters=JsonFilter
>> least_cost_functions=nova.scheduler.least_cost.compute_fill_first_cost_fn
>> compute_fill_first_cost_fn_weight=-1.0
>>
>> so I can use the json filter
>> however, when I was using it, if I boot a vm without any --hint to the scheduler, then the vm started fine, but if I use
>>
>> nova --debug boot --image 827d564a-e636-4fc4-a376-
>> d36f7ebe1747 --flavor 1 --hint query=['>=','$free_ram_mb',1024] server1
>>
>> my vm started with error, and the following were output from the command above
>>
>> +-----------------------------
>> --------+--------------------------------------+
>> | Property | Value |
>> +-------------------------------------+--------------------------------------+
>> | OS-DCF:diskConfig | MANUAL |
>> | OS-EXT-SRV-ATTR:host | None |
>> | OS-EXT-SRV-ATTR:hypervisor_hostname | None |
>> | OS-EXT-SRV-ATTR:instance_name | instance-0000002b |
>> | OS-EXT-STS:power_state | 0 |
>> | OS-EXT-STS:task_state | scheduling |
>> | OS-EXT-STS:vm_state | error |
>> | accessIPv4 | |
>> | accessIPv6 | |
>> | adminPass | dKvrsv4MZtfc |
>> | config_drive | |
>> | created | 2012-08-02T14:25:10Z |
>> | flavor | m1.tiny |
>> | hostId | |
>> | id | 9d4a5855-3c69-40ba-b50d-3a2aa6a92edc |
>> | image | cirros-0.3.0-x86_64-uec |
>> | key_name | |
>> | metadata | {} |
>> | name | server1 |
>> | progress | 0 |
>> | status | BUILD |
>> | tenant_id | d99ffa1b0c43455ab8dbbd81cf4380a7 |
>> | updated | 2012-08-02T14:25:10Z |
>> | user_id | d5e02f1810a44575b99a147f94507da1 |
>> +-------------------------------------+--------------------------------------+
>>
>> as you can see, the vm is in error, this also happens whenever I need to pass a hint to the scheduler, as in samehostfilter and differenthostfilter,
>>
>> Does anyone know what's going on, thanks in advance.
>> Heng
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>

# Copyright (c) 2011 OpenStack, LLC.
# All Rights Reserved.
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.


import operator

from nova.openstack.common import jsonutils
from nova.scheduler import filters


class JsonFilter(filters.BaseHostFilter):
    """Host Filter to allow simple JSON-based grammar for
    selecting hosts.
    """
    def _op_compare(self, args, op):
        """Returns True if the specified operator can successfully
        compare the first item in the args with all the rest. Will
        return False if only one item is in the list.
        """
        if len(args) < 2:
            return False
        if op is operator.contains:
            bad = not args[0] in args[1:]
        else:
            bad = [arg for arg in args[1:]
                    if not op(args[0], arg)]
        return not bool(bad)

    def _equals(self, args):
        """First term is == all the other terms."""
        return self._op_compare(args, operator.eq)

    def _less_than(self, args):
        """First term is < all the other terms."""
        return self._op_compare(args, operator.lt)

    def _greater_than(self, args):
        """First term is > all the other terms."""
        return self._op_compare(args, operator.gt)

    def _in(self, args):
        """First term is in set of remaining terms"""
        return self._op_compare(args, operator.contains)

    def _less_than_equal(self, args):
        """First term is <= all the other terms."""
        return self._op_compare(args, operator.le)

    def _greater_than_equal(self, args):
        """First term is >= all the other terms."""
        return self._op_compare(args, operator.ge)

    def _not(self, args):
        """Flip each of the arguments."""
        return [not arg for arg in args]

    def _or(self, args):
        """True if any arg is True."""
        return any(args)

    def _and(self, args):
        """True if all args are True."""
        return all(args)

    commands = {
        '=': _equals,
        '<': _less_than,
        '>': _greater_than,
        'in': _in,
        '<=': _less_than_equal,
        '>=': _greater_than_equal,
        'not': _not,
        'or': _or,
        'and': _and,
    }

    def _parse_string(self, string, host_state):
        """Strings prefixed with $ are capability lookups in the
        form '$variable' where 'variable' is an attribute in the
        HostState class.  If $variable is a dictionary, you may
        use: $variable.dictkey
        """
        if not string:
            return None
        if not string.startswith("$"):
            return string

        path = string[1:].split(".")
        obj = getattr(host_state, path[0], None)
        if obj is None:
            return None
        for item in path[1:]:
            obj = obj.get(item, None)
            if obj is None:
                return None
        return obj

    def _process_filter(self, query, host_state):
        """Recursively parse the query structure."""
        if not query:
            return True
        cmd = query[0]
        method = self.commands[cmd]
        cooked_args = []
        for arg in query[1:]:
            if isinstance(arg, list):
                arg = self._process_filter(arg, host_state)
            elif isinstance(arg, basestring):
                arg = self._parse_string(arg, host_state)
            if arg is not None:
                cooked_args.append(arg)
        result = method(self, cooked_args)
        return result

    def host_passes(self, host_state, filter_properties):
        """Return a list of hosts that can fulfill the requirements
        specified in the query.
        """
        try:
            query = filter_properties['scheduler_hints']['query']
        except KeyError:
            query = None
        if not query:
            return True

        # NOTE(comstud): Not checking capabilities or service for
        # enabled/disabled so that a provided json filter can decide

        result = self._process_filter(jsonutils.loads(query), host_state)
        if isinstance(result, list):
            # If any succeeded, include the host
            result = any(result)
        if result:
            # Filter it out.
            return True
        return False