savanna-all team mailing list archive

[Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx>]

Yes, that is it Jon


2013/5/20 Jon Maron <jmaron@xxxxxxxxxxxxxxx>

>
> On May 17, 2013, at 12:20 PM, Dmitry Mescheryakov <
> dmescheryakov@xxxxxxxxxxxx> wrote:
>
> Yes, we'll do it. One thing - we expect plugin to return in
> applicable_targets field name of the actual processes like "dn", "tt".
> Savanna will use this field in templates to filter relevant configs.
>
> Since it is plugin which returns config objects in get_configs(...) call,
> you can insert arbitrary field into the object to save filename there.
> Controller will not persist config objects and the plugin will receive back
> the same object instance it created.
>
>
> To be clear:  you're suggesting that we include additional attributes in
> the returned config object rather than additional items in the list of
> applicable_targets.  Correct?
>
>
>
> 2013/5/17 Jon Maron <jmaron@xxxxxxxxxxxxxxx>
>
>>
>> On May 16, 2013, at 12:09 PM, Jon Maron <jmaron@xxxxxxxxxxxxxxx> wrote:
>>
>>
>> On May 16, 2013, at 12:05 PM, Dmitry Mescheryakov <
>> dmescheryakov@xxxxxxxxxxxx> wrote:
>>
>> Jon,
>>
>> In the user_input object we can put reference to the related 'config'
>> object instead of just config_name. I think that should be fine, right?
>>
>>
>> Yes
>>
>>
>> Are you going to proceed with this modification?  Also, I would still
>> like to leverage the "applicable_targets" field of the Config objects to
>> indicate the target destination config file (at least as one of the
>> indicators in the list).  Are there any objections?  the utility of that
>> field seems to be plugin specific.
>>
>>
>>
>> Thanks,
>>
>> Dmitry
>>
>>
>> 2013/5/16 Jon Maron <jmaron@xxxxxxxxxxxxxxx>
>>
>>> Looks like I overlooked the new "user_input" object defined in the
>>> latest docs…
>>>
>>> It appears that now "config" is an object returned by the plugin, while
>>> "user_input" is the value specified by the user.  So…
>>>
>>> In order to assist the plugin with understanding the given input, could
>>> we add the "target" attribute to the user input?  While I realize that it
>>> would be possible for the plugin to maintain a mapping of properties to
>>> configuration files, adding this attribute will streamline the process of
>>> extending the infrastructure to support new properties ( modification in
>>> get_config() alone rather than having to ensure that the config returned is
>>> properly mapped to a config file during user input processing).  It simply
>>> is a convenience that simplifies the plugins processing burden and doesn't
>>> cost much from the perspective of the controller or UI.
>>>
>>> -- Jon
>>>
>>> On May 16, 2013, at 10:33 AM, Jon Maron <jmaron@xxxxxxxxxxxxxxx> wrote:
>>>
>>>
>>> On May 16, 2013, at 10:26 AM, "Alexander Ignatov" <aignatov@xxxxxxxxxxxx>
>>> wrote:
>>>
>>> Jon,****
>>>
>>> Configs grouping by node processes is more convenient from user
>>> perspective.****
>>> It is more likely user wants to override some conf property for certain
>>> process (DataNode, TaskTracker, etc) rather than some. xml, .sh or some
>>> other type of files.****
>>> By the way the same picture is on the Ambari and Cloudera UIs. I mean
>>> grouping by processes.
>>>
>>>
>>> That is true.  However, Ambari has discovered that by grouping config in
>>> this manner things became pretty complicated and resulted in huge
>>> numbers of web service invocations once the number of hosts increased in a
>>> cluster.  They are therefore actively moving away from such a design.
>>>
>>> ****
>>> Each plugin can store its own information about configuration parameters
>>> and theirs file destination.****
>>>
>>> It seems 'applicable_node_processes' is not correct name. Because this
>>> attribute of Config object is not only applicable for hadoop node processes.
>>> ****
>>> It can be as general property of the whole cluster like you mentioned in
>>> the your second concern. Also this attribute can describe node OS property
>>> like ulimits, ssh configs etc.****
>>> I think we should rename 'applicable_node_processes' to 'target' or just
>>> 'destination' where destination could be node process, node OS specific
>>> property or general cluster property.
>>>
>>>
>>> I gather that you are proposing that this setting is essentially opaque
>>> to the controller, since:
>>>
>>> 1)  The config object is an object returned by the plugin and displayed
>>> by the UI via some interface
>>> 2)  The user should not modify this setting
>>> 3)  This setting is subsequently interpreted by the plugin during
>>> cluster configuration and creation.
>>>
>>> If that is correct, can the plugin use this "target" attribute in a
>>> manner they feel is appropriate to their implementation?  In other words,
>>> couldn't we simply specify the target XML file if we deemed that as an
>>> appropriate use of the attribute?
>>>
>>> ****
>>>
>>> Regards,****
>>> Alexander Ignatov****
>>>
>>> *From:* Savanna-all [mailto:savanna-
>>> all-bounces+aignatov=mirantis.com@xxxxxxxxxxxxxxxxxxx] *On Behalf Of *Jon
>>> Maron
>>> *Sent:* Thursday, May 16, 2013 1:24 AM
>>> *To:* savanna-all@xxxxxxxxxxxxxxxxxxx
>>> *Subject:* [Savanna-all] Questions concerning config object****
>>> ** **
>>> The current Savanna documentation proposes a Config object with the
>>> following attributes:****
>>> ** **
>>>             name****
>>>             description****
>>>             type****
>>>             default_value****
>>>             is_optional****
>>>             *applicable_node_processes *****
>>> ** **
>>>   Node processes seem to correlate to Hadoop components.  ****
>>> ** **
>>> * *I see a number of problems with this proposal:****
>>> ** **
>>>  1)  The proposal makes an assumption that properties are grouped by
>>> node processes/components.  Although some properties are clearly dedicated
>>> to certain processes, it appears that for the most part properties are
>>> associated with, and grouped by, specific site configuration files.  As a
>>> matter of fact, there is some effort in Ambari around decoupling services
>>> and configuration.****
>>>  2)  There are some general properties that aren't necessarily dedicated
>>> to a specific process but are rather more general in nature.  In those
>>> cases it seems that an indicator specifying which configuration file the
>>> property resides in is more appropriate.****
>>> ** **
>>>  It just seems like the categorization by node process (or component) is
>>> somewhat artificial in the Hadoop environment.  Rather, it seems like it's
>>> be more natural to have the following structure:****
>>>  ** **
>>>             name****
>>>             description****
>>>             type****
>>>             default_value****
>>>             is_optional****
>>>             *destination_file*****
>>> *           *****
>>> *  *I welcome your thoughts on the matter.  Thanks!****
>>> ** **
>>> -- Jon****
>>>
>>>
>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~savanna-all
>>> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~savanna-all
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>>
>
>