savanna-all team mailing list archive

[Jon Maron <jmaron@xxxxxxxxxxxxxxx>]

On Jun 5, 2013, at 9:11 AM, Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx> wrote:

> Philip,
> 
> AFAIU, John is speaking of 'cluster/cluster_configs' and 'cluster/node_groups/node_configs' fields. They contain only configs defined by plugin's get_configs(...) call. So actually plugin could set any of them.
> 
> John,
> 
> Agree, plugin needs ability to set these fields during convert(...) call. Another thing we want to discuss is the structure of these fields. After an internal discussion we see it as follows:
> {config_applicable_target: {config_name: value}}
> For example:
> {"service:mapreduce": {"task_tracker_heap_size": "512", "mapred.task.timeout": "10000"}, "service:hdfs": .....}
> 
> Here applicable_target and config_name serve as config 'coordinates'. In that way configs structure becomes lightweight and easy-readable. Obviously it will be easy for plugin to populate such fields.
> 
> Note that there is no reference to the original 'config' object here. Right now we use a single object to both persist and provide config value via SPI. If we add a reference to 'config' object, we'll need to introduce unnecessary crunches to avoid persisting these references. On the other side, having applicable_target/config_name plugin can uniquely identify the config and hence easily find related config object and required information.

I understand your desire for a lightweight and readable structure.  In an ideal world the mapping of properties to associated services would be readily available and clear.  However, in the current setup, most properties are not clearly associated to services (or even processes/components), but rather are mapped to specific configuration files (e.g. hdfs-site.xml).  So I would propose that rather than having a somewhat contrived mapping like above, we at least do the following:

{"mapred-site":  {"task_tracker_heap_size": "512", "mapred.task.timeout": "10000"}, "service:hdfs": .....}

The only modification would be to include the "file" or source as an attribute in the config object and then have the controller create the appropriate mapping.

I recognize that we could maintain the mapping as you've laid it out, but that needlessly complicates the work of the plugin - the logic it would have to implements to map properties to their appropriate destination is needlessly made more complex by not relating information that could be readily available at the time the config object is being created and returned by the plugin (i.e. the plugin is forced to do work at config interpretation time that it already did at config creation time to determine where the property belongs).

Again, I understand the motivation for displaying the associated target for a property and feel that it is a  useful feature.  However, I simply don't see the need to needlessly complicate the processing in the plugin by having it determine AGAIN where the property belongs.

> 
> Thanks,
> 
> Dmitry
> 
> 
> 
> 2013/6/4 Philip Langdale <philipl@xxxxxxxxxxxx>
> My understanding was that this would be questionable exercise. It's certain that the native configuration will be a strict superset of what Savanna is capable of expressing, so it's not going to be possible to provide a complete Savanna representation of it. Essentially, you're opting out of Savanna handling the configuration by going this route, with the advantage that you get the full capability of the underlying platform. You'd presumably use native tools to inspect and do ongoing configuration in this model.
> 
> --phil
> 
> 
> On 3 June 2013 13:54, John Speidel <jspeidel@xxxxxxxxxxxxxxx> wrote:
> I am looking at the doc:
> https://wiki.openstack.org/wiki/Savanna/PluggableProvisioning/ClusterLifecycle
> 
> I am confused as to why the hadoop plugin will not set cluster/cluster_configs during the convert call.  Considering that the purpose of this call is to normalize a hadoop plugin configuration to a cluster template, I would expect to set the configuration information on the passed in cluster instance. According to the above doc, cluster_configs is never modifiable by the plugin.  Is this a mistake in the doc or do I completely misunderstand the purpose of the convert function?
> 
> Thanks,
> John
> 
> -- 
> Mailing list: https://launchpad.net/~savanna-all
> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~savanna-all
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> --
> Mailing list: https://launchpad.net/~savanna-all
> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~savanna-all
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> -- 
> Mailing list: https://launchpad.net/~savanna-all
> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~savanna-all
> More help   : https://help.launchpad.net/ListHelp