← Back to team overview

savanna-all team mailing list archive

Re: Configuration approach recommendations

 

On May 22, 2013, at 9:52 AM, Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx> wrote:

> Hello Jon,
> 
> We considered using services instead of processes and found that they also have disadvantage. The problem with the services approach is in the UI. Consider the following example:
> 
> User configures node group to run MapReduce workers, i.e. TaskTrackers processes. It does not matter if he creates a Node Group Template, or edits Node Group during cluster creation, since the UI will be similar in both cases. Since properties are categorized by services, user is asked to fill in all the parameters for that service (MapReduce) including those for JobTracker like “JobTracker heap size”, etc. As a result user has to dig through many irrelevant options during configuration, which is very bad. I think we should not blindly copy files configuration even if users got used to it. We are making a web console and we should use the advantages it provides over editing files. I.e. if we can filter out irrelevant options for user, then why not do it? That does not change configuration flow much, but at the same time it is much more convenient for a user.

In the case above, it is true that we would return all service related configuration properties.  However, we do specify the default values, and we may be able to specify components as well for the purposes of enhancing the UI.  The user would only have to modify the values for which the default does not makes sense.  The bottom line is that the user is presented with a single set of properties for a service because doing otherwise introduces race conditions and uncertainty with respect to which property value is actually used.  Let me illustrate again with the name node and secondary name node which likely are both deployed on the same instance.  In that case the property choices for each process/component are obviously the same.  If a user decided to vary the values for the same properties based on the two separate component property selection panels, which property value are we expected to actually use?  Remember, Hadoop does not configure components.  These values end up in node or service based configuration files, so the selections from both the name node and secondary name node ultimately end up in the same configuration file.  Which value selection are expected to select?  Again, this is not a contrived example.  More generally, component based configuration is simply not a configuration scheme with which Hadoop users are familiar.  We are lucky enough to have multiple developers of Hadoop working in the company, and every single one we've spoken to has questioned the component based configuration of Savanna.

> 
> As for Templates, our intention is to provide all users with ability to create their own templates, both for Cluster and Node Group. In fact we see no reason to reject a user have his own templates. We just _think_ that users will prefer administrators or more experienced users to prepare templates for them, because these people should be better at Hadoop cluster tweaking.
> 
> Right now the main concern we have is timing. We already spent much time designing phase 2 and I believe that our initial design evolved into something much better. But we think that it is about time to freeze specs we have for phase 2 and start implementing them. At the same time we can have a background conversation on how we can improve the design in phase 3. We believe that it will not be hard to change this part specifically.
> 
> The solution for the problem we see is to unify processes and services categorization. Config object can have the following 2-dimensional “coordinates”:
> applicable_target = <process name> | <service name> | “general”
> scope = “node” | “cluster”
> where
> <process name> = “process:tasktracker” | “process:jobtracker” | “process:datanode” | …
> <service name> = “service:mapreduce” | “service:hdfs” | …
> 
> Here is a table example parameters for various combinations of target/scope:
> 
> 
> Cluster
> Node
> Process
> Don’t use this combination, use Service/Cluster instead
> JobTracker heap size, mapred.tasktracker.map.tasks.maximum
> 
> Service
> dfs.replication, mapred.output.compression.type
> ?
> General
> user SSL key for cluster machines
> OS parameters like ulimits
> 
> 
> 
> Again, as I said we propose to do this only after we complete Pluggable Provisioning Mechanism and make sure it is working. Right now we suggest to implement our old proposal with with processes, just to avoid further changes in design in this phase.

This approach is acceptable with some modification:

1)  applicable target should be modified to a list of targets, allowing for the specification of service/component or general, e.g

applicable_targets = ["service:mapreduce", "process:job tracker"]

or 

applicable_targets = ["general"]

2)  The scope attribute is probably unnecessary since the scope is implied by where the user_input is specified (It's not required for the returned config objects).  When attached to a node_group, the scope is "node".  When attached to the cluster object directly, the scope is "cluster".

Also, I don't think this can wait till the next phase and I don't think it affects your development much for the following reasons:

- the current form of the config and user_input objects remains unchanged.  We are just changing some of the values
- for the time being we can try to return the associated process as part of the applicable targets list so that you can at least provide a hint on the service configuration page (and I don't believe the UIs work has begun anyhow)

But from our perspective this will greatly enhance our capability of implementing the plugin since the bottom line is that we need to associate properties to services/node.  Attempting the artificial grouping of the properties per component/process complicates our ability to property structure the configuration of the Hadoop cluster.

> 
> Thanks,
> 
> Dmitry
> 
> 
> 2013/5/22 Jon Maron <jmaron@xxxxxxxxxxxxxxx>
> We still have some concerns regarding the current configuration approach.  I'd like to highlight two major issues:
> 
> 1)  Component level configuration - Configuring at the component level is contrary to the Hadoop configuration approach which is structured around host level configuration (as opposed to component level).  Trying to configure at the component level runs contrary to that configuration philosophy and would likely be questioned by Hadoop users.  In addition, this approach can be rather error prone.  For example, consider a deployment in which the name node and secondary name node are hosted on the same server (a not uncommon approach).  Both components obviously share a great deal of configuration properties.  If a user is leveraging the UI and is configuring these items at the component level he/she will:
> 
> 	- repeat the configuration for each component, a process that may be rather frustrating
> 	- will have no idea which of the settings will actually be leveraged since there is essentially a race condition here - the properties potentially end up in the same configuration file on the given node group, so which of the properties actually win?
> 
> 2)  There doesn't appear to be a facility for making changes that span node groups.  The cluster templates are essentially immutable - they are created by an admin and are not seen as modifiable via the UI by users (at least as far as we can tell).  The other configuration alternative currently is to configure the specific node groups.  So, for example, how do I approach the task of modifying 3 HDFS properties across all 10 node groups I've defined?  It seems that with the current approach I will have to repeatedly make the same modifications to each node group in turn?
> 
> We believe both of these issues can be remedied by modifying the configuration approach to be more service centric rather than component centric.  While the cluster template still provides for global, immutable settings, allowing users to configure at the service level will allow for global changes across node groups.  We still want to address the node group level configuration (host level overrides), so perhaps the config structure could be redesigned as follows:
> 
> config
> 	Describes a single config parameter.
> 	name
> 	description
> 	type
> 		Type could be string, integer, enum, array of [int, string]
> 	default_value
> 	is_optional
> 	service - a service name or "GENERAL"
> 
> To be clear about what the service attribute values mean:
> 
> service name - this property value is a service-based property that is valid at to be applied to the service across a cluster or to a specific node group (i.e. the property can be applied to all instances of the service across the cluster or to a specific set of hosts in a node group).  The scope is determined by where the user selected the property value (the node group interface or the cluster interface) and specified in the user_input "scope" attribute (see below)
> "GENERAL" - this property value is not specific to a hadoop service.  It can be specified at the cluster or node group level as well.
> 	
> In the UI, the interfaces can provide the setting of all values.  The UI can categorize the properties based on service etc to present to the user.  If the user is in a node group configuration panel, the configuration settings will be scoped to the node group.  If they are in a cluster template or the like, the property value should be scoped to the entire cluster.
> 
> The user_input object remains unchanged.  User input values assigned to the cluster_configs attribute of the Cluster object are cluster scope properties (GENERAL or service based).  User input values associated to the embedded node groups (node_configs attribute within a particular node_group in node_groups list  of the cluster object) are associated to the specific node group (GENERAL or service based).
> 
> Again, we feel that this aligns the interface much more closely with the way users interact with Hadoop.  The attempt to align configuration with specific service components is somewhat contrived and introduces an impedance mismatch that users will probably reject.
> 
> 
> --
> Mailing list: https://launchpad.net/~savanna-all
> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~savanna-all
> More help   : https://help.launchpad.net/ListHelp
> 
> 


Follow ups

References