savanna-all team mailing list archive

[John Speidel <jspeidel@xxxxxxxxxxxxxxx>]

Here are more details on the advanced Hadoop configuration that we 
discussed the other day.


Savanna Advanced Hadoop Configuration

In addition to the proposed "config items" based Hadoop configuration, 
it will be necessary to provide an advanced configuration mechanism in 
Savanna.This mechanism should allow for very fine-grained and extensive 
configuration of a Hadoop cluster provisioned by Savanna.It is expected 
that a user would likely use the simple node group based configuration 
for cases where little configuration is required and use the advanced 
configuration where more control is desired.The advanced cluster 
configuration would be specific to a Hadoop plugin and it's content 
opaque to the Savanna controller.

For reference, here is a link to the Hadoop Blueprint Specification 
<https://issues.apache.org/jira/browse/AMBARI-1783> proposed by the Ambari.


     Advanced Hadoop Configuration Use Cases

·A user has an existing on premise or non-virtualized cluster and wants 
to clone the cluster(topology/configuration not data) in a virtualized 
environment using Savanna.

In this case, the user will export a configuration for the existing 
cluster using provider/management product specific tooling.This 
configuration can then be used to create a new cluster using Savanna.

·A user wants to provision a new cluster in a virtualized environment 
using savanna and needs very fine-grained control of the Hadoop cluster 
configuration.This could include configuration of host level roles, 
configuration of a large number of properties across many optional 
services and potentially even Hadoop stack configuration related to 
packages and repository locations.


     Changes to UI Workflow

To allow a user to specify an advanced configuration, some UI changes 
are necessary.

The create cluster screen would need an "advanced Hadoop configuration" 
tab or button.In the initial implementation, the advanced configuration 
screen would allow a user to specify the location of a plugin specific 
configuration file (select file dialog).This configuration file would 
contain all necessary Hadoop related configuration.In future releases, 
we may want a link to provider specific tooling, which could be used to 
create/edit provider configurations.

The UI would still need to allow a user to specify VM details such as 
flavor, count, etc., but the user wouldn't specify node groups or 
configuration for the VM's.Instead, host/role mapping would be specified 
in the provider specific configuration file.


     Changes to Hadoop Plugin SPI

The addition of "Advanced Hadoop Configuration" using plugin specific 
configuration will result in small changes to the proposed Hadoop Plugin 
SPI.

cluster_description: The cluster description object would need to be 
updated to contain an advanced_configuration field in addition to 
cluster_configs.In the case of a user providing an advanced 
configuration, it would be available in advanced_configuration and 
cluster_configs would be empty.

configure_cluster(..)

Because the provider specific configuration is opaque to Savanna, it 
might be necessary for the plugin to return some cluster topology 
information from this method for rendering purposes.The specifics of 
this information would be dependent on what cluster information is 
required by Savanna.


**