savanna-all team mailing list archive

[Jon Maron <jmaron@xxxxxxxxxxxxxxx>]

On May 8, 2013, at 1:31 PM, Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx> wrote:

> Erik,
> 
> >> This API is meant to allow a client using the Savanna API to make invocations on the underlying provider (Savanna effectively intercepts that call and forwards to the appropriate end-point). The alternative is of course that a client leverages the get_management_URIs() call to get a hold of the right end-point; it's just a bit more cumbersome. However, we believe that this will be pretty standard fare for all but the most basic use-cases.
> Please see the document savanna_invoke_provider_rest_api_flow for an example flow of invoking a plugin specific rest api via savanna (https://wiki.openstack.org/w/images/a/a6/Savanna_invoke_provider_rest_api_flow.pdf).
> 
> The implementation of Savanna logic supporting exec_resource_action() call will require significant amount of work. It will include HTTP proxy functionality, error handling, etc. On the other side, as you said, client can just directly access the provider by URL received from get_management_URLs().

Perhaps you misunderstood.  The intent here is that the plugin will interpret the URI and make the backend invocation against the plugin's specific management system.  For example, in a situation where a user wants to invoke a REST based query for services in a cluster, he could make the following HTTP GET request to the savanna server:

GET /v0.2/{tenant_id}/{plugin_id}/clusters/openstack1/services

The controller would parse the URI and, based on the plugin_id, realize that this is actually a request destined for the management platform.  It would then pass the URI to the plugin which could discern the correct host and port (based on cluster name), invoke the REST request against the backend management system, and return the response.

This facility would allow for:

- centralized point for backend management queries
- an abstraction layer allowing for the user to not have to figure out host/port for the hadoop management endpoint.  Rather, they simply continue to interact with the savanna server as their management node.
- this could even possibly play a role in security if the savanna server sits in the DMZ?

> 
> >> We feel that get_logger()  is critical functionality to make available throughout development, QA as well as during deployments. The implementation should be trivial and would help normalize logging across providers vs. asking providers to log on their own.
> 
> We will definitely provide plugin with some logging facility. When I said that it is not very important, I meant that it is not very important which way we will provide logging. It might be either via get_logger() method, or plugin will have to import some module and use it.

OK.

> 
> >> Today Ambari does not allow for providing arguments in a non-interactive way and without this functionality it would be a hardship for us to work around this issue. We already have code around this area that we would be glad to provide as a reference implementation. Changing Ambari to fit into this model is not something that can be done in short order as it would require changing Ambari, which in our opinion should not be necessary to fit into Savanna.
> 
> The requirement to have interactive execute does not seem to be common among other providers. On the other side we want to keep our API as small as possible. We'd rather prefer that you implement that function inside your plugin. And if it becomes needed in some other plugin, we will move it to the Savanna API then.

An interactive command execution which takes a command prompt responses list is not that complicated to implement.  We can provide an example.  Alternatively perhaps we can make use of the command pattern to feed you a command object to execute, assuming we have a proper abstraction for the SSH-like functionality.

> 
> >> Although Ambari takes care of changing 99% of the necessary configuration, there are still cases where we (the plugin provider) need to make a few changes that are OpenStack environment specific. For large scale clusters (think 1,000's of nodes), copying a large number of files back and forth provides quite a bit of overhead vs. allowing for a remote file handle. We have an initial implementation of this that we can provide.
> 
> Our vision is that the general use case for copy_file (open_file) will be to copy (or edit) only small files. I.e. we think that this operation will be mainly used to generate config files on VMs. I am not sure if remote handle will be faster than copying files in that case. At first sight, both operations will require similar amount of traffic and connections to do the same job. 

We think there is generally going to be more traffic involved with file copy functionality (at a minimum you're passing the entire file contents) versus manipulating the files remotely on the target VMs.  The remote file handle can be readily provided via SSH.  As you indicate, it may be appropriate to do a test, though it would be most appropriate to do so on a large scale to understanding the implications with a large number of nodes.

> 
> Do you have a different use case in mind? Also if you provide an implementation for open_file(), we can compare its speed with copy_file().
> 
> Thanks,
> 
> Dmitry
> 
> 
> 2013/5/6 Erik Bergenholtz <ebergenholtz@xxxxxxxxxxxxxxx>
> Team,
> 
> Comments in-line:
> 
> On May 3, 2013, at 10:57 PM, Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx> wrote:
> 
>> Hello Erik, team,
>> 
>> Below is the list of methods from Erik's team proposal, which functionality is absent in our proposal. I would like to express my opinion on that.
>> 
>> Plugin API:
>> An interfaces exposed by plugin for Savanna
>> init() and shutdown() - I think it would be nice to provide plugin with these hooks.
>> get_management_URIs() - we don't have it, but we definitely need it.
>> get_cluster_status() - right now we show statuses from the viewpoint of Savanna  {STARTING, ACTIVE, SCALING}. Obviously it would be nice to display more detailed info about cluster health, so I guess we can use that method.
>> exec_resource_action() - As far as I understand, that method allows Savanna to execute arbitrary API call on the management tool. But that requires knowledge of the tool API on Savanna side, which is quite strange. Do we really need this or does it do something different?
> 
> This API is meant to allow a client using the Savanna API to make invocations on the underlying provider (Savanna effectively intercepts that call and forwards to the appropriate end-point). The alternative is of course that a client leverages the get_management_URIs() call to get a hold of the right end-point; it's just a bit more cumbersome. However, we believe that this will be pretty standard fare for all but the most basic use-cases.
> Please see the document savanna_invoke_provider_rest_api_flow for an example flow of invoking a plugin specific rest api via savanna (https://wiki.openstack.org/w/images/a/a6/Savanna_invoke_provider_rest_api_flow.pdf).
>> get_node_template() - in our model, Savanna manages and stores all the configuration and does not need to query plugin to get config details. Do we still need to get some details from plugin? Any examples?
>> 
>> Savanna Context:
>> Interface exposed by Savann for plugin use
>> persist() and lookup() - we don't have them but we definitely need to provide persistence for plugin in some way. Sergey had an idea on how to do that, so I will leave it for him to comment.
>> exec_http_request() - right now we think that plugin should directly access the tool without any helper on Savanna side. With that approach plugin can leverage existing python client for the tool, instead of composing HTTP requests. 
>> get_credentials() - since it will be responsibility of plugin to first setup the management tool and then communicate with it, I think it would make sense to provide plugin with facility for storing and retrieving credentials. We need to think on it further.
>> get_logger() - I am not yet sure, how are we going to provide logging facility to plugin. On the other hand, IMHO that is not very important and we can decide it later.
> We feel that get_logger()  is critical functionality to make available throughout development, QA as well as during deployments. The implementation should be trivial and would help normalize logging across providers vs. asking providers to log on their own.
>  
>> 
>> Cloud Context:
>> An interface exposing a single cluster for plugin
>> get_servers() - the server list is exposed via vm_specs.servers object
>> 
>> Server (VM Manager):
>> Provides interface for plugin to work with VMs
>> Basically we would like to provide plugin with only simple means to work with VMs.
>> execute() - in your proposal that method executes a command in interactive manner, responding to command queries. We think that it is more appropriate to implement this command in non-interactive way. If a command requires some input, it could be provided via command arguments.
> Today Ambari does not allow for providing arguments in a non-interactive way and without this functionality it would be a hardship for us to work around this issue. We already have code around this area that we would be glad to provide as a reference implementation. Changing Ambari to fit into this model is not something that can be done in short order as it would require changing Ambari, which in our opinion should not be necessary to fit into Savanna.
> 
>> open_file() - again, we think it is better to provide plugin with more robust primitives like copy_to_vm() or copy_from_vm(). The open_file() working on remote host seems to be more complicated.
> 
> Although Ambari takes care of changing 99% of the necessary configuration, there are still cases where we (the plugin provider) need to make a few changes that are OpenStack environment specific. For large scale clusters (think 1,000's of nodes), copying a large number of files back and forth provides quite a bit of overhead vs. allowing for a remote file handle. We have an initial implementation of this that we can provide.
> 
> For what it is worth, we are talking with several customers that would like to use Savanna in large scale deployments exceeding the 1,000 node barrier.
> 
>> 
>> Thanks,
>> 
>> Dmitry
>> 
>> 
>> 
>> 2013/5/3 Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx>
>> Team,
>> 
>> I've created the mockups for UI dialogs responsible for launching cluster and creating node template:
>> https://wiki.openstack.org/wiki/File:Savanna-create-cluster-mockup.png
>> 
>> https://wiki.openstack.org/wiki/File:Savanna-node-template-mockup.png
>> 
>> I think they should help understand our view in the whole, and clarify our Plugin API design reasoning. IMHO, the create cluster dialog maps pretty nicely to cluster_description object from the Plugin API. The main thing which is different in the API is that Node Templates are replaced with configs. 
>> 
>> Initially we introduced Node Templates to let user specify some parameters only once during template creation. When user creates a cluster, he don't need to specify each parameter, he just needs to select one of the templates. To simplify Node Templates usage and management, we want to organize them in hierarchical structure, so that a template can extend another template. This approach will let user specify most parameters in base template, and override only few of them in each descending template.
>> 
>> As you can see from the second mockup, a template in nutshell is just a list of key-value pairs.
>> 
>> We don't want to burden plugin with processing Templates, deciding who inherits who, etc. We want Savanna to handle this and provide plugin only the resulting list of configs (key-value pairs), hence such transition from UI to API.
>> 
>> Hope that clarifies,
>> 
>> Dmitry
>> 
>> 
>> 
>> 2013/5/2 Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx>
>> Erik, team,
>> 
>> We've composed a document which more precisely describes our vision of Plugin API:
>> https://wiki.openstack.org/w/images/1/19/Savanna-plugin-api.pdf
>> 
>> The document mostly focuses on describing functions we've already introduced. And it almost does not incorporate your input yet, this is work to be done.
>> 
>> Thanks,
>> 
>> Dmitry
>> 
>> 
>> 
>> 
>> 
>> 2013/5/1 Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx>
>> Hey Erik, team,
>> 
>> Thank you for deeper dive into plugin mechanism architecture. That is really a step forward. 
>> 
>> Let us discuss the general approach first. When we started the architecture design, we also first thought of an IoC concept, similar to what you suggest. But after some thinking, we found that it is better to split one big “create cluster” call into a number of smaller ones, retaining more control on the core side. The benefits of such approach (and disadvantages of the opposite one) are:
>> 
>>  * In case of one “create cluster” call each plugin will be allowed to have different behavior, which means we will introduce a lot of undefined behavior. Especially in error cases, because each plugin could handle error case differently. Another thing is that handling such cases might require deep knowledge of OpenStack from plugin creator. Our goal is to simplify plugin creation process by handling all the OpenStack related logic inside the Savanna code, not in the plugin code.
>> 
>> In case of separate calls for each step:
>>  * Plugin is separated into several consecutive parts/methods. Transitions between these methods might be persisted which would increase reliability of the workflow.
>> In perspective, it might allow to run plugin in distributed environment.
>> 
>>  * Separate and defined by API plugin methods allow code reuse. It also serves as a documentation of plugins responsibilities. 
>> 
>>  * It’ll allow timeout handling for each step on the core side.
>> 
>> 
>> Methods for interacting directly with a VM/Server instance like install(), open_file(), interactive execute() don’t seem to be relevant to the Plugin API. It’s better to keep such method away from the API to keep that API as simple as possible. It can be just a set of helper methods under utils package.
>> 
>> We suggest to move provider specific details to separate blueprints. Examples are “HDP specific details” in create cluster flow and in add hosts flow.
>> 
>> Thanks,
>> 
>> Dmitry, on behalf of Mirantis team
>> 
>> 
>> 
>> 2013/4/30 Erik Bergenholtz <ebergenholtz@xxxxxxxxxxxxxxx>
>> Dmitry - uploaded a new doc that looks a bit better: https://wiki.openstack.org/w/images/5/5e/Savanna_Deployment_Engine_Architecture.pdf
>> 
>> Erik
>> 
>> On Apr 30, 2013, at 4:20 PM, Dmitry Mescheryakov <dmescheryakov@xxxxxxxxxxxx> wrote:
>> 
>>> Hey Erik,
>>> 
>>> Some tables in "Savanna Deployment Engine Architecture" doc are flattened out, see attached screenshot for example. Could you reassemble the PDF with correct tables' sizes?
>>> 
>>> Thanks,
>>> 
>>> Dmitry
>>> 
>>> 
>>> 2013/4/30 Erik Bergenholtz <ebergenholtz@xxxxxxxxxxxxxxx>
>>> Team - yet another update. See https://wiki.openstack.org/w/images/9/97/Savanna_hadoop_host_group_mapping.pdf for a document illustrating how hadoop nodes get mapped to provisioned VMs. 
>>> Erik
>>> 
>>> 
>>> 
>>> On Apr 30, 2013, at 11:07 AM, Erik Bergenholtz <ebergenholtz@xxxxxxxxxxxxxxx> wrote:
>>> 
>>>> Team,  John has updated the below referenced documents with better descriptions of the flows (same links apply).
>>>> 
>>>> Cheers,
>>>> 
>>>> Erik
>>>> 
>>>> On Apr 30, 2013, at 6:35 AM, Erik Bergenholtz <ebergenholtz@xxxxxxxxxxxxxxx> wrote:
>>>> 
>>>>> Team,
>>>>> 
>>>>> Below are a few documents intended to describe only a slightly modified approach to hadoop provider integration into Savanna (see existing blueprint in draft: https://blueprints.launchpad.net/savanna/+spec/pluggable-cluster-provisioning). These documents should not be considered a blueprint, but a vehicle for continuing discussion on the topic. 
>>>>> 
>>>>> To summarize there are two changes to note:
>>>>> 
>>>>> 1. There is some IoC introduced into the design allowing hadoop plugin providers flexibility to integrate into Savanna while reducing the burden on the Savanna controller itself. This differs from the current approach of the controller invoking APIs on the provider at specific lifecycle points.
>>>>> 
>>>>> 2. Attempts have been made to keep normalization of management APIs across providers at a minimum at the controller level. Our view is that existing Hadoop users are already familiar with their Hadoop distribution management API (CDH, Ambari, MapR etc.) and as such would want to leverage existing investments vs. learning a new management API specific to Savanna. This eases adoption and lowers the barrier of entry of adoption of Hadoop on OpenStack.
>>>>> 
>>>>> Documents to review:
>>>>> 
>>>>> Savanna Deployment Engine Architecture - Puts forth the architecture of the deployment engine 
>>>>> Savanna_add_hosts_flow - Describes sequence of steps executed in order to add a host to an existing Hadoop Cluster
>>>>> Savanna_create_cluster_flow - Describes the sequence of steps executed in order to create a new cluster
>>>>> Savanna_invoke_provider_rest_api_flow - Describes the sequence of steps executed in a REST request by the provider plugin on the controller.
>>>>> 
>>>>> Please review at your earliest convenience and let us know your feedback.
>>>>> 
>>>>> Best,
>>>>> 
>>>>> Jon Maron, John Speidel and Erik Bergenholtz
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Mailing list: https://launchpad.net/~savanna-all
>>> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~savanna-all
>>> More help   : https://help.launchpad.net/ListHelp
>>> 
>>> 
>>> <Screen Shot 2013-04-30 at 1.15.11 PM.png>
>> 
>> 
>> 
>> 
>> 
> 
> 
> -- 
> Mailing list: https://launchpad.net/~savanna-all
> Post to     : savanna-all@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~savanna-all
> More help   : https://help.launchpad.net/ListHelp