fuel-dev team mailing list archive

[Oleg Gelbukh <ogelbukh@xxxxxxxxxxxx>]

What if we implement plugins not as separate Puppet modules, but as
separate Mcollective modules?


On Thu, Jan 16, 2014 at 3:43 PM, Dmitry Ilyin <dilyin@xxxxxxxxxxxx> wrote:

> Actually you have risen a very complicated question...
>
> We have been thinking about separating our deployment process for a long
> time already but have not done anything so far.
> Currently we are using a single 'main' manifest that is applied to every
> node. It uses puppet modules as 'library' and data from astute.yaml.
> This huge manifest is far too large to be easily understood and is full of
> hacks and kludges.
> It makes reference architectures very unflexible and custom roles hard to
> implement.
> And I'm not even talking about joys of debugging all this stuff and a lot
> of dependency and ordering fun we have with it.
> Adding something new requires not only deep understanding of this
> structure but also a lot of luck not to break something accidently.
> It makes pretty much impossible for an external developer to write a
> plugin or custom role he could just add to Fuel an have it working as
> expected.
>
> So we have an idea to separate this manifest to many small ones. Each of
> them would do only one task and orchestration engine would apply them in
> required order
> to deploy the node user have requested. With such architecture each
> 'stage' would be more or less isolated and it would be possible for a
> single developer
> to add a new 'stage' that would install his 'plugin'.
>
> Yes, Dmitry's idea about implementing such plugins as separate puppet
> modules is good, but it's not that easy to do.
>
> * Modules can contain their own manifests, files and templates and can use
> configuration from astute.yaml. It will makes possible to represent every
>   service we install as plugin and every 'role' would be a set of
> 'plugins' installed in certain order. Orchestrator will have to run 'puppet
> apply' on each
>   module's init.pp to get the job done.
> * Separating single puppet run to many runs will let us to bypass resource
> uniqness restriction and we'll be able to manage same resources on
>   different 'stages' without hacks to prevent conflicts.
> * Some 'stages' could even be done not by puppet if they are really hard
> to do by puppet.
>
> But there are questins I have no ansver yet.
>
> * Even if we'll make each 'stage' a separate manifest there still we be a
> need to use our current puppet modules as library. Or we should make every
>   modules fully self-contained? Or we could use something like python's
> virtualenv to provide each 'stage' with correct set of library modules?
> * What is the best way to install packages then? Keep our current
> repository on master node or invent somethin else?
> * How and where to store information on what 'stages' should be done for
> which roles? There should be some metadata with fixed scenarios or there
> should be dynamic graph?
> * Some 'plugins' and 'stages' may require to add options to settings
> panel. If we are going to have external plugins there should be a way to
> hook int setting and add your options.
>   This data will the be available in astute.yaml.
> * There should be some system to run pre-deploy and post-deploy tests for
> each stage to determine was it successful.
> * Our orchestation engine will have to become much more complex then what
> we have now. There should be queues of stages for each deploying node. And
> there should be support
>   for inter-node dependencies of stages.
>
> I'm working new architecture blueprint. I'll send updates when i'll be
> ready.
>
>
>
>
>
> 2013/12/28 Roman Alekseenkov <ralekseenkov@xxxxxxxxxxxx>
>
>>  Dmitry - sounds like a great start.
>>
>> Team,
>>
>> Pluggable architecture is a priority for 4.1. Given that we are now done
>> with Havana release, please make an effort to review and give prompt
>> feedback to Dmitry.
>>
>> Thanks,
>> Roman
>>
>>
>> On Wednesday, December 25, 2013, Dmitry Ukov wrote:
>>
>>>  Hello folks,
>>> recently I've refactored integration of Neutron Nicira NSX plugin
>>> installation into fuel library (https://review.openstack.org/#/c/64002/).
>>> All modifications are gathered into one puppet module.
>>> I have some thoughts on implementation of plugin system in Fuel library
>>>
>>>    1. Each plugin has to be implemented as a separate module. Init
>>>    manifest withouta input parameters (init.pp) should be used as an entry
>>>    point to each plugin.
>>>    2. All parameters for plugin should be kept in astute.yaml
>>>    3. Plugin naming convention should be developed.
>>>    4. Function which automatically loads each plugin (e.g. it should
>>>    execute init manifest of each class with name mask "plugin_.*") is required.
>>>    5. Staging system should be implemented. All plugins have to be
>>>    executed in a separate stage (e.g. Stage['main'] -> stage {'plugins':} )
>>>    6. Each puppet class which is used with library has to have the
>>>    ability to be disabled from plugin
>>>    This is necessary because plugins may require to disable
>>>    some services. Alternatively this can be implemented by using resource
>>>    collections (e.g. Service <| title == l3-agent|> { noop => true }) but
>>>    we can't execute plugin in a separate stage in this case.
>>>
>>> What do you think on this? I'd like hear a feedback on my ideas and
>>> module implementation.
>>> Thank you in advance.
>>>
>>> --
>>> Kind regards
>>> Dmitry Ukov
>>> IT Engineer
>>> Mirantis, Inc.
>>>
>>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>