← Back to team overview

openerp-expert-framework team mailing list archive

Re: Making our way out from the bloated extra-addons repository

 

We already moved some modules from addons that we want to remove from the core, to their own branch (sometimes grouped by topic) here:

https://code.launchpad.net/openerp-extra

We also started to move some modules from extra-addons there like

https://code.launchpad.net/~openerp-community/openerp-extra/7.0-olap

I used fast-export fast-import with a filter to keep the history.

We use the convention "MAJOR-OPENERP-VERSION-branch-name" like we do for the addons, server and web branches.

On 10/29/2012 05:13 PM, Raphael Valyi wrote:
Dear OpenERP experts,

Some of you might not have already think about it, but those who have all came
to the same conclusion I will re-explain here:

*The problem*

The system of the single extra-addons branch should now be avoided because it
doesn't scale in term of code and community management:

 1. enabling or restricting commit rights doesn't scale: you may want to allow
    commit rights to anybody of your team for you little non critical
    extra-addons but the other users may want to be sure the extra financial
    addons they are using in production are rock solid and not screwed
    overnight by some beginner.
 2. extra-addons receive commits for a huge number of addons. Hence it's
    really challenging to track the code evolution of the critical
    extra-addons you use
 3. because there are so many extra-addons, the branch is huge! Often you just
    need 2 or 3 extra addons and you are still forced to download hundreds of
    Mb of code. This is specially discouraging contributions in all the
    countries where Internet isn't so fast...
 4. because there has been no quality management of those modules, the
    extra-addons branch is full of poor quality non reusable, non migrated
    modules. Probably 75% of the extra-addons modules are broken and not
    reusable outside from the initial company or even POC they where developed
    for. Worse than that, at the beginning, say back in 2008, Tiny and Axelor
    were putting almost all their POC modules with not enough quality to
    qualify as addons in the extra-addons branch, resulting in garbage modules
    lagging the extra-addons quality until today. Following this "example",
    few companies refrained themselves from pushing their low quality modules
    in the extra-addons during those years.
 5. often you need to improve some extra-addons for several reasons, mostly:
     1. adapt it further to the current stable OpenERP version your are using,
     2. add new features for the company you are working for if you think it
        makes sense to have that feature inside the module
     3. fix a bug in a non backward compatible way
     4. refactor the code to make the module more compatible with some other
        modules or localization

    In all those cases, the proper way is to create a new "feature branch" and
    eventually merge it back into the "stable" branch. But because things
    aren't so mature yet, often changes are not backward compatibles and it's
    better to stay in a new branch people should explicitly pick, while
    keeping the "stable" branch for really stable things people can update
    without special care (ideally). If a branch has many modules like the
    extra-addons, then it creates many parallel versions of the same modules
    which is a real hassle to maintain (addons_path won't do the job etc...)


*The solution*

Some of us know that for some time already, so instead of bloating the
extra-addons with new modules, we have been creating new modules in new
branches since one or two years.

But now, we should go further, we should move the extra-addons we use often
out from the extra-addons into smaller manageable branches.

Because ERP means big money involved, OpenERP contributor copyrights are often
abused while copyrights tend to be an essential pillar of how open source
works. So it's essential that the bzr history of the extracted module don't
get lost and that real authors can be traced back (if that get lost, it will
increase the risk somebody comes and change the license claiming they are the
only author).
At Akretion we developed some semi-manual script to do that and replay the
module commits in the new branch. I'll pass it soon in this list.

Also what's the solution: one module per branch or several modules in a branch?

Well eco-systems that use well working module managements system tend to use
the one module per branch approach as one can easily rebuild at anytime all
the dependency tree with the proper module version that have been tested to
work together.

But given the OpenERP approach of not dealing with module version dependencies
http://openerp-server.readthedocs.org/en/latest/module-versioning.html, it
will not be easy to spot the compatible modules versions (that information is
not even stored!). Imagine if you used 20 extra-addons, you would now need to
do bzr pull in 20 different branches and ensure manually you are taking the
proper branches and updating to compatible revisions...
So given this situation, at Akretion we think it can still make sense to group
some related modules inside the same branch until possible better module
management appears and possible further branch split.


*Grouping modules together*

Then comes the grouping problem: what modules go together into the same
branch? Well, unfortunately there won't be some clear rule, it has to be a
tradeoff. One could think just put everything that depends on sale together,
everything that depend on stock together. Unfortunately this isn't that
simple: there are modules having multiple core dependencies and many border
line cases (where would the stock_rma module fit: stock or CRM?, where would
the account_fiscal_position_rule module fit: account or sale?)

So let's start discussing those groupings if you like. I'm taking several
factors into account:

  * dependencies
  * functional domain
  * authors
  * avoid to leave just one required module alone in a new branch to checkout
    if possible
  * usage in localizations...


*Example*

Sebastien Beau (Akretion) already extracted the fiscal rule modules together
in a new project/branch:
https://code.launchpad.net/openerp-fiscal-rules
It contains those modules who where extra-addons originally:

  * account_fiscal_position_rule
  * account_fiscal_position_rule_purchase
  * account_fiscal_position_rule_sale
  * account_fiscal_position_rule_stock
  * account_product_fiscal_classification


We developed those above modules for the Brazilian localization. But in fact
they are potentially useful for managing the fiscal positions in all countries
which are federations (possibly even the USA) and are useful for international
fiscal operations. Recently we started using them more for ecommerce projects
in Europe and Canada for instance to select the right tax accounts even if the
ecommerce front-end (like Magento) already pass the proper VAT ratio to OpenERP.

We should continue that migration process further and ideally stop depending
on the extra-addons for OpenERP v7 projects.

Also, by removing the dependencies on that fat extra-addons repository, we
also make automated Continuous Integration testing easier. At Akretion we are
experimenting with Github mirrors and Travis-CI triggers to run automated test
at every commit. It's not that we don't like the Runbot, but we can run the
same tests for free on any branch and easily add support for next testing
suites like OERPScenario test suites freely. Also, OpenERP SA re-affirmed they
won't care about module dependencies, but this doesn't match what we do at
Akretion (who here already tried to install the proper Magentoerpconnect
dependencies?), so we will need to be able to specify the proper branch
dependencies for all the testing and have no faith the Runbot can do that when
the version dependencies will be missing from the module format. Well, instead
free open source tools like Travis-CI provision a new Virtualbox machine at
every run and do a fresh repository checkout, so by depending on lighter
repositories, we also make all that easier to happen. This is part of the road
to a better quality of those OpenERP modules.


I would appreciate your view on that question and will post again soon to
submit a new concrete module extraction proposal for the financial
extra-addons used both by the Spanish and Brazilian localization at least.


Best regards,


--
Raphaël Valyi
Founder and consultant
http://twitter.com/rvalyi <http://twitter.com/#!/rvalyi>
+55 21 2516 2954 <tel:%2B55%2021%202516%202954>
www.akretion.com <http://www.akretion.com/>




_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : openerp-expert-framework@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp


Follow ups

References