openerp-community team mailing list archive

[Georges Racinet <gracinet@xxxxxxxxx>]

A recurring question about the OpenERP buildout recipe evolution is how
to make it evolve to support automatic upgrade and creation.

I apolodgize in advance for this very long message. I can't help stating
the context and what my interpretation of it beforehand…
Hurry people can jump to the PROPOSAL section to see what I plan to
implement for version 1.8 of the recipe.

In short, that is a high level API, and the contents of that API is open
to discussion, until 1.8 is out, after which it must be really stable.
Users feedback on this is really needed, unless you trust that our use
cases include yours :-)

STATING THE PROBLEM

Managing upgrades is a hard problem in general, and it's especially
difficult in case of the recipe, precisely because it allows a
tremendous flexibility in addons and python dependencies.

So far, the answer from my part has been something along the lines of "I
can't predict anything in general about what should be done for an
upgrade or database creation, so that has to stay outside of the scope
of the recipe, just write the 'start_openerp' line you need in the
documentation, or go issue a two-line shell script if you want don't
want to correct the documentation for each release"

At this point, I should state explicitely that I'm not ignoring that
addons have version numbers, which already get used to select pre/post
migration logic in the modules upgrade sequence. Our problem lies one
step upper in the stack, namely:

 - deciding which modules to upgrade or install (all is often not the
right answer)
 - performing actions that can't be the responsibility of one specific
module (especially if there is no identified terminal module)

The recipe is meant to build final assemblies, client projects or
vertical integrations, the buildout configuration is subject to have its
own versioning and release cycle, tags etc, with released configuration
referring themselves to precise versions of addons and eggs etc.  The
author of that released buildout configuration is the only one knowing
exactly what to do to upgrade from version 1.0.0 to version 1.0.1, hence
the typical answer quoted above : go manual or make a script.

CURRENT LIMITATIONS, AND GOALS

Still, there are obvious drawbacks to the dedicated script approach

1. Uniformity.

We want to test upgrades in our favorite continuous integration tool
(actually Anybox does, and that gave me incentive to tackle this
problem). We want to perform massive deployments of vertical integrations.
We want that big "upgrade" button in the hosting admin console !
Even for projects with dedicated, separate hosting, and human operators
applying upgrade procedures, we want the application part to stay simple
and not to change.

All of that screams : uniform command line options, exit with proper
status codes in case of failures. Have a good quality in logs.

2. No real version logic.

While it's currently easy to include alongside yout buildout
configuration a script able to upgrade an installation from version 1.0
to 1.0.1 (expect a few filesystem paths problems, though it's a bit more
complicated to write a script that can upgrade from any intermediate
1.x.y version to 2.0.0.
In the long run, a shell script doing that will become ugly and
unreadable. Besids, our lingua franca is Python, not sh. Performing a
bunch of subprocess calls is a bit clumsy too.

PROPOSAL

The recipe will provide a high level API for people to write the main
upgrade script in Python while focusing on the decisional part, not on
the boiler plate.

After so long verbose explanations, an example will do us some good. An
upgrade script would look like this:

    def run(session):
        old, new = session.get_update_versions()
        if new >= (1, 1) and old < (1, 0):
           session install_modules(['some_new_module'])

        session.update_modules(['module1', 'module2'])
        session.update_translations(['module2'])

Yes, that 'session' object is the same as provided by the

In your buildout configuration part, you'd put a reference to that
script and the expected version file.

    [openerp]
    (...)
    upgrade = my_upgrade.py run   ; an entry point would be neat
    version_file = VERSION.txt    ; used to know the target version

Then the recipe would issue a 'bin/upgrade_openerp' script, with a fixed
set of options.

    -d, --databases   databases to upgrade
    --log-file        log file for the upgrade
    --log-level       level of logging.

It will take care of respecting the industry standards, notably issuing
proper logs stating that it's been succesful in xxx time, exiting with
appropriate status, etc. It will also set the version in DB for the next
upgrade.

To me, this looks like the proper articulation between human knowledge
and automatisation of boring tasks. Besides, people wanting to automate
even further can have write higher level tools with introspection logic
and target them to that API. This is regular Python, imports are allowed.

Much can be discussed, before recipe 1.8 after which it must be really
stable:

 - the standardised set of options for the produced script
 - the related buildout options and their default values
 - the shape of version strings (I really like to be able to compare
tuples of ints)
 - the API itself

As a final note, at Anybox, we have a release script that takes care of
producing a frozen configuration, committing it in Mercurial, issue a
tag named after the produced version and set the version in various
useful places, including the VERSION.txt file mentioned above.
Then it's just a matter of producing a tarball, test it and upload it
where appropriate (more boring work, at which builbdot is less sloppy
than we are).

We may publish this release script also at some point if people are
interested. Don't know where, though.

Regards

-- 
Georges Racinet
Anybox SAS, http://anybox.fr
Bureau: 09 72 39 50 97 / 09 72 39 13 06
Portable: 06 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics

Attachment: signature.asc
Description: OpenPGP digital signature