openupgrade-drivers team mailing list archive

[Valentin LAB <valentin.lab@xxxxxxxxxxx>]

Hi Holger,

Thanks for your in depth answer,

> Mind you that the migrate-mechanism itself is supported by the
> standard-version of the server too. Openupgrade merely adds a bunch
> of convenience methods that ease writing migrations scripts a lot.

I've noticed all this while having to deal with merge conflicts with 
saas-3 and trunk branch.


>> Well, I noticed several bits of code that will cancel some actions,
>> but I don't understand why these bits of code are there. Same thing
>> for the way cr.commit is diverted: why not use savepoints ? Well,
>> okay, there are a lot of cr.commit() in the legacy code, but they
>> are useless, we can remove them it seems (or replace them with
>> savepoints)
>
> we added some savepoints and rollbacks where appropriate (grep for
> openupgrade in orm.py). The rationale behind defusing the cursor's
> transaction methods instead of removing their calls from code is that
> we want so make as little code changes as possible to simplify merges
> from upstream and porting openupgrade to following OpenERP versions.

I understand the rationale of making the less possible code change. What 
about my current merge proposal on trunk ? 
https://code.launchpad.net/~0k.io/openobject-server/trunk-new-prevent-cr-commit-to-allow-super-transaction/+merge/210884
This would remove the need for any 'cr.commit()' divertion if accepted 
in trunk.

I based my code around recent code that introduced easy savepoint usage 
through a context manager in python.

A lot of code will make some change to database even if not in migration 
mode (look at all the cr.commit() in orm.py in function _auto_init), and 
thus, this code doesn't ensure that the migration process will be 
reproduceable identically depending on the modules installed, but also 
the update path... this last notion referring the exact order in which 
you encounter errors (and relaunch update) or modules.

It seems that my point of view (about one transaction for init/update) 
is partly shared with Anthony Lesuisse on the openobject-framework list. 
This is why I wrote the merge proposal.

>> What is the general policy to follow regarding transaction ?
>
> One transaction per module migration, see cr.commit_org() in
> openerp/modules/loading.py

Yes, I've come accross this. But I think that updating openerp is so 
complex that it should be reproducible at least: if any module fails, 
forcing the whole process to go back to the exact same state before the 
process was started seems to me a saner way to develop migration scripts 
and to apply them. How can you garantee that your script will not fail 
especially when all the dependencies and migration script can break one 
another ? This won't clear all issues, but it certainly remove all the 
steps of your "update path".
Leaving database with half modules installed seems strange to me. I 
often deal with modules that should have deleted a view and prevent 
another module from upgrading. My only way to deal with them is to 
upload a new copy of the database before the import.

Shouldn't we consider migrations script good only if they are able to 
migrate completely in one go a database ?

>> More precisely: why the whole initilialisation / update process is
>> not in one big transaction ?
>
> For example, the timezone migration for 6.1 touches every row of
> nearly every table in the database. For big databases, this might
> become an issue by itself, but probably would also for smaller
> databases if we stack a lot of other stuff onto that in the same
> transaction.

That's a good point that I had in mind, but lacked a concrete example. 
Thanks.

>> For now, update scenario is really bad, and
>> involves re-launching update script and slowly updating modules by
>> modules the database. This update path will be unique, and won't
>> catch all errors that could occurs in other slightly different
>> paths. Not to mention that modules are marked 'installed' to the
>> final version if their script passes, and this forces you to re-load
>> the database in the first state if you just forgot some code to add
>> in the upgrade module.
>
> The important part here is the version number in your database. This
> decides if a migration script is run at all. So if the migration is
> done for one module, it won't be run again. But you're right, an
> --update=all which most people do triggers a module *update* (which
> can take a while for i.e. account), but no migration.

hm ? not sure to understand you completely: my ``pre-*.py`` scripts are 
run when I do a '--update=all' ! Could this come from trunk or saas 
branch ? Moreover: every version's directory in migrations directory 
that is between the current version and the target version will be run.

> Please publish that code, it might save time when we start working on
> openupgrade for 8.0. If the above doesn't apply to your server, then
> indeed something went wrong with your merges.

There was no big deal when merging, few conflict appeared. We did a 
quick dirty run without delving in the code as much as we should have. 
Subtle bugs or brutal conflit resolution might have broken things so we 
will probably look at it again before sending the code in a merge 
proposal. However, we have migrated a small database with a few 
classical modules and custom modules from our custom branch based on 
trunk toward saas without too much difficulties.

I have code that protects the whole initialisation process and update 
process in a single transaction because I needed that functionality. I'm 
willing to make a MP of this. But if you think this is not a good 
feature, we will keep it for us, the code change is really minimal (thx 
to the previously cited MP) and we can afford maintaining it on our side 
along along your branch.

> Are you aware of the code sprint planned for openupgrade before the
> community days?

Yes, I've noticed the thread in the mailing list. I haven't made up my 
mind, especially because I'm not sure that I'm aware enough of the 
functional and technical parts of the openupgrade process to really be 
of any help at this event... I'll poke up directly in the concerned 
thread if I know I come.

Many thanks for your complete answer,

--
Valentin LAB
Ingénieur Développement

tel:  +33 6 71 39 62 13
mail: valentin.lab@xxxxxxxxxxx