openerp-connector-community team mailing list archive

[Guewen Baconnier <guewen.baconnier@xxxxxxxxxxxxxx>]

Hi there,

I just want to send to you a summary of the short-medium term goals of the
connectors and what you can expect in the next months.

*TL; DR*
- Proposal in the community projects
- Refactoring of the Mappers (framework)
- New feature in the Mappers: modifiers
- Some non backward compatible changes in the framework, but kept as
minimal as possible
- Proposition of default implementations of some pieces in the framework
like the Binder
- Magento Connector: refactoring of the "special lines", remove legacy code
- Export of the Magento catalog

Long story
==========

The first thing to denote, is that as though that we have now publicly
released the connector, our next step is to propose the framework and the
implementations in the Community Addons.
We'll need to propose merges for the connector_ecommerce add-on in
lp:e-commerce-addons too.

On a more technical side, I come with exciting news.

I started to work on the project a bit less than one year ago, and then, if
I had a global vision of how I would articulate the framework, it was
difficult to foresee all the use cases. The only way to know them is to
challenge the framework with real implementations! After some
implementations, I know some limitations or thwarted possibilities which
frustrate me. I'm now prepared for a second round with the framework. I'll
present you the upcoming changes, which will unfortunately not all be
totally backward compatible (but the required changes as kept as minimal as
possible), but at this point I still prefer to break the backward
compatibility than to keep inefficient pieces until we have gained more
maturity.

Modifiers
---------

In the mappings, we have the possibility to defined 'direct' mappings.
Example:

    direct = [('source_field', 'target_field')]

Integers, chars, etc. are copied. IDs of m2o relations are passed to their
Binder which returns the external ID from the binding ID (or conversely
returns the binding ID from the external ID).

I observed that we often need to apply a similar modification on the source
values, noted examples:
 - A backend sends '0000-00-00' for empty dates, we have to replace that by
None
 - A backend sends dates in ISO 8601 format, we need to convert them to UTC
then to a naive timestamp
 - A backend sends floats as integers, needs to divide by 100
 - We want a default value when a value is empty
 - And in fact the most common operation is to find an external ID for a
local ID and conversely, it was automatic but only for IDs of bindings

We could already do the modifications using @mapping methods, but it was
very repetitive.

That's why I introduced the "modifiers". Example:

    direct [(iso8601_to_utc('source_date'), 'target_date'),
            (ifmissing('source_field', 'default value'), 'target_field')]

They also resolve an issue with the m2o: they were implicitly trying to
search a Binder for the relation of the m2o, so not working if the relation
was a "normal record". Now, I recommend to explicit the operation:

        # partner_id relation's 'res.partner' (normal record)
        # mag_partner_id relation's 'magento.res.partner' (binding)
        # s_partner_id is the id on the backend
    direct = [(backend_to_m2o('s_partner_id',
binding='magento.res.partner'), 'partner_id'),  # we needed to give the
binding model so it can use it to find the normal ID
              (backend_to_m2o('s_partner_id'), 'mag_partner_id')]  # we
don't give the binding model if it is the same than the relation

Merge proposal:
https://code.launchpad.net/~camptocamp/openerp-connector/7.0-connector-closure-functions/+merge/193389

Mappers
-------
One of the things I was not very satisfied were the Mappers.
I started a refactoring that I'm currently challenging on a customer's
project. I'll soon propose it for the merge.
If you are interested, you will find much information on the WIP proposal:

https://code.launchpad.net/~openerp-connector-core-editors/openerp-connector/7.0-connector-mapper-refactor/+merge/194485

Here is what I needed to change on the Magento Connector:

https://code.launchpad.net/~camptocamp/openerp-connector-magento/7.0-new-mapper-api-gbr/+merge/194630

While it is not ready to be merged yet, I *strongly encourage* the
developers of connectors to already base their development branches on this
one.
The branch also include the "Modifiers".

Data passed to on_record_create, on_record_write
------------------------------------------------

Laurent Mignon proposed to modify the pass the complete dict of values
instead of the list of modified fields in the "on_record_create" and
"on_record_write" events, which I agreed with.
Discussion:
https://lists.launchpad.net/openerp-connector-community/msg00010.html
Merge proposal (connector):
https://code.launchpad.net/~lmi/openerp-connector/7.0-pass-vals-to-events/+merge/189234
Merge proposal (magento):
https://code.launchpad.net/~openerp-connector-community/openerp-connector-magento/7.0-magentoerpconnect-pass-vals-to-events/+merge/189295
Merge proposal (prestashop):
https://code.launchpad.net/~openerp-community/prestashoperpconnect/7.0-pass-vals-to-events/+merge/190227

This is a non backward compatible change, but easy to change. I am still
waiting for reviews and tests on the Magento and Prestashop branches before
merging.

Default Implementations
-----------------------

Excepted for the Mapper, the framework does not propose default
implementations of the other ConnectorUnit classes (Binder, Synchronizer
for export and import, BackendAdapter) but just interfaces. This is on
purpose: we can't really now what could be generic and what couldn't. I
prefer to implement some connectors, correlate them and only then see if we
can generalize them. Actually, the first good candidate I can see is the
Binder. And even there, when I thought it was OK, on my last connector I
saw that I needed to change it. I'll try to propose a default
implementation soon.

Magento Connector: refactoring of the "special lines"
-----------------------------------------------------

What I call special lines are: shipping, cash on delivery and gift lines.
The actual implementation uses legacy code from base_sale_multichannels
which is a hardly readable.
I decided to take over it and, with help of the new Mapper API, I
reimplemented them (adding the cash on delivery which was missing).

WIP (connector_ecommerce):
https://code.launchpad.net/~openerp-connector-core-editors/openerp-connector/7.0-e-commerce-addons-refactor-so-extra-lines/+merge/194629
WIP (magento):
https://code.launchpad.net/~openerp-connector-core-editors/openerp-connector-magento/7.0-magentoerpconnect-refactor-so-extra-lines/+merge/194631

I still need to add some tests. And I would be happy if people could test
the change for real as many different setups can exist (taxes incl./excl.,
etc.)
(The new Mapper branch is required for the connector)

Magento Connector: Export of the catalog
----------------------------------------

Sébastien Beau, David Beal and Arthur Vuillard from Akretion and Augustin
Cisterne-Kaas from Elico Corp are currently working on the export of the
catalog.

--

I know that I have been very long, thank you very much if you have read me
up to there.

Bye

-- 
Guewen Baconnier
Business Solutions Software Developer

Camptocamp SA
PSE A, CH-1015 Lausanne
Phone: +41 21 619 10 39
Office: +41 21 619 10 10
http://www.camptocamp.com/