openerp-connector-community team mailing list archive

[Thomas Rehn <thomas.rehn@xxxxxxxxxx>]

Hi all,

We at initOS are currently dealing with various projects where we have
to import data into and export data from OpenERP. We looked at the
existing solutions (connector_logistic_center, connector_file) but these
do not really fit our needs. Both modules are helpful especially if
there is a fixed work flow with data in standard formats. In our case we
have to support several input and output methods (user interface, file
system, FTP, email) and formats (CSV and proprietary logistic formats),
split into small modules (different customers). Therefore we rather aim
at a minimal, flexible framework on top of the 'connector'. The overall
goal is to have export and import functions running asynchronously and
in parallel. We also want to provide an easy way to add your own format
handling definition or your own source or destination point for file data.

Over the last days we have implemented a prototypical module called
'connector_flow' [1]. The idea is to have small encapsulated tasks that
can be plugged together into a task flow like states in an OpenERP
workflow. Possible tasks could be parsing a CSV file into structured
chunks of data, uploading a file to FTP or sending a file by email. For
our prototype we borrowed some ideas from the existing solutions to
implement very basic CSV import/export tasks and an FTP upload task. We
also provide an example module 'connector_flow_example' in which we
realize a demo flow in the demo data that exports res.partner as CSV
file onto an FTP server and imports res.partner from a CSV file.

Currently there are still a lot of rough edges and missing things like:
- Execution of tasks (or task flows) can only be triggered manually by a
wizard.
- Configuration of tasks is done by Python dictionaries.
There also are issues that we have not spent much time on yet, for
instance, figuring out whether the export can also be parallelized by
the connector. The problem here seems to be that one needs dependencies
of jobs in the connector queue because a (CSV) file can only be written
after all rows have been generated.

We also did not implement all concepts that the connector framework
offers, for example backends and bindings. At the moment we want to have
a minimal prototype and add more from the connector concept if we see
how to benefits in the long run.We uploaded our prototype into our
repository at [1] to continue the discussion of how a connector-based
file import/export could look like and which aspects may be relevant.
We see this as our contribution to go stepwise towards a file handler
based on the connector framework. So we very much appreciate any
comments, review and discussions.

Best,
 Thomas.

[1] https://code.launchpad.net/~initos.com/+junk/connector_flow

-- 
Dipl.-Math. Dipl.-Inf. Thomas Rehn
Software developer

initOS GmbH & Co. KG
An der Eisenbahn 1
21224 Rosengarten
Germany
http://www.initos.com