| Thread Previous • Date Previous • Date Next • Thread Next |
Hi, Thanks for bringing this discussion. On 01/03/2014 06:23 PM, Leonardo Pistone wrote:
Hi all, this general message is just to bring the everybody who is interested in the discussion about using the connector for generic csv imports. Like for e-commerce, this can bring realiability, ability to catch many errors (in retrieving the file, parsing, inserting into the db etc) and repeat what went wrong if necessary. Apart from the connector itself, there is already some excellent work in progress by Akretion, at: * https://code.launchpad.net/~akretion-team/file-exchange/file-exchange * https://code.launchpad.net/~akretion-team/+junk/logistic-center * https://code.launchpad.net/~akretion-team/+junk/poc-import-data I gave those modules a try, and here are some points I wish to discuss. First of all, well done. * If I inderstand correctly, we are spawning a job to import each line of the file. That line is stored in the "buffer", associated to the job. Still, some data needs to be imported in chunks (invoices with lines, for example). That needs a kind of multi-line buffer, probably not with a JSON dictionary "data" field, but something like a list of lists (i.e. a table). What do you think?
You mean the JSON being a list instead of a dictionary? How are represented the data given to the load_data OpenERP methods?
On a technical point, if I can understand why a 'buffer_id' field has been added on the queue.job (for the UI, easy access to the buffer), I dislike that. A job should stay agnostic to their function, and stay as close as possible to jobs queues based on MQs (ideally replaceable by RabbitMQ for instance). Another reason is that the buffer_id is already there in the stored arguments for the function (duplicated data). If we wanted to have an access to the buffer from the jobs, I would prefer to be able to associate an action to a job, the action having the knowledge on how to parse the arguments and open the record.
* The file-exchange modules has the ability to move imported files to the "archive" folder. Should we have also a place to move the files we can't read?
I think this is is something that should definitely be taken in account. We could also need a "failures" folder that contains the files with errors or lines with errors (according to the atomicity of the entire file / lines). It can be necessary for the sender of the files to have theses files so it can correct them. I don't know how and if it can be handled though.
Thanks! -- Guewen Baconnier Business Solutions Software Developer Camptocamp SA PSE A, CH-1015 Lausanne Phone: +41 21 619 10 39 Office: +41 21 619 10 10 http://www.camptocamp.com/
| Thread Previous • Date Previous • Date Next • Thread Next |
