openerp-connector-community team mailing list archive

[Nicolas Clavier <nicoclavier@xxxxxxxxx>]

Hi,

I switched to multi-process according to this article <http://odoo-connector.com/guides/multiprocessing.htmlr>, but met various errors (see listing below).

All my import jobs are directBatchImports, (for some precise reason I cannot use the delayed jobs yet).
Would it be why only 1 python process is full at 100% while the other 8 are sleeping ?
Or is it to do with the openerp-connector-worker that’s not working properly ?

Following are some errors I met during setup of multiprocess :
Error message when running the openerp-connector-worker script:
https://github.com/OCA/connector/issues/22 <https://github.com/OCA/connector/issues/22>
to which I used the workaround of surrounding the db_worker_id with try / except block
While initializing, the file connector/queue/jop.py line 11 reported : 
"Attempted relative import beyond toplevel package “ when importing from ..exception 
this was solved modifying the file using absolute pass (likely to be a conf issue on my side)
Postgres message : 
bad query: delete from queue_worker where id IN (544, 545, 546, 547, 548, 549, ...)
[…]
TransactionRollbackError: ERROR:  could not serialize an access due to concurrente update (translated from french)
I can’t reproduce the message again, no idea.

Thanks for your lights,
Nicolas

> On 21 nov. 2014, at 18:48, Nicolas Clavier <nicoclavier@xxxxxxxxx <mailto:nicoclavier@xxxxxxxxx>> wrote:
> 
>> http://odoo-connector.com/guides/multiprocessing.htmlr <http://odoo-connector.com/guides/multiprocessing.htmlr>
> Ho ! I missed reading this page .. (!?)
> 
> Meanwhile I can add that shared memory was already turned to 55% (2.25GB),
> XML parsing is blazing fast (lxml does a wonderful job at it).
> Now as to the linking, I respect the import_dependencies / after_import pattern but it is indeed pretty slow.
> I’ll have to optimize that aspect.
> 
> Of course the load() method of the orm would be faster, I used to use it before, 
> but it requires developing another tools which wasn’t the goal.
> I’m going to try the workers tunning right away. 
> 
> Thanks for your advises,
> regards
> Nicolas 
> 
>> On 21 nov. 2014, at 17:14, Markus Schneider <markus.schneider@xxxxxxxxxx <mailto:markus.schneider@xxxxxxxxxx>> wrote:
>> 
>> Hi Nicolas,
>> 
>> How many connector-workers do you using?
>> 
>> If you have 4 cores, and only one is full then your multiprocessing is
>> not correct, se more here:
>> 
>> http://odoo-connector.com/guides/multiprocessing.html <http://odoo-connector.com/guides/multiprocessing.html>
>> 
>> Also tun the Postgres Database and give them more shared memory.
>> 
>> A other problem is how you link objects. Maybe your search is not
>> effizient. But i guess the problem is XML parsing, this is always slower
>> then csv-import or JSON.
>> 
>> Kind Regards
>> 
>> Markus
>> 
>> On 21.11.2014 16:57, Nicolas Clavier wrote:
>>> Hi folks,
>>> I need you advice on this :
>>> 
>>> I am running a custom developed connector that imports partner to a odoo
>>> server 8.0 (I tried that on a 7.0 too).
>>> The code is very similar to what the magento_connector does, I re-used
>>> almost exact same patterns.
>>> 
>>> * I import only partners and addresses, (without fancy thinking around
>>>   the addresses => straight import, linked to partner),
>>> * the sync is one way : import only, no export,
>>> * the data source is an xml file :
>>>     o during a run, the xml is parsed into a list of dictionaries
>>>       which are cached in memory and accessed directly during backend
>>>       reads, this is fast.
>>> 
>>> 
>>> The first xml file has 10 000 partners with average 1 linked address
>>> each and many tags, and It takes more than 5h for the import run to
>>> complete . 
>>> The next file to process contains about 200 000 partners with linked
>>> parent company and addresses, I fear the worst ! ( = 23 days based on my
>>> calculations, this can’t be)
>>> Those imports are only initial imports, the average load should be few
>>> hundreds a day max on a daily basis.
>>> 
>>> The server is a stand-alone 4 cores x2.5ghz / 4 GB RAM VMWare machine on
>>> ubuntu 14.04 lts. 
>>> During the run :
>>> 
>>> *  RAM is at 14% 
>>> * 1 proc is 100% full 
>>> 
>>> 
>>> Tthe server is started with 9 workers which shouldn’t impact much since
>>> we’re dealing with a single request I suppose.
>>> It looks like the import mapper is slow.
>>> 
>>> So …  my question is : is there any way to speed things up ?
>>> 
>>> PS : I’m already
>>> reading http://fr.slideshare.net/openobject/performance2014-35689113 <http://fr.slideshare.net/openobject/performance2014-35689113>
>>> 
>>> Thanks for your advises ))
>>> Nicolas 
>>> 
>>> 
>> 
>> -- 
>> Dipl.-Comp.-Math. Markus Schneider
>> Softwareentwickler
>> 
>> initOS GmbH & Co. KG
>> An der Eisenbahn 1
>> 21224 Rosengarten
>> 
>> Mobil:   +49 (0)172 2303699
>> Phone:   +49 (0)4105 5615613
>> Fax:     +49 (0)4105 5615610
>> 
>> Email:   markus.schneider@xxxxxxxxxx <mailto:markus.schneider@xxxxxxxxxx>
>> Web:     http://www.initos.com <http://www.initos.com/>
>> 
>> Geschäftsführung:
>> Dipl. Wirt.-Inf. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke
>> Haftende Gesellschafterin: initOS Verwaltungs GmbH
>> 
>> Sitz der Gesellschaft: Rosengarten – Klecken
>> Amtsgericht Tostedt, HRA 201840
>> USt-IdNr: DE 275698169
>> Steuer-Nr: 15/205/21402
>