openerp-connector-community team mailing list archive

[Nicolas Clavier <nicoclavier@xxxxxxxxx>]

> http://odoo-connector.com/guides/multiprocessing.htmlr
Ho ! I missed reading this page .. (!?)

Meanwhile I can add that shared memory was already turned to 55% (2.25GB),
XML parsing is blazing fast (lxml does a wonderful job at it).
Now as to the linking, I respect the import_dependencies / after_import pattern but it is indeed pretty slow.
I’ll have to optimize that aspect.

Of course the load() method of the orm would be faster, I used to use it before, 
but it requires developing another tools which wasn’t the goal.
I’m going to try the workers tunning right away. 

Thanks for your advises,
regards
Nicolas 

> On 21 nov. 2014, at 17:14, Markus Schneider <markus.schneider@xxxxxxxxxx> wrote:
> 
> Hi Nicolas,
> 
> How many connector-workers do you using?
> 
> If you have 4 cores, and only one is full then your multiprocessing is
> not correct, se more here:
> 
> http://odoo-connector.com/guides/multiprocessing.html
> 
> Also tun the Postgres Database and give them more shared memory.
> 
> A other problem is how you link objects. Maybe your search is not
> effizient. But i guess the problem is XML parsing, this is always slower
> then csv-import or JSON.
> 
> Kind Regards
> 
> Markus
> 
> On 21.11.2014 16:57, Nicolas Clavier wrote:
>> Hi folks,
>> I need you advice on this :
>> 
>> I am running a custom developed connector that imports partner to a odoo
>> server 8.0 (I tried that on a 7.0 too).
>> The code is very similar to what the magento_connector does, I re-used
>> almost exact same patterns.
>> 
>>  * I import only partners and addresses, (without fancy thinking around
>>    the addresses => straight import, linked to partner),
>>  * the sync is one way : import only, no export,
>>  * the data source is an xml file :
>>      o during a run, the xml is parsed into a list of dictionaries
>>        which are cached in memory and accessed directly during backend
>>        reads, this is fast.
>> 
>> 
>> The first xml file has 10 000 partners with average 1 linked address
>> each and many tags, and It takes more than 5h for the import run to
>> complete . 
>> The next file to process contains about 200 000 partners with linked
>> parent company and addresses, I fear the worst ! ( = 23 days based on my
>> calculations, this can’t be)
>> Those imports are only initial imports, the average load should be few
>> hundreds a day max on a daily basis.
>> 
>> The server is a stand-alone 4 cores x2.5ghz / 4 GB RAM VMWare machine on
>> ubuntu 14.04 lts. 
>> During the run :
>> 
>>  *  RAM is at 14% 
>>  * 1 proc is 100% full 
>> 
>> 
>> Tthe server is started with 9 workers which shouldn’t impact much since
>> we’re dealing with a single request I suppose.
>> It looks like the import mapper is slow.
>> 
>> So …  my question is : is there any way to speed things up ?
>> 
>> PS : I’m already
>> reading http://fr.slideshare.net/openobject/performance2014-35689113
>> 
>> Thanks for your advises ))
>> Nicolas 
>> 
>> 
> 
> -- 
> Dipl.-Comp.-Math. Markus Schneider
> Softwareentwickler
> 
> initOS GmbH & Co. KG
> An der Eisenbahn 1
> 21224 Rosengarten
> 
> Mobil:   +49 (0)172 2303699
> Phone:   +49 (0)4105 5615613
> Fax:     +49 (0)4105 5615610
> 
> Email:   markus.schneider@xxxxxxxxxx
> Web:     http://www.initos.com
> 
> Geschäftsführung:
> Dipl. Wirt.-Inf. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke
> Haftende Gesellschafterin: initOS Verwaltungs GmbH
> 
> Sitz der Gesellschaft: Rosengarten – Klecken
> Amtsgericht Tostedt, HRA 201840
> USt-IdNr: DE 275698169
> Steuer-Nr: 15/205/21402