openerp-connector-community team mailing list archive

[Markus Schneider <markus.schneider@xxxxxxxxxx>]

Hi Nicolas,

How many connector-workers do you using?

If you have 4 cores, and only one is full then your multiprocessing is
not correct, se more here:

http://odoo-connector.com/guides/multiprocessing.html

Also tun the Postgres Database and give them more shared memory.

A other problem is how you link objects. Maybe your search is not
effizient. But i guess the problem is XML parsing, this is always slower
then csv-import or JSON.

Kind Regards

Markus

On 21.11.2014 16:57, Nicolas Clavier wrote:
> Hi folks,
> I need you advice on this :
> 
> I am running a custom developed connector that imports partner to a odoo
> server 8.0 (I tried that on a 7.0 too).
> The code is very similar to what the magento_connector does, I re-used
> almost exact same patterns.
> 
>   * I import only partners and addresses, (without fancy thinking around
>     the addresses => straight import, linked to partner),
>   * the sync is one way : import only, no export,
>   * the data source is an xml file :
>       o during a run, the xml is parsed into a list of dictionaries
>         which are cached in memory and accessed directly during backend
>         reads, this is fast.
> 
> 
> The first xml file has 10 000 partners with average 1 linked address
> each and many tags, and It takes more than 5h for the import run to
> complete . 
> The next file to process contains about 200 000 partners with linked
> parent company and addresses, I fear the worst ! ( = 23 days based on my
> calculations, this can’t be)
> Those imports are only initial imports, the average load should be few
> hundreds a day max on a daily basis.
> 
> The server is a stand-alone 4 cores x2.5ghz / 4 GB RAM VMWare machine on
> ubuntu 14.04 lts. 
> During the run :
> 
>   *  RAM is at 14% 
>   * 1 proc is 100% full 
> 
> 
> Tthe server is started with 9 workers which shouldn’t impact much since
> we’re dealing with a single request I suppose.
> It looks like the import mapper is slow.
> 
> So …  my question is : is there any way to speed things up ?
> 
> PS : I’m already
> reading http://fr.slideshare.net/openobject/performance2014-35689113
> 
> Thanks for your advises ))
> Nicolas 
> 
> 

-- 
Dipl.-Comp.-Math. Markus Schneider
Softwareentwickler

initOS GmbH & Co. KG
An der Eisenbahn 1
21224 Rosengarten

Mobil:   +49 (0)172 2303699
Phone:   +49 (0)4105 5615613
Fax:     +49 (0)4105 5615610

Email:   markus.schneider@xxxxxxxxxx
Web:     http://www.initos.com

Geschäftsführung:
Dipl. Wirt.-Inf. Frederik Kramer & Dipl.-Ing. (FH) Torsten Francke
Haftende Gesellschafterin: initOS Verwaltungs GmbH

Sitz der Gesellschaft: Rosengarten – Klecken
Amtsgericht Tostedt, HRA 201840
USt-IdNr: DE 275698169
Steuer-Nr: 15/205/21402