openerp-connector-community team mailing list archive

[Nicolas Clavier <nicoclavier@xxxxxxxxx>]

Hi folks,
I need you advice on this :

I am running a custom developed connector that imports partner to a odoo server 8.0 (I tried that on a 7.0 too).
The code is very similar to what the magento_connector does, I re-used almost exact same patterns.
I import only partners and addresses, (without fancy thinking around the addresses => straight import, linked to partner),
the sync is one way : import only, no export,
the data source is an xml file :
during a run, the xml is parsed into a list of dictionaries which are cached in memory and accessed directly during backend reads, this is fast.

The first xml file has 10 000 partners with average 1 linked address each and many tags, and It takes more than 5h for the import run to complete . 
The next file to process contains about 200 000 partners with linked parent company and addresses, I fear the worst ! ( = 23 days based on my calculations, this can’t be)
Those imports are only initial imports, the average load should be few hundreds a day max on a daily basis.

The server is a stand-alone 4 cores x2.5ghz / 4 GB RAM VMWare machine on ubuntu 14.04 lts. 
During the run :
 RAM is at 14% 
1 proc is 100% full 

Tthe server is started with 9 workers which shouldn’t impact much since we’re dealing with a single request I suppose.
It looks like the import mapper is slow.

So …  my question is : is there any way to speed things up ?

PS : I’m already reading http://fr.slideshare.net/openobject/performance2014-35689113

Thanks for your advises ))
Nicolas