dhis2-devs team mailing list archive
Mailing list archive
Re: Importing large data files - server settings
If you are indeed using nginx, the "client_max_body_size" directive is part
of the installation docs example, can be increased as appropriate:
On Thu, Dec 3, 2015 at 3:02 PM, Lars Helge Øverland <larshelge@xxxxxxxxx>
> Hi Calle,
> I think this depends on the web server configuration. One can configure
> max file size for uploads in both the proxy (nginx, apache) and servlet
> container (tomcat
> On nginx
> the directive is:
> client_max_body_size 200M;
> On Thu, Dec 3, 2015 at 2:44 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>> I have found that there is a limitation in file size when importing data
>> into our SERVER-based instances, while I have found no equivalent
>> limitation when importing large data files (e.g. XML format) into an
>> equivalent localhost instance. A few key aspects:
>> 1. Both the server and localhost are running the latest version of 2.20
>> 2. Both are running java 8 64 bits and tomcat 8.026 or 8.029
>> 3. Localhost tomcat has 4GB (min) and 8GB (max) allocated
>> 4. The server instance (running under Ubuntu Linux) has ~5.3GB RAM, but
>> increasing/decreasing RAM has no effect on the issue.
>> The problem is related to the upload process.
>> When importing a 75MB data file with around 8 mill data records (XML,
>> zipped) on localhost, the initial upload step is almost instantaneous (2-3
>> seconds) and then the actual import starts (takes about 10 minutes overall).
>> When importing the same file to the equivalent instance on the server, it
>> takes around 30 seconds to reach 2% upload and then the upload re-starts at
>> 0% - this goes on ad infinitum.
>> Smaller files - e.g. 10-20MB - will maybe import 15-20%, then reset to 0%
>> and start over.
>> It seems to me that the problem is related to the DHIS2 web server
>> configuration, it do not allow sufficient time for the upload to happen.
>> Any indications of how to fix this would be appreciated. While dumping
>> the server instance into localhost, import the data, and then
>> upload/restore the instance does work, it is a pain in the b....
>> Regards from a sunny Cape Town
>> Calle Hedberg
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>> Tel/fax (home): +27-21-685-6472
>> Cell: +27-82-853-5352
>> Iridium SatPhone: +8816-315-19119
>> Email: calle.hedberg@xxxxxxxxx
>> Skype: calle_hedberg
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help : https://help.launchpad.net/ListHelp
> Lars Helge Øverland
> Lead developer, DHIS 2
> University of Oslo
> Skype: larshelgeoverland
> http://www.dhis2.org <https://www.dhis2.org>
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo