dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

Jason,

I fully understand that, and it's only done infrequently (and outside
office hours if it's a production instance).

The 75MB xml-zip file that I just uploaded had only around 8 mill records -
the upload took 20 minutes and the actual import around 13 minutes. No
problemo...

Regards
Calle

On 3 December 2015 at 20:12, Jason Pickering <jason.p.pickering@xxxxxxxxx>
wrote:

> Hi Calle,
>
> I think you  would want to be very careful with this. If you change the
> maximum file size to 200 MB, this could potentially be an unzipped file of
> several (tens) of gigabytes , or several million rows of data. This could
> put significant stress on the server, and is the entire point of the
> restriction really, to prevent huge uploads from being imported.  If you
> have limited who can upload data to the server, it may be OK, but just be
> aware that a zip file of 200 MB, can be much much larger (by an order or
> magnitude or two), and result in a very long process.
>
> Regards,
>
> Jason
>
>
>
> On Thu, Dec 3, 2015, 18:52 Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:
>
>> Hi
>>
>> We are now standardising on 200M
>>
>> Regards
>> Calle
>>
>> On 3 December 2015 at 17:38, Alan Ivey <aivey@xxxxxxxxxxxxxx> wrote:
>>
>>> Also, it's worth noting that the default for "client_max_body_size" is
>>> only 1 MB:
>>> http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size .
>>> It will need to be increased on most deployments of DHIS2.
>>>
>>> On Thu, Dec 3, 2015 at 9:09 AM, Lars Helge Øverland <larshelge@xxxxxxxxx
>>> > wrote:
>>>
>>>> If you are indeed using nginx, the "client_max_body_size" directive is
>>>> part of the installation docs example, can be increased as appropriate:
>>>>
>>>>
>>>> https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s04.html#d5e575
>>>>
>>>>
>>>>
>>>> Lars
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Dec 3, 2015 at 3:02 PM, Lars Helge Øverland <
>>>> larshelge@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Calle,
>>>>>
>>>>> I think this depends on the web server configuration. One can
>>>>> configure max file size for uploads in both the proxy (nginx, apache) and
>>>>> servlet container (tomcat
>>>>> <http://stackoverflow.com/questions/2947683/httprequest-maximum-allowable-size-in-tomcat>
>>>>> ).
>>>>>
>>>>> On nginx
>>>>> <http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size>
>>>>> the directive is:
>>>>>
>>>>> client_max_body_size 200M;
>>>>>
>>>>>
>>>>> regards,
>>>>>
>>>>> Lars
>>>>>
>>>>> On Thu, Dec 3, 2015 at 2:44 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have found that there is a limitation in file size when importing
>>>>>> data into our SERVER-based instances, while I have found no equivalent
>>>>>> limitation when importing large data files (e.g. XML format) into an
>>>>>> equivalent localhost instance. A few key aspects:
>>>>>>
>>>>>> 1. Both the server and localhost are running the latest version of
>>>>>> 2.20
>>>>>> 2. Both are running java 8 64 bits and tomcat 8.026 or 8.029
>>>>>> 3. Localhost tomcat has 4GB (min) and 8GB (max) allocated
>>>>>> 4. The server instance (running under Ubuntu Linux) has ~5.3GB RAM,
>>>>>> but increasing/decreasing RAM has no effect on the issue.
>>>>>>
>>>>>> The problem is related to the upload process.
>>>>>>
>>>>>> Example:
>>>>>> When importing a 75MB data file with around 8 mill data records (XML,
>>>>>> zipped) on localhost, the initial upload step is almost instantaneous (2-3
>>>>>> seconds) and then the actual import starts (takes about 10 minutes overall).
>>>>>>
>>>>>> When importing the same file to the equivalent instance on the
>>>>>> server, it takes around 30 seconds to reach 2% upload and then the upload
>>>>>> re-starts at 0% - this goes on ad infinitum.
>>>>>>
>>>>>> Smaller files - e.g. 10-20MB - will maybe import 15-20%, then reset
>>>>>> to 0% and start over.
>>>>>>
>>>>>> It seems to me that the problem is related to the DHIS2 web server
>>>>>> configuration, it do not allow sufficient time for the upload to happen.
>>>>>>
>>>>>> Any indications of how to fix this would be appreciated. While
>>>>>> dumping the server instance into localhost, import the data, and then
>>>>>> upload/restore the instance does work, it is a pain in the b....
>>>>>>
>>>>>> Regards from a sunny Cape Town
>>>>>> Calle
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>> Calle Hedberg
>>>>>>
>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>
>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>
>>>>>> Cell: +27-82-853-5352
>>>>>>
>>>>>> Iridium SatPhone: +8816-315-19119
>>>>>>
>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>
>>>>>> Skype: calle_hedberg
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lars Helge Øverland
>>>>> Lead developer, DHIS 2
>>>>> University of Oslo
>>>>> Skype: larshelgeoverland
>>>>> http://www.dhis2.org <https://www.dhis2.org>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lars Helge Øverland
>>>> Lead developer, DHIS 2
>>>> University of Oslo
>>>> Skype: larshelgeoverland
>>>> http://www.dhis2.org <https://www.dhis2.org>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19119
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>


-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************