dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Calle,

I think you  would want to be very careful with this. If you change the
maximum file size to 200 MB, this could potentially be an unzipped file of
several (tens) of gigabytes , or several million rows of data. This could
put significant stress on the server, and is the entire point of the
restriction really, to prevent huge uploads from being imported.  If you
have limited who can upload data to the server, it may be OK, but just be
aware that a zip file of 200 MB, can be much much larger (by an order or
magnitude or two), and result in a very long process.

Regards,

Jason



On Thu, Dec 3, 2015, 18:52 Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:

> Hi
>
> We are now standardising on 200M
>
> Regards
> Calle
>
> On 3 December 2015 at 17:38, Alan Ivey <aivey@xxxxxxxxxxxxxx> wrote:
>
>> Also, it's worth noting that the default for "client_max_body_size" is
>> only 1 MB:
>> http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size .
>> It will need to be increased on most deployments of DHIS2.
>>
>> On Thu, Dec 3, 2015 at 9:09 AM, Lars Helge Øverland <larshelge@xxxxxxxxx>
>> wrote:
>>
>>> If you are indeed using nginx, the "client_max_body_size" directive is
>>> part of the installation docs example, can be increased as appropriate:
>>>
>>>
>>> https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s04.html#d5e575
>>>
>>>
>>>
>>> Lars
>>>
>>>
>>>
>>>
>>> On Thu, Dec 3, 2015 at 3:02 PM, Lars Helge Øverland <larshelge@xxxxxxxxx
>>> > wrote:
>>>
>>>> Hi Calle,
>>>>
>>>> I think this depends on the web server configuration. One can configure
>>>> max file size for uploads in both the proxy (nginx, apache) and servlet
>>>> container (tomcat
>>>> <http://stackoverflow.com/questions/2947683/httprequest-maximum-allowable-size-in-tomcat>
>>>> ).
>>>>
>>>> On nginx
>>>> <http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size>
>>>> the directive is:
>>>>
>>>> client_max_body_size 200M;
>>>>
>>>>
>>>> regards,
>>>>
>>>> Lars
>>>>
>>>> On Thu, Dec 3, 2015 at 2:44 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have found that there is a limitation in file size when importing
>>>>> data into our SERVER-based instances, while I have found no equivalent
>>>>> limitation when importing large data files (e.g. XML format) into an
>>>>> equivalent localhost instance. A few key aspects:
>>>>>
>>>>> 1. Both the server and localhost are running the latest version of 2.20
>>>>> 2. Both are running java 8 64 bits and tomcat 8.026 or 8.029
>>>>> 3. Localhost tomcat has 4GB (min) and 8GB (max) allocated
>>>>> 4. The server instance (running under Ubuntu Linux) has ~5.3GB RAM,
>>>>> but increasing/decreasing RAM has no effect on the issue.
>>>>>
>>>>> The problem is related to the upload process.
>>>>>
>>>>> Example:
>>>>> When importing a 75MB data file with around 8 mill data records (XML,
>>>>> zipped) on localhost, the initial upload step is almost instantaneous (2-3
>>>>> seconds) and then the actual import starts (takes about 10 minutes overall).
>>>>>
>>>>> When importing the same file to the equivalent instance on the server,
>>>>> it takes around 30 seconds to reach 2% upload and then the upload re-starts
>>>>> at 0% - this goes on ad infinitum.
>>>>>
>>>>> Smaller files - e.g. 10-20MB - will maybe import 15-20%, then reset to
>>>>> 0% and start over.
>>>>>
>>>>> It seems to me that the problem is related to the DHIS2 web server
>>>>> configuration, it do not allow sufficient time for the upload to happen.
>>>>>
>>>>> Any indications of how to fix this would be appreciated. While dumping
>>>>> the server instance into localhost, import the data, and then
>>>>> upload/restore the instance does work, it is a pain in the b....
>>>>>
>>>>> Regards from a sunny Cape Town
>>>>> Calle
>>>>>
>>>>> *******************************************
>>>>>
>>>>> Calle Hedberg
>>>>>
>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>
>>>>> Tel/fax (home): +27-21-685-6472
>>>>>
>>>>> Cell: +27-82-853-5352
>>>>>
>>>>> Iridium SatPhone: +8816-315-19119
>>>>>
>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>
>>>>> Skype: calle_hedberg
>>>>>
>>>>> *******************************************
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lars Helge Øverland
>>>> Lead developer, DHIS 2
>>>> University of Oslo
>>>> Skype: larshelgeoverland
>>>> http://www.dhis2.org <https://www.dhis2.org>
>>>>
>>>>
>>>
>>>
>>> --
>>> Lars Helge Øverland
>>> Lead developer, DHIS 2
>>> University of Oslo
>>> Skype: larshelgeoverland
>>> http://www.dhis2.org <https://www.dhis2.org>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>