dhis2-devs team mailing list archive

[Timothy Harding <hardingt@xxxxxxxxx>]

Thanks everyone, I very much appreciate the help. I will give both of these
a shot! Node.js can be damn confusing at times.

Timothy Harding
RPCV Vanuatu
Skype: hardingt@xxxxxxxxx
+1 (541) 632-6623

On Tue, Aug 25, 2015 at 1:57 AM, Lars Helge Øverland <larshelge@xxxxxxxxx>
wrote:

> Hi Tim,
>
> you can use promises to handle the async requests, or simply use
> synchronous requests. No definite answers here but I find using synchronous
> requests easier to work with and user interface responsiveness is not an
> issue for server side ad-hoc scripts like this.
>
> The urllib-sync library will wrap urllib and provide sync behavior:
>
> https://www.npmjs.com/package/urllib-sync
>
> Install with npm:
>
> npm install urllib-sync
>
> Import:
>
> var urlsync = require('urllib-sync');
>
> Then do synchronous requests/responses like this:
>
> var response = urlsync.request(url, postOptions);
>
>
>
>
> Lars
>
>
>
>
>
>
>
> On Tue, Aug 25, 2015 at 10:35 AM, Mark Polak <mark@xxxxxxxxxxxxxxx> wrote:
>
>> Hey Timothy,
>>
>> Is there a reason why you would not want to wait for the response? You
>> have figured out you're sort of flooding the server with a bunch of
>> requests.
>>
>> You can still send them in batches of say 5, wait till those are done
>> then send the next 5, etc.
>>
>> Node now supports Promises so you could transform your requests into
>> promises and handle them in batches by using Promise.all().
>>
>> The code below i just wrote up quickly without running it so you might
>> need to see if it works and adjust it, but it was more to give you an idea.
>>
>> Regards,
>>
>> Mark
>>
>> function doRequests(dataElementsToUpdate) {
>>     //Map over the items
>>     return dataElementsToUpdate.map(function (dataElementToUpdate) {
>>         //Create a promise for each item and call resolve when that
>> request is done.
>>         return new Promise(function (resolve, reject) {
>>
>>             var postData = JSON.stringify(dataElementsToUpdate[i]);
>>                 serverConfigPOST.dhis2server.path =
>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].
>> object.id;
>>                 serverConfigPOST.dhis2server.headers = {
>>                     'Content-Type': 'application/json',
>>                     'Content-Length': postData.length
>>                 };
>>                 console.log(serverConfigPOST);
>>
>>                 var req = https.request(serverConfigPOST.dhis2server,
>> function(res) {
>>                     console.log('STATUS: ' + res.statusCode);
>>                     console.log('HEADERS: ' +
>> JSON.stringify(res.headers));
>>                     res.setEncoding('utf8');
>>                     res.on('data', function (chunk) {
>>                         console.log('BODY: ' + chunk);
>>                     });
>>                     res.on('end', function () {
>>                         resolve({item: dataElementsToUpdate[i].object,
>> success: true);
>>                     });
>>                 });
>>
>>                 req.on('error', function(e) {
>>                     //console.log('problem with request: ' + e.message);
>>                     resolve({item: dataElementsToUpdate[i].object,
>> success: false, message: e.message);
>>                 });
>>         });
>>     });
>> }
>>
>> function updateSharingInBatches() {
>>     //Pick 5 elements to process next
>>     var itemsToProcess = dataElementsToUpdate.splice(0, 5);
>>
>>     //Only continue if there are more elements
>>     if (itemsToProcess.length) {
>>         //Wait til the passed batch is done and excute the next after
>>         return Promise.all(doRequests(itemsToProcess))
>>             .then(function (responses) {
>>                 updateSharingInBatches(dataElementsToUpdate);
>>             })
>>             //If you want to
>>             .catch(function (responses) {
>>                 console.log('One or more did not work');
>>                 //If you wish to continue with the batches you can just
>> call
>>                 //updateSharingInBatches(dataElementsToUpdate);
>>                 //Or you can try run the failed one again. :)
>>             })
>>     }
>> }
>>
>> //Run the whole thing
>> updateSharingInBatches(dataElementsToUpdate);
>>
>>
>>
>> On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding <hardingt@xxxxxxxxx>
>> wrote:
>>
>>> Thanks Pierre,
>>>
>>> Not sure how I would set up the recursive loop in javascript, I had
>>> heard about async from stack overflow as well, I was thinking of trying
>>> that, but I kind of like how quickly this works, not needing to wait for
>>> the response from one request before sending the next. The DHIS2 hardware I
>>> am working with just doesn't seem to keep up after a certain number of
>>> requests however, hence my attempt to slow them down.
>>>
>>> Timothy Harding
>>> RPCV Vanuatu
>>> Skype: hardingt@xxxxxxxxx
>>> +1 (541) 632-6623
>>>
>>> On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane <pierre@xxxxxxxxx> wrote:
>>>
>>>> might be best to try a recursive loop using setInterval or
>>>> setTimeout with a configurable value (I.e. a method calling itself) .  or
>>>> you could look at the async library to make things synchronous .
>>>>
>>>>
>>>> On Monday, August 24, 2015, Timothy Harding <hardingt@xxxxxxxxx> wrote:
>>>>
>>>>> Hello DHIS2 Devs,
>>>>>
>>>>> I've got a script that works pretty well for updating a bunch of
>>>>> sharing settings for various objects using node.js. The only problem is, I
>>>>> am getting timeouts when the object update counts get too high (500+).
>>>>>
>>>>> I am a bit new to the asynchronous nature of node.js, and I like that
>>>>> I can bundle a lot of updates into one without needing to worry that I need
>>>>> a response for one before sending the request for the next. The downside
>>>>> is, I have no idea where to put the sleep timer to slow down the volume of
>>>>> requests. Everywhere I've put it so far increases the timeouts, rather than
>>>>> decreases them. I know there are some node.js experts out there on the
>>>>> DHIS2 dev list so, perhaps someone has run into something similar:
>>>>>
>>>>> var serverConfigPOST = serverConfig;
>>>>> serverConfigPOST.dhis2server.method = "POST";
>>>>> var i=0;
>>>>> for(i=0; i < dataElementsToUpdate.length; i++){
>>>>>
>>>>>                             Tried here
>>>>> var postData = JSON.stringify(dataElementsToUpdate[i]);
>>>>> serverConfigPOST.dhis2server.path =
>>>>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].
>>>>> object.id;
>>>>> serverConfigPOST.dhis2server.headers = {
>>>>> 'Content-Type': 'application/json',
>>>>> 'Content-Length': postData.length
>>>>> };
>>>>> console.log(serverConfigPOST);
>>>>> var req = https.request(serverConfigPOST.dhis2server, function(res) {
>>>>> console.log('STATUS: ' + res.statusCode);
>>>>> console.log('HEADERS: ' + JSON.stringify(res.headers));
>>>>> res.setEncoding('utf8');
>>>>> res.on('data', function (chunk) {
>>>>> console.log('BODY: ' + chunk);
>>>>> });
>>>>> });
>>>>> req.on('error', function(e) {
>>>>> //console.log('problem with request: ' + e.message);
>>>>> });
>>>>> // write data to request body
>>>>> req.write(postData);
>>>>> req.end();
>>>>>
>>>>>                             Tried here
>>>>> // if(i==20){
>>>>> // break;
>>>>> // }
>>>>> }
>>>>>
>>>>> Not quite sure how to slow this one down, I may just have to run/rerun
>>>>> it with different starting and stopping i values in the mean time.
>>>>>
>>>>> Timothy Harding
>>>>> RPCV Vanuatu
>>>>> Skype: hardingt@xxxxxxxxx
>>>>> +1 (541) 632-6623
>>>>>
>>>>
>>>>
>>>> --
>>>> Pierre Dane
>>>>
>>>> Development Manager
>>>> Jembi Health Systems
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Lars Helge Øverland
> Lead developer, DHIS 2
> University of Oslo
> Skype: larshelgeoverland
> http://www.dhis2.org <https://www.dhis2.org>
>
>