dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

Hi Tim,

you can use promises to handle the async requests, or simply use
synchronous requests. No definite answers here but I find using synchronous
requests easier to work with and user interface responsiveness is not an
issue for server side ad-hoc scripts like this.

The urllib-sync library will wrap urllib and provide sync behavior:

https://www.npmjs.com/package/urllib-sync

Install with npm:

npm install urllib-sync

Import:

var urlsync = require('urllib-sync');

Then do synchronous requests/responses like this:

var response = urlsync.request(url, postOptions);




Lars







On Tue, Aug 25, 2015 at 10:35 AM, Mark Polak <mark@xxxxxxxxxxxxxxx> wrote:

> Hey Timothy,
>
> Is there a reason why you would not want to wait for the response? You
> have figured out you're sort of flooding the server with a bunch of
> requests.
>
> You can still send them in batches of say 5, wait till those are done then
> send the next 5, etc.
>
> Node now supports Promises so you could transform your requests into
> promises and handle them in batches by using Promise.all().
>
> The code below i just wrote up quickly without running it so you might
> need to see if it works and adjust it, but it was more to give you an idea.
>
> Regards,
>
> Mark
>
> function doRequests(dataElementsToUpdate) {
>     //Map over the items
>     return dataElementsToUpdate.map(function (dataElementToUpdate) {
>         //Create a promise for each item and call resolve when that
> request is done.
>         return new Promise(function (resolve, reject) {
>
>             var postData = JSON.stringify(dataElementsToUpdate[i]);
>                 serverConfigPOST.dhis2server.path =
> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].
> object.id;
>                 serverConfigPOST.dhis2server.headers = {
>                     'Content-Type': 'application/json',
>                     'Content-Length': postData.length
>                 };
>                 console.log(serverConfigPOST);
>
>                 var req = https.request(serverConfigPOST.dhis2server,
> function(res) {
>                     console.log('STATUS: ' + res.statusCode);
>                     console.log('HEADERS: ' + JSON.stringify(res.headers));
>                     res.setEncoding('utf8');
>                     res.on('data', function (chunk) {
>                         console.log('BODY: ' + chunk);
>                     });
>                     res.on('end', function () {
>                         resolve({item: dataElementsToUpdate[i].object,
> success: true);
>                     });
>                 });
>
>                 req.on('error', function(e) {
>                     //console.log('problem with request: ' + e.message);
>                     resolve({item: dataElementsToUpdate[i].object,
> success: false, message: e.message);
>                 });
>         });
>     });
> }
>
> function updateSharingInBatches() {
>     //Pick 5 elements to process next
>     var itemsToProcess = dataElementsToUpdate.splice(0, 5);
>
>     //Only continue if there are more elements
>     if (itemsToProcess.length) {
>         //Wait til the passed batch is done and excute the next after
>         return Promise.all(doRequests(itemsToProcess))
>             .then(function (responses) {
>                 updateSharingInBatches(dataElementsToUpdate);
>             })
>             //If you want to
>             .catch(function (responses) {
>                 console.log('One or more did not work');
>                 //If you wish to continue with the batches you can just
> call
>                 //updateSharingInBatches(dataElementsToUpdate);
>                 //Or you can try run the failed one again. :)
>             })
>     }
> }
>
> //Run the whole thing
> updateSharingInBatches(dataElementsToUpdate);
>
>
>
> On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding <hardingt@xxxxxxxxx>
> wrote:
>
>> Thanks Pierre,
>>
>> Not sure how I would set up the recursive loop in javascript, I had heard
>> about async from stack overflow as well, I was thinking of trying that, but
>> I kind of like how quickly this works, not needing to wait for the response
>> from one request before sending the next. The DHIS2 hardware I am working
>> with just doesn't seem to keep up after a certain number of requests
>> however, hence my attempt to slow them down.
>>
>> Timothy Harding
>> RPCV Vanuatu
>> Skype: hardingt@xxxxxxxxx
>> +1 (541) 632-6623
>>
>> On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane <pierre@xxxxxxxxx> wrote:
>>
>>> might be best to try a recursive loop using setInterval or
>>> setTimeout with a configurable value (I.e. a method calling itself) .  or
>>> you could look at the async library to make things synchronous .
>>>
>>>
>>> On Monday, August 24, 2015, Timothy Harding <hardingt@xxxxxxxxx> wrote:
>>>
>>>> Hello DHIS2 Devs,
>>>>
>>>> I've got a script that works pretty well for updating a bunch of
>>>> sharing settings for various objects using node.js. The only problem is, I
>>>> am getting timeouts when the object update counts get too high (500+).
>>>>
>>>> I am a bit new to the asynchronous nature of node.js, and I like that I
>>>> can bundle a lot of updates into one without needing to worry that I need a
>>>> response for one before sending the request for the next. The downside is,
>>>> I have no idea where to put the sleep timer to slow down the volume of
>>>> requests. Everywhere I've put it so far increases the timeouts, rather than
>>>> decreases them. I know there are some node.js experts out there on the
>>>> DHIS2 dev list so, perhaps someone has run into something similar:
>>>>
>>>> var serverConfigPOST = serverConfig;
>>>> serverConfigPOST.dhis2server.method = "POST";
>>>> var i=0;
>>>> for(i=0; i < dataElementsToUpdate.length; i++){
>>>>
>>>>                             Tried here
>>>> var postData = JSON.stringify(dataElementsToUpdate[i]);
>>>> serverConfigPOST.dhis2server.path =
>>>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].
>>>> object.id;
>>>> serverConfigPOST.dhis2server.headers = {
>>>> 'Content-Type': 'application/json',
>>>> 'Content-Length': postData.length
>>>> };
>>>> console.log(serverConfigPOST);
>>>> var req = https.request(serverConfigPOST.dhis2server, function(res) {
>>>> console.log('STATUS: ' + res.statusCode);
>>>> console.log('HEADERS: ' + JSON.stringify(res.headers));
>>>> res.setEncoding('utf8');
>>>> res.on('data', function (chunk) {
>>>> console.log('BODY: ' + chunk);
>>>> });
>>>> });
>>>> req.on('error', function(e) {
>>>> //console.log('problem with request: ' + e.message);
>>>> });
>>>> // write data to request body
>>>> req.write(postData);
>>>> req.end();
>>>>
>>>>                             Tried here
>>>> // if(i==20){
>>>> // break;
>>>> // }
>>>> }
>>>>
>>>> Not quite sure how to slow this one down, I may just have to run/rerun
>>>> it with different starting and stopping i values in the mean time.
>>>>
>>>> Timothy Harding
>>>> RPCV Vanuatu
>>>> Skype: hardingt@xxxxxxxxx
>>>> +1 (541) 632-6623
>>>>
>>>
>>>
>>> --
>>> Pierre Dane
>>>
>>> Development Manager
>>> Jembi Health Systems
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org <https://www.dhis2.org>