dhis2-devs team mailing list archive

[Mark Polak <mark@xxxxxxxxxxxxxxx>]

Hey Timothy,

Is there a reason why you would not want to wait for the response? You have
figured out you're sort of flooding the server with a bunch of requests.

You can still send them in batches of say 5, wait till those are done then
send the next 5, etc.

Node now supports Promises so you could transform your requests into
promises and handle them in batches by using Promise.all().

The code below i just wrote up quickly without running it so you might need
to see if it works and adjust it, but it was more to give you an idea.

Regards,

Mark

function doRequests(dataElementsToUpdate) {
    //Map over the items
    return dataElementsToUpdate.map(function (dataElementToUpdate) {
        //Create a promise for each item and call resolve when that request
is done.
        return new Promise(function (resolve, reject) {

            var postData = JSON.stringify(dataElementsToUpdate[i]);
                serverConfigPOST.dhis2server.path =
"/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].object.id
;
                serverConfigPOST.dhis2server.headers = {
                    'Content-Type': 'application/json',
                    'Content-Length': postData.length
                };
                console.log(serverConfigPOST);

                var req = https.request(serverConfigPOST.dhis2server,
function(res) {
                    console.log('STATUS: ' + res.statusCode);
                    console.log('HEADERS: ' + JSON.stringify(res.headers));
                    res.setEncoding('utf8');
                    res.on('data', function (chunk) {
                        console.log('BODY: ' + chunk);
                    });
                    res.on('end', function () {
                        resolve({item: dataElementsToUpdate[i].object,
success: true);
                    });
                });

                req.on('error', function(e) {
                    //console.log('problem with request: ' + e.message);
                    resolve({item: dataElementsToUpdate[i].object, success:
false, message: e.message);
                });
        });
    });
}

function updateSharingInBatches() {
    //Pick 5 elements to process next
    var itemsToProcess = dataElementsToUpdate.splice(0, 5);

    //Only continue if there are more elements
    if (itemsToProcess.length) {
        //Wait til the passed batch is done and excute the next after
        return Promise.all(doRequests(itemsToProcess))
            .then(function (responses) {
                updateSharingInBatches(dataElementsToUpdate);
            })
            //If you want to
            .catch(function (responses) {
                console.log('One or more did not work');
                //If you wish to continue with the batches you can just call
                //updateSharingInBatches(dataElementsToUpdate);
                //Or you can try run the failed one again. :)
            })
    }
}

//Run the whole thing
updateSharingInBatches(dataElementsToUpdate);



On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding <hardingt@xxxxxxxxx> wrote:

> Thanks Pierre,
>
> Not sure how I would set up the recursive loop in javascript, I had heard
> about async from stack overflow as well, I was thinking of trying that, but
> I kind of like how quickly this works, not needing to wait for the response
> from one request before sending the next. The DHIS2 hardware I am working
> with just doesn't seem to keep up after a certain number of requests
> however, hence my attempt to slow them down.
>
> Timothy Harding
> RPCV Vanuatu
> Skype: hardingt@xxxxxxxxx
> +1 (541) 632-6623
>
> On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane <pierre@xxxxxxxxx> wrote:
>
>> might be best to try a recursive loop using setInterval or
>> setTimeout with a configurable value (I.e. a method calling itself) .  or
>> you could look at the async library to make things synchronous .
>>
>>
>> On Monday, August 24, 2015, Timothy Harding <hardingt@xxxxxxxxx> wrote:
>>
>>> Hello DHIS2 Devs,
>>>
>>> I've got a script that works pretty well for updating a bunch of sharing
>>> settings for various objects using node.js. The only problem is, I am
>>> getting timeouts when the object update counts get too high (500+).
>>>
>>> I am a bit new to the asynchronous nature of node.js, and I like that I
>>> can bundle a lot of updates into one without needing to worry that I need a
>>> response for one before sending the request for the next. The downside is,
>>> I have no idea where to put the sleep timer to slow down the volume of
>>> requests. Everywhere I've put it so far increases the timeouts, rather than
>>> decreases them. I know there are some node.js experts out there on the
>>> DHIS2 dev list so, perhaps someone has run into something similar:
>>>
>>> var serverConfigPOST = serverConfig;
>>> serverConfigPOST.dhis2server.method = "POST";
>>> var i=0;
>>> for(i=0; i < dataElementsToUpdate.length; i++){
>>>
>>>                             Tried here
>>> var postData = JSON.stringify(dataElementsToUpdate[i]);
>>> serverConfigPOST.dhis2server.path =
>>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].
>>> object.id;
>>> serverConfigPOST.dhis2server.headers = {
>>> 'Content-Type': 'application/json',
>>> 'Content-Length': postData.length
>>> };
>>> console.log(serverConfigPOST);
>>> var req = https.request(serverConfigPOST.dhis2server, function(res) {
>>> console.log('STATUS: ' + res.statusCode);
>>> console.log('HEADERS: ' + JSON.stringify(res.headers));
>>> res.setEncoding('utf8');
>>> res.on('data', function (chunk) {
>>> console.log('BODY: ' + chunk);
>>> });
>>> });
>>> req.on('error', function(e) {
>>> //console.log('problem with request: ' + e.message);
>>> });
>>> // write data to request body
>>> req.write(postData);
>>> req.end();
>>>
>>>                             Tried here
>>> // if(i==20){
>>> // break;
>>> // }
>>> }
>>>
>>> Not quite sure how to slow this one down, I may just have to run/rerun
>>> it with different starting and stopping i values in the mean time.
>>>
>>> Timothy Harding
>>> RPCV Vanuatu
>>> Skype: hardingt@xxxxxxxxx
>>> +1 (541) 632-6623
>>>
>>
>>
>> --
>> Pierre Dane
>>
>> Development Manager
>> Jembi Health Systems
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>