dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

Jason,

We have another strange Data Integrity check result - this one is not
related to any known change in the database. This one relates to the "Data
Elements in data set but not in form or sections" - we are seeing that
check returning one data element that definitely exist in both the single
section and in the relevant form.

Is there a way of getting a list of those Hibernate-generated sQL queries
out of the system, so that we can run any of them manually to see exactly
what is going on?

Regards
Calle

On 2 September 2015 at 11:40, Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:

> Jason,
>
> 1. Yes
> 2. Yes, I changed both name and shortname (identical) -  but I do not know
> if the Data Integrity checks display name or short name (settings = short
> name for that instance).
> 3. We are not using any caching proxy
>
> We did restart Tomcat - which should clear the server cache (?) - and I
> did clear all my local caches.
>
> BUT we had some other strange behaviour with regard to dashboard
> privileges yesterday, and it was seemingly fixed by clearing the DHIS2
> cache MULTIPLE times (not sure how many, but I only cleared the cache one
> time...)
>
> Regards
> Calle
>
> On 2 September 2015 at 10:10, Jason Pickering <jason.p.pickering@xxxxxxxxx
> > wrote:
>
>> So, coming back to Lars's point.
>>
>> 1) Did you check the "dataset" table to confirm that your intended
>> changes were persisted?
>>
>> 2) Did you alter the dataset "shortname" and is that the same which you
>> are seeing in your integrity checks?
>>
>> 3) You indicated you cleared the DHIS 2 server cache. Can you confirm
>> that you have no other upstream caching proxy which might be caching some
>> response?
>>
>> The data integrity checks are a series of SQL (actually generated by
>> Hibernate) queries, so as is usually the case, if you make direct database
>> manipulations (and do not clear the server cache and possibly your local
>> browser cache) you may see inconsistent results. For this type of
>> operation, use of the UI is probably better, or alternative with the
>> WebAPI.
>>
>> Regards,
>> Jason
>>
>>
>>
>> On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>> wrote:
>>
>>> Halvdan,
>>>
>>> Ah, OK, and not - I was not using any queries, I simply updated the Data
>>> Set names directly in the table.
>>>
>>> As I've said - the weird thing is that all those new names immediately
>>> showed up in the UI data set list, but the old names are still appearing
>>> wherever the Data Integrity process throws up integrity violations
>>>
>>> Regards
>>> Calle
>>>
>>> On 1 September 2015 at 18:01, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>>> wrote:
>>>
>>>> Calle, I'm pretty sure Jason was referring to whatever you were doing
>>>> to update the records (your point #1 in original email) which was not, as
>>>> far as I can understand, using the UI or the web-api. I am not familiar
>>>> with "pgremote" but am assuming it's some kind of postgres connector
>>>> allowing you to run (sql) queries. We'd need to see those queries in order
>>>> to figure out what could have gone wrong.
>>>>
>>>> 2015-09-01 14:50 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>
>>>>> Jason
>>>>>
>>>>> I was not executing any custom sql - I'm running the Data
>>>>> Administration -> Data Integrity process checks.
>>>>>
>>>>> My reference to the sql queries being run relates to my ASSUMPTION
>>>>> that the Data Integrity checks are a series of SQL queries - but maybe not,
>>>>> maybe the integrity checks are a series of web api calls followed by some
>>>>> kind of front-end processing? I just don't know, and I'm not a Java
>>>>> developer and I don't have the code base on my laptop so I cannot find out
>>>>> myself...
>>>>>
>>>>> Seems my initial post was not clear enough on this - apologies for
>>>>> that.
>>>>>
>>>>> Regards
>>>>> Calle
>>>>>
>>>>> On 1 September 2015 at 14:31, Jason Pickering <
>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Calle,
>>>>>> It is very difficult to say much without knowing what SQL it was you
>>>>>> executed. In general, use of the API or UI is usually much safer.
>>>>>>
>>>>>> Do you have any sort of cache in front of your DHIS2 instance?
>>>>>>
>>>>>> Can you confirm from the database that the names were actually
>>>>>> changed?
>>>>>>
>>>>>> Regards,
>>>>>> Jason
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg <
>>>>>> calle.hedberg@xxxxxxxxx> wrote:
>>>>>>
>>>>>>> Lars,
>>>>>>>
>>>>>>> Nope - all data set names and short names are the same.
>>>>>>>
>>>>>>> It's just weird, and I don't like inexplicable things happening to a
>>>>>>> database....
>>>>>>>
>>>>>>> When you run data integrity - WHERE are those queries fetching data
>>>>>>> from? Are the integrity queries running against the database itself, or
>>>>>>> some kind of cache repository?
>>>>>>>
>>>>>>> Regards
>>>>>>> Calle
>>>>>>>
>>>>>>> On 1 September 2015 at 13:00, Lars Helge Øverland <
>>>>>>> larshelge@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Hi Calle,
>>>>>>>>
>>>>>>>> perhaps you did not update the data set short names?
>>>>>>>>
>>>>>>>> Lars
>>>>>>>>
>>>>>>>> On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg <
>>>>>>>> calle.hedberg@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I've got a baffling problem (presumably related to some kind of
>>>>>>>>> caching):
>>>>>>>>>
>>>>>>>>> 1. yesterday I updated various data records using pgremote,
>>>>>>>>> including data set names. We restarted tomcat after the updates.
>>>>>>>>>
>>>>>>>>> 2. I've also cleared my browser cache, the DHIS2 cache, updated
>>>>>>>>> resource tables, etc.
>>>>>>>>>
>>>>>>>>> 3. When I now open the list of data sets, all the changed/new
>>>>>>>>> names are showing as expected.
>>>>>>>>>
>>>>>>>>> 4. BUT when I run Data Integrity, a few violations related to data
>>>>>>>>> sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.
>>>>>>>>>
>>>>>>>>> Any ideas what could cause this weird behaviour?
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Calle
>>>>>>>>>
>>>>>>>>> *******************************************
>>>>>>>>>
>>>>>>>>> Calle Hedberg
>>>>>>>>>
>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>>>
>>>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>>>
>>>>>>>>> Cell: +27-82-853-5352
>>>>>>>>>
>>>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>>>
>>>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>>>
>>>>>>>>> Skype: calle_hedberg
>>>>>>>>>
>>>>>>>>> *******************************************
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Lars Helge Øverland
>>>>>>>> Lead developer, DHIS 2
>>>>>>>> University of Oslo
>>>>>>>> Skype: larshelgeoverland
>>>>>>>> http://www.dhis2.org <https://www.dhis2.org>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>> Calle Hedberg
>>>>>>>
>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>
>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>
>>>>>>> Cell: +27-82-853-5352
>>>>>>>
>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>
>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>
>>>>>>> Skype: calle_hedberg
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason P. Pickering
>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>> tel:+46764147049
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *******************************************
>>>>>
>>>>> Calle Hedberg
>>>>>
>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>
>>>>> Tel/fax (home): +27-21-685-6472
>>>>>
>>>>> Cell: +27-82-853-5352
>>>>>
>>>>> Iridium SatPhone: +8816-315-19274
>>>>>
>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>
>>>>> Skype: calle_hedberg
>>>>>
>>>>> *******************************************
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *******************************************
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472
>>>
>>> Cell: +27-82-853-5352
>>>
>>> Iridium SatPhone: +8816-315-19274
>>>
>>> Email: calle.hedberg@xxxxxxxxx
>>>
>>> Skype: calle_hedberg
>>>
>>> *******************************************
>>>
>>>
>>
>>
>> --
>> Jason P. Pickering
>> email: jason.p.pickering@xxxxxxxxx
>> tel:+46764147049
>>
>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
>


-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************