dhis2-devs team mailing list archive

[Morten Olav Hansen <morten@xxxxxxxxx>]

Hi again

Would it be possible to share your database with me? I haven't seen this
issues anywhere else.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
sultanm@xxxxxxxxxxxxxxxx> wrote:

> Hi Morten,
>
> This is what our table is showing up when we say \d dataelement. I think
> there is nothing suspicious out here.
>
>
> ​
> Regards,
> Sultan Ahamar.
>
> On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx>
> wrote:
>
>> I'm not sure what is causing it, but it shouldn't be allowed to begin
>> with. Is there any constraint on the UID column in your database?
>>
>> [image: Inline image 1]
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi Morten,
>>>
>>> Hope you are doing good. Let us know your thoughts on the above issue
>>> and please let us know if you need any information that you might require.
>>>
>>> Regards,
>>> Sultan Ahamar.
>>>
>>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Hi Morten,
>>>>
>>>> We have pulled latest code from 2.24 and tried import /  export again.
>>>> We have some interesting observations
>>>>
>>>> *DB snapshot:*
>>>>
>>>>
>>>> ​
>>>> There are no duplicate items in DB. We double checked.
>>>>
>>>>
>>>>
>>>>
>>>> *Entity endpoint snapshot:*
>>>>
>>>> ​Interestingly we found few duplicate entities when we try to fetch
>>>> this data through the API. In Maintenance app, we are not able to see these
>>>> duplicate entities. Does it also explain why we get repeating entries (with
>>>> same UID's) in the full export JSON as well?  Let us know on how to proceed
>>>> and if you need more input.
>>>>
>>>> Thanks for all the help in advance.
>>>>
>>>> Regards,
>>>> Sultan Ahamar.
>>>>
>>>>
>>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Vanya
>>>>>
>>>>> This should now have been fixed in master and 224. It will now do a
>>>>> global UID check first, and report back any duplicates, if it finds a
>>>>> duplicate it will remove them from the import and report back an
>>>>> `ErrorReport` (as it would do with other validation issues). So if it finds
>>>>> a duplicate, you will need to set atomic mode to NONE to have it import and
>>>>> ignore the duplicates.
>>>>>
>>>>> (btw, there was several duplicates in your file, not sure how that
>>>>> happened.. maybe something needs to be cleaned in your source instance)
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Vanya
>>>>>>
>>>>>> This should have been caught by the importer... but there are several
>>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
>>>>>> option groups, same with oqeVQ71LCgY..
>>>>>>
>>>>>> I will look into making the validation process more robust.. but at
>>>>>> least you should know that this file have issues
>>>>>>
>>>>>> --
>>>>>> Morten Olav Hansen
>>>>>> Senior Engineer, DHIS 2
>>>>>> University of Oslo
>>>>>> http://www.dhis2.org
>>>>>>
>>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok, thanks Vanya, I'm looking into it now
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Morten
>>>>>>>>
>>>>>>>> Thanks for the response.
>>>>>>>>
>>>>>>>> We ran into another issue while trying to do a full export and
>>>>>>>> import.
>>>>>>>>
>>>>>>>> As per the previous conversations we took care of all data
>>>>>>>> migrations (as per the new constraints on the DB- pretty much caught by the
>>>>>>>> Importer during the validation phase itself).
>>>>>>>>
>>>>>>>> Having done that we run into this issue:
>>>>>>>> The 'categoryoptiongroup' import is failing with constraint
>>>>>>>> violation for the UID. We checked the data and there is no repeating UID in
>>>>>>>> the  database.
>>>>>>>>
>>>>>>>> The error text is attached for your reference, as well the payload
>>>>>>>> used for the import.
>>>>>>>>
>>>>>>>> Thanks for all the help in advance.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Vanya
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some
>>>>>>>>> objects don't get their deps properly attached..
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Morten Olav Hansen
>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>> University of Oslo
>>>>>>>>> http://www.dhis2.org
>>>>>>>>>
>>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Morten,
>>>>>>>>>>
>>>>>>>>>> Please find the payload for below request
>>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>>> username:password
>>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
>>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
>>>>>>>>>> metadata.224.newapi.created.json*
>>>>>>>>>>
>>>>>>>>>> Import the payload in a fresh instance for import using
>>>>>>>>>>
>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>>>> output_created_besteffort.txt*
>>>>>>>>>>
>>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is run with
>>>>>>>>>> atomicMode=NONE, it is throwing the error for which the complete stack
>>>>>>>>>> trace is attached earlier. filename: '
>>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>>>>>>>
>>>>>>>>>> Let me know if the complete payload or any other details are
>>>>>>>>>> required from my end.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Aamer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Aamer
>>>>>>>>>>>
>>>>>>>>>>> Could you please share the payload of the object where this
>>>>>>>>>>> happen?
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Morten,
>>>>>>>>>>>>
>>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload is
>>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per errors in
>>>>>>>>>>>> previous ImportSummary, I have done below.
>>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
>>>>>>>>>>>> 'data elements' and 'data element group' as well
>>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>>>>>>>>>>>> payload to be same to where it is getting imported.
>>>>>>>>>>>> I have manually resolved them and ran the importer again with
>>>>>>>>>>>> atomicMode=ALL.
>>>>>>>>>>>>
>>>>>>>>>>>> I got exception as attached in trace file
>>>>>>>>>>>> 'output_besteffort_trace.txt'
>>>>>>>>>>>> Would these kind of issues be known only after the import has
>>>>>>>>>>>> failed? and we need to resolve them one-by-one and as-and-when the importer
>>>>>>>>>>>> throws them? Is there any note which has the details about the constraints
>>>>>>>>>>>> being introduced in a new DHIS version?
>>>>>>>>>>>>
>>>>>>>>>>>> Alternatively, If we export a filtered set of metadata using '
>>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
>>>>>>>>>>>> importer with atomicMode=NONE
>>>>>>>>>>>>
>>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>>>>>> output_created_besteffort_trace.txt*
>>>>>>>>>>>>
>>>>>>>>>>>> Getting below exception in trace. Complete trace also attached.
>>>>>>>>>>>> -----
>>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of
>>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>>>>>>>>>>>> object references an unsaved transient instance - save the transient
>>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
>>>>>>>>>>>> ------
>>>>>>>>>>>> Any suggestions please.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Aamer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Aamer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Probably what is happening is that the new database has some
>>>>>>>>>>>>> constraints that the old one doesn't. We have had some issues with
>>>>>>>>>>>>> hibernate in the past, and for certain databases that means that many of
>>>>>>>>>>>>> the constraints was not applied..
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you look at the messages, you will see that e.g `shortName`
>>>>>>>>>>>>> must be unique for data elements.. the list of errors is not big though,
>>>>>>>>>>>>> probably you can go through
[truncated for moderation]