dhis2-devs team mailing list archive

[Sultanahamar Mohammad <sultanm@xxxxxxxxxxxxxxxx>]

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think
there is nothing suspicious out here.


​
Regards,
Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx> wrote:

> I'm not sure what is causing it, but it shouldn't be allowed to begin
> with. Is there any constraint on the UID column in your database?
>
> [image: Inline image 1]
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
> sultanm@xxxxxxxxxxxxxxxx> wrote:
>
>> Hi Morten,
>>
>> Hope you are doing good. Let us know your thoughts on the above issue and
>> please let us know if you need any information that you might require.
>>
>> Regards,
>> Sultan Ahamar.
>>
>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi Morten,
>>>
>>> We have pulled latest code from 2.24 and tried import /  export again.
>>> We have some interesting observations
>>>
>>> *DB snapshot:*
>>>
>>>
>>> ​
>>> There are no duplicate items in DB. We double checked.
>>>
>>>
>>>
>>>
>>> *Entity endpoint snapshot:*
>>>
>>> ​Interestingly we found few duplicate entities when we try to fetch this
>>> data through the API. In Maintenance app, we are not able to see these
>>> duplicate entities. Does it also explain why we get repeating entries (with
>>> same UID's) in the full export JSON as well?  Let us know on how to proceed
>>> and if you need more input.
>>>
>>> Thanks for all the help in advance.
>>>
>>> Regards,
>>> Sultan Ahamar.
>>>
>>>
>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Vanya
>>>>
>>>> This should now have been fixed in master and 224. It will now do a
>>>> global UID check first, and report back any duplicates, if it finds a
>>>> duplicate it will remove them from the import and report back an
>>>> `ErrorReport` (as it would do with other validation issues). So if it finds
>>>> a duplicate, you will need to set atomic mode to NONE to have it import and
>>>> ignore the duplicates.
>>>>
>>>> (btw, there was several duplicates in your file, not sure how that
>>>> happened.. maybe something needs to be cleaned in your source instance)
>>>>
>>>> --
>>>> Morten Olav Hansen
>>>> Senior Engineer, DHIS 2
>>>> University of Oslo
>>>> http://www.dhis2.org
>>>>
>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Vanya
>>>>>
>>>>> This should have been caught by the importer... but there are several
>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
>>>>> option groups, same with oqeVQ71LCgY..
>>>>>
>>>>> I will look into making the validation process more robust.. but at
>>>>> least you should know that this file have issues
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Ok, thanks Vanya, I'm looking into it now
>>>>>>
>>>>>> --
>>>>>> Morten Olav Hansen
>>>>>> Senior Engineer, DHIS 2
>>>>>> University of Oslo
>>>>>> http://www.dhis2.org
>>>>>>
>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Morten
>>>>>>>
>>>>>>> Thanks for the response.
>>>>>>>
>>>>>>> We ran into another issue while trying to do a full export and
>>>>>>> import.
>>>>>>>
>>>>>>> As per the previous conversations we took care of all data
>>>>>>> migrations (as per the new constraints on the DB- pretty much caught by the
>>>>>>> Importer during the validation phase itself).
>>>>>>>
>>>>>>> Having done that we run into this issue:
>>>>>>> The 'categoryoptiongroup' import is failing with constraint
>>>>>>> violation for the UID. We checked the data and there is no repeating UID in
>>>>>>> the  database.
>>>>>>>
>>>>>>> The error text is attached for your reference, as well the payload
>>>>>>> used for the import.
>>>>>>>
>>>>>>> Thanks for all the help in advance.
>>>>>>>
>>>>>>> Regards
>>>>>>> Vanya
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some
>>>>>>>> objects don't get their deps properly attached..
>>>>>>>>
>>>>>>>> --
>>>>>>>> Morten Olav Hansen
>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>> University of Oslo
>>>>>>>> http://www.dhis2.org
>>>>>>>>
>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi Morten,
>>>>>>>>>
>>>>>>>>> Please find the payload for below request
>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>> username:password
>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
>>>>>>>>> metadata.224.newapi.created.json*
>>>>>>>>>
>>>>>>>>> Import the payload in a fresh instance for import using
>>>>>>>>>
>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>>> output_created_besteffort.txt*
>>>>>>>>>
>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is run with
>>>>>>>>> atomicMode=NONE, it is throwing the error for which the complete stack
>>>>>>>>> trace is attached earlier. filename: '
>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>>>>>>
>>>>>>>>> Let me know if the complete payload or any other details are
>>>>>>>>> required from my end.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Aamer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Aamer
>>>>>>>>>>
>>>>>>>>>> Could you please share the payload of the object where this
>>>>>>>>>> happen?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> University of Oslo
>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Morten,
>>>>>>>>>>>
>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload is
>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per errors in
>>>>>>>>>>> previous ImportSummary, I have done below.
>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
>>>>>>>>>>> 'data elements' and 'data element group' as well
>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>>>>>>>>>>> payload to be same to where it is getting imported.
>>>>>>>>>>> I have manually resolved them and ran the importer again with
>>>>>>>>>>> atomicMode=ALL.
>>>>>>>>>>>
>>>>>>>>>>> I got exception as attached in trace file
>>>>>>>>>>> 'output_besteffort_trace.txt'
>>>>>>>>>>> Would these kind of issues be known only after the import has
>>>>>>>>>>> failed? and we need to resolve them one-by-one and as-and-when the importer
>>>>>>>>>>> throws them? Is there any note which has the details about the constraints
>>>>>>>>>>> being introduced in a new DHIS version?
>>>>>>>>>>>
>>>>>>>>>>> Alternatively, If we export a filtered set of metadata using '
>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
>>>>>>>>>>> importer with atomicMode=NONE
>>>>>>>>>>>
>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>>>>> output_created_besteffort_trace.txt*
>>>>>>>>>>>
>>>>>>>>>>> Getting below exception in trace. Complete trace also attached.
>>>>>>>>>>> -----
>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of
>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>>>>>>>>>>> object references an unsaved transient instance - save the transient
>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
>>>>>>>>>>> ------
>>>>>>>>>>> Any suggestions please.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Aamer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Aamer
>>>>>>>>>>>>
>>>>>>>>>>>> Probably what is happening is that the new database has some
>>>>>>>>>>>> constraints that the old one doesn't. We have had some issues with
>>>>>>>>>>>> hibernate in the past, and for certain databases that means that many of
>>>>>>>>>>>> the constraints was not applied..
>>>>>>>>>>>>
>>>>>>>>>>>> If you look at the messages, you will see that e.g `shortName`
>>>>>>>>>>>> must be unique for data elements.. the list of errors is not big though,
>>>>>>>>>>>> probably you can go through the payload and manually update the objects
>>>>>>>>>>>> which has issues?
>>>>>>>>>>>>
>>>>>>>>>>>> Another issue I see is that you are importing users, and those
>>>>>>>>>>>> UIDs was only stabilized in 2.23.. so you might want to also manually
>>>>>>>>>>>> update those (again, should not be a big job), this is not done as a
>>>>>>>>>>>> startup routine.. just done when creating the admin user etc in a fresh
>>>>>>>>>>>> database
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>>> University of Oslo
[truncated for moderation]