dhis2-devs team mailing list archive

[Morten Olav Hansen <morten@xxxxxxxxx>]

I'm not sure what is causing it, but it shouldn't be allowed to begin with.
Is there any constraint on the UID column in your database?

[image: Inline image 1]

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
sultanm@xxxxxxxxxxxxxxxx> wrote:

> Hi Morten,
>
> Hope you are doing good. Let us know your thoughts on the above issue and
> please let us know if you need any information that you might require.
>
> Regards,
> Sultan Ahamar.
>
> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
> sultanm@xxxxxxxxxxxxxxxx> wrote:
>
>> Hi Morten,
>>
>> We have pulled latest code from 2.24 and tried import /  export again. We
>> have some interesting observations
>>
>> *DB snapshot:*
>>
>>
>> ​
>> There are no duplicate items in DB. We double checked.
>>
>>
>>
>>
>> *Entity endpoint snapshot:*
>>
>> ​Interestingly we found few duplicate entities when we try to fetch this
>> data through the API. In Maintenance app, we are not able to see these
>> duplicate entities. Does it also explain why we get repeating entries (with
>> same UID's) in the full export JSON as well?  Let us know on how to proceed
>> and if you need more input.
>>
>> Thanks for all the help in advance.
>>
>> Regards,
>> Sultan Ahamar.
>>
>>
>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> Hi Vanya
>>>
>>> This should now have been fixed in master and 224. It will now do a
>>> global UID check first, and report back any duplicates, if it finds a
>>> duplicate it will remove them from the import and report back an
>>> `ErrorReport` (as it would do with other validation issues). So if it finds
>>> a duplicate, you will need to set atomic mode to NONE to have it import and
>>> ignore the duplicates.
>>>
>>> (btw, there was several duplicates in your file, not sure how that
>>> happened.. maybe something needs to be cleaned in your source instance)
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Vanya
>>>>
>>>> This should have been caught by the importer... but there are several
>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
>>>> option groups, same with oqeVQ71LCgY..
>>>>
>>>> I will look into making the validation process more robust.. but at
>>>> least you should know that this file have issues
>>>>
>>>> --
>>>> Morten Olav Hansen
>>>> Senior Engineer, DHIS 2
>>>> University of Oslo
>>>> http://www.dhis2.org
>>>>
>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Ok, thanks Vanya, I'm looking into it now
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Morten
>>>>>>
>>>>>> Thanks for the response.
>>>>>>
>>>>>> We ran into another issue while trying to do a full export and
>>>>>> import.
>>>>>>
>>>>>> As per the previous conversations we took care of all data migrations
>>>>>> (as per the new constraints on the DB- pretty much caught by the Importer
>>>>>> during the validation phase itself).
>>>>>>
>>>>>> Having done that we run into this issue:
>>>>>> The 'categoryoptiongroup' import is failing with constraint violation
>>>>>> for the UID. We checked the data and there is no repeating UID in the
>>>>>>  database.
>>>>>>
>>>>>> The error text is attached for your reference, as well the payload
>>>>>> used for the import.
>>>>>>
>>>>>> Thanks for all the help in advance.
>>>>>>
>>>>>> Regards
>>>>>> Vanya
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <morten@xxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some
>>>>>>> objects don't get their deps properly attached..
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Hi Morten,
>>>>>>>>
>>>>>>>> Please find the payload for below request
>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>> username:password
>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
>>>>>>>> metadata.224.newapi.created.json*
>>>>>>>>
>>>>>>>> Import the payload in a fresh instance for import using
>>>>>>>>
>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>> output_created_besteffort.txt*
>>>>>>>>
>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>>>>>>>> 'Invalid references' errors which is acceptable. But if it is run with
>>>>>>>> atomicMode=NONE, it is throwing the error for which the complete stack
>>>>>>>> trace is attached earlier. filename: '
>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>>>>>
>>>>>>>> Let me know if the complete payload or any other details are
>>>>>>>> required from my end.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Aamer.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi Aamer
>>>>>>>>>
>>>>>>>>> Could you please share the payload of the object where this happen?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Morten Olav Hansen
>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>> University of Oslo
>>>>>>>>> http://www.dhis2.org
>>>>>>>>>
>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Morten,
>>>>>>>>>>
>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload is
>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per errors in
>>>>>>>>>> previous ImportSummary, I have done below.
>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the 'data
>>>>>>>>>> elements' and 'data element group' as well
>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>>>>>>>>>> payload to be same to where it is getting imported.
>>>>>>>>>> I have manually resolved them and ran the importer again with
>>>>>>>>>> atomicMode=ALL.
>>>>>>>>>>
>>>>>>>>>> I got exception as attached in trace file
>>>>>>>>>> 'output_besteffort_trace.txt'
>>>>>>>>>> Would these kind of issues be known only after the import has
>>>>>>>>>> failed? and we need to resolve them one-by-one and as-and-when the importer
>>>>>>>>>> throws them? Is there any note which has the details about the constraints
>>>>>>>>>> being introduced in a new DHIS version?
>>>>>>>>>>
>>>>>>>>>> Alternatively, If we export a filtered set of metadata using '
>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
>>>>>>>>>> importer with atomicMode=NONE
>>>>>>>>>>
>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>>>>>> output_created_besteffort_trace.txt*
>>>>>>>>>>
>>>>>>>>>> Getting below exception in trace. Complete trace also attached.
>>>>>>>>>> -----
>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of
>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>>>>>>>>>> object references an unsaved transient instance - save the transient
>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
>>>>>>>>>> ------
>>>>>>>>>> Any suggestions please.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Aamer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Aamer
>>>>>>>>>>>
>>>>>>>>>>> Probably what is happening is that the new database has some
>>>>>>>>>>> constraints that the old one doesn't. We have had some issues with
>>>>>>>>>>> hibernate in the past, and for certain databases that means that many of
>>>>>>>>>>> the constraints was not applied..
>>>>>>>>>>>
>>>>>>>>>>> If you look at the messages, you will see that e.g `shortName`
>>>>>>>>>>> must be unique for data elements.. the list of errors is not big though,
>>>>>>>>>>> probably you can go through the payload and manually update the objects
>>>>>>>>>>> which has issues?
>>>>>>>>>>>
>>>>>>>>>>> Another issue I see is that you are importing users, and those
>>>>>>>>>>> UIDs was only stabilized in 2.23.. so you might want to also manually
>>>>>>>>>>> update those (again, should not be a big job), this is not done as a
>>>>>>>>>>> startup routine.. just done when creating the admin user etc in a fresh
>>>>>>>>>>> database
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed <
>>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Any thoughts please on the below approach. Let us know if any
>>>>>>>>>>>> details are required from our end around this.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed <
>>>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Morten,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have our metadata configured on a DHIS 2.21 instance. We
>>>>>>>>>>>>> have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by
>>>>>>>>>>>>> following the upgrade notes for each of them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We want to export the metadata from this upgraded 2.24
>>>>>>>>>>>>> instance and import it to a fresh instance which is already on 2.24. We
>>>>>>>>>>>>> followed the below steps.
>>>>>>>>>>>>> 1) Export the metadata from the upgraded 2.24 instance using
>>>>>>>>>>>>> the new exporter
>>>>>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>>>>>> username:password http://xxxx:8080/api/24/metadata.json
>>>>>>>>>>>>> <http://xxxx:8080/api/24/metadata.json> > metadata.224.newapi.json *
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Import the metadata using the new importer to a fresh
>>>>>>>>>>>>> instance which is already on 2.24
>>>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>>>>>> @metadata.224.newapi.json -u username:password
>>>>>>>>>>>>> http://yyyy:8080/api/24/metadata <http://yyyy:8080/api/24/metadata> >
>>>>>>>>>>>>> output.txt*
>>>>>>>>>>>>>
>>>>>>>>>>>>> We see that none of the objects are getting imported. The
>>>>>>>>>>>>> complete response of the importer is attached below. Any thoughts please.
>>>>>>>>>>>>> Also let me know if any further details are required from our end.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Aamer.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> With Regards
>>>>>> ThoughtWorks Technologies
>>>>>> Hyderabad
>>>>>>
>>>>>> --Stay Hungry Stay Foolish!!
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>