dhis2-devs team mailing list archive

[Morten Olav Hansen <morten@xxxxxxxxx>]

Hi Vanya

This should have been caught by the importer... but there are several
duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
option groups, same with oqeVQ71LCgY..

I will look into making the validation process more robust.. but at least
you should know that this file have issues

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@xxxxxxxxx> wrote:

> Ok, thanks Vanya, I'm looking into it now
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi Morten
>>
>> Thanks for the response.
>>
>> We ran into another issue while trying to do a full export and import.
>>
>> As per the previous conversations we took care of all data migrations (as
>> per the new constraints on the DB- pretty much caught by the Importer
>> during the validation phase itself).
>>
>> Having done that we run into this issue:
>> The 'categoryoptiongroup' import is failing with constraint violation for
>> the UID. We checked the data and there is no repeating UID in the  database.
>>
>> The error text is attached for your reference, as well the payload used
>> for the import.
>>
>> Thanks for all the help in advance.
>>
>> Regards
>> Vanya
>>
>>
>>
>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some
>>> objects don't get their deps properly attached..
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx
>>> > wrote:
>>>
>>>> Hi Morten,
>>>>
>>>> Please find the payload for below request
>>>> *curl -H "Content-Type: application/json" -X GET -u username:password
>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
>>>> metadata.224.newapi.created.json*
>>>>
>>>> Import the payload in a fresh instance for import using
>>>>
>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>> @metadata.224.newapi.created.json -u admin:district
>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>> output_created_besteffort.txt*
>>>>
>>>> If the same payload is run with atomicMode=ALL, I am getting 'Invalid
>>>> references' errors which is acceptable. But if it is run with
>>>> atomicMode=NONE, it is throwing the error for which the complete stack
>>>> trace is attached earlier. filename: '
>>>> *output_created_besteffort_trace.txt'*
>>>>
>>>> Let me know if the complete payload or any other details are required
>>>> from my end.
>>>>
>>>>
>>>> Thanks
>>>> Aamer.
>>>>
>>>>
>>>>
>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Aamer
>>>>>
>>>>> Could you please share the payload of the object where this happen?
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Morten,
>>>>>>
>>>>>> The Importer was run with atomicMode=ALL. Though the payload is huge,
>>>>>> only few errors were shown in ImportSummary. As per errors in previous
>>>>>> ImportSummary, I have done below.
>>>>>> 1) ensured the 'short name' field is unique for each of the 'data
>>>>>> elements' and 'data element group' as well
>>>>>> 2) UID of admin, Super User, tracked entity was changed in payload to
>>>>>> be same to where it is getting imported.
>>>>>> I have manually resolved them and ran the importer again with
>>>>>> atomicMode=ALL.
>>>>>>
>>>>>> I got exception as attached in trace file
>>>>>> 'output_besteffort_trace.txt'
>>>>>> Would these kind of issues be known only after the import has failed?
>>>>>> and we need to resolve them one-by-one and as-and-when the importer throws
>>>>>> them? Is there any note which has the details about the constraints being
>>>>>> introduced in a new DHIS version?
>>>>>>
>>>>>> Alternatively, If we export a filtered set of metadata using '
>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the importer
>>>>>> with atomicMode=NONE
>>>>>>
>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>>>> output_created_besteffort_trace.txt*
>>>>>>
>>>>>> Getting below exception in trace. Complete trace also attached.
>>>>>> -----
>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type
>>>>>> ReportTable (DefaultObjectBundleService.java [qtp289378424-12])
>>>>>> org.hibernate.TransientObjectException: object references an unsaved
>>>>>> transient instance - save the transient instance before flushing:
>>>>>> org.hisp.dhis.indicator.Indicator
>>>>>> ------
>>>>>> Any suggestions please.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Aamer.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>
>>>>>>> Hi Aamer
>>>>>>>
>>>>>>> Probably what is happening is that the new database has some
>>>>>>> constraints that the old one doesn't. We have had some issues with
>>>>>>> hibernate in the past, and for certain databases that means that many of
>>>>>>> the constraints was not applied..
>>>>>>>
>>>>>>> If you look at the messages, you will see that e.g `shortName` must
>>>>>>> be unique for data elements.. the list of errors is not big though,
>>>>>>> probably you can go through the payload and manually update the objects
>>>>>>> which has issues?
>>>>>>>
>>>>>>> Another issue I see is that you are importing users, and those UIDs
>>>>>>> was only stabilized in 2.23.. so you might want to also manually update
>>>>>>> those (again, should not be a big job), this is not done as a startup
>>>>>>> routine.. just done when creating the admin user etc in a fresh database
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed <
>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Any thoughts please on the below approach. Let us know if any
>>>>>>>> details are required from our end around this.
>>>>>>>>
>>>>>>>> On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed <
>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi Morten,
>>>>>>>>>
>>>>>>>>> We have our metadata configured on a DHIS 2.21 instance. We have
>>>>>>>>> sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the
>>>>>>>>> upgrade notes for each of them.
>>>>>>>>>
>>>>>>>>> We want to export the metadata from this upgraded 2.24 instance
>>>>>>>>> and import it to a fresh instance which is already on 2.24. We followed the
>>>>>>>>> below steps.
>>>>>>>>> 1) Export the metadata from the upgraded 2.24 instance using the
>>>>>>>>> new exporter
>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>> username:password http://xxxx:8080/api/24/metadata.json
>>>>>>>>> <http://xxxx:8080/api/24/metadata.json> > metadata.224.newapi.json *
>>>>>>>>>
>>>>>>>>> 2) Import the metadata using the new importer to a fresh instance
>>>>>>>>> which is already on 2.24
>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>>>> @metadata.224.newapi.json -u username:password
>>>>>>>>> http://yyyy:8080/api/24/metadata <http://yyyy:8080/api/24/metadata> >
>>>>>>>>> output.txt*
>>>>>>>>>
>>>>>>>>> We see that none of the objects are getting imported. The complete
>>>>>>>>> response of the importer is attached below. Any thoughts please. Also let
>>>>>>>>> me know if any further details are required from our end.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Aamer.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> With Regards
>> ThoughtWorks Technologies
>> Hyderabad
>>
>> --Stay Hungry Stay Foolish!!
>>
>
>