dhis2-devs team mailing list archive

[Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>]

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as
per the new constraints on the DB- pretty much caught by the Importer
during the validation phase itself).

Having done that we run into this issue:
The 'categoryoptiongroup' import is failing with constraint violation for
the UID. We checked the data and there is no repeating UID in the  database.

The error text is attached for your reference, as well the payload used for
the import.

Thanks for all the help in advance.

Regards
Vanya



On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <morten@xxxxxxxxx>
wrote:

> Thanks Aamer, I'm looking at a similar bug right now.. seems some objects
> don't get their deps properly attached..
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi Morten,
>>
>> Please find the payload for below request
>> *curl -H "Content-Type: application/json" -X GET -u username:password
>> "http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
>> <http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
>> metadata.224.newapi.created.json*
>>
>> Import the payload in a fresh instance for import using
>>
>> *curl -H "Content-Type: application/json" -X POST --data
>> @metadata.224.newapi.created.json -u admin:district
>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>> output_created_besteffort.txt*
>>
>> If the same payload is run with atomicMode=ALL, I am getting 'Invalid
>> references' errors which is acceptable. But if it is run with
>> atomicMode=NONE, it is throwing the error for which the complete stack
>> trace is attached earlier. filename: '
>> *output_created_besteffort_trace.txt'*
>>
>> Let me know if the complete payload or any other details are required
>> from my end.
>>
>>
>> Thanks
>> Aamer.
>>
>>
>>
>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> Hi Aamer
>>>
>>> Could you please share the payload of the object where this happen?
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx
>>> > wrote:
>>>
>>>> Hi Morten,
>>>>
>>>> The Importer was run with atomicMode=ALL. Though the payload is huge,
>>>> only few errors were shown in ImportSummary. As per errors in previous
>>>> ImportSummary, I have done below.
>>>> 1) ensured the 'short name' field is unique for each of the 'data
>>>> elements' and 'data element group' as well
>>>> 2) UID of admin, Super User, tracked entity was changed in payload to
>>>> be same to where it is getting imported.
>>>> I have manually resolved them and ran the importer again with
>>>> atomicMode=ALL.
>>>>
>>>> I got exception as attached in trace file 'output_besteffort_trace.txt'
>>>> Would these kind of issues be known only after the import has failed?
>>>> and we need to resolve them one-by-one and as-and-when the importer throws
>>>> them? Is there any note which has the details about the constraints being
>>>> introduced in a new DHIS version?
>>>>
>>>> Alternatively, If we export a filtered set of metadata using '
>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the importer
>>>> with atomicMode=NONE
>>>>
>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>> @metadata.224.newapi.json -u admin:district
>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>> output_created_besteffort_trace.txt*
>>>>
>>>> Getting below exception in trace. Complete trace also attached.
>>>> -----
>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type
>>>> ReportTable (DefaultObjectBundleService.java [qtp289378424-12])
>>>> org.hibernate.TransientObjectException: object references an unsaved
>>>> transient instance - save the transient instance before flushing:
>>>> org.hisp.dhis.indicator.Indicator
>>>> ------
>>>> Any suggestions please.
>>>>
>>>>
>>>> Thanks
>>>> Aamer.
>>>>
>>>>
>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Aamer
>>>>>
>>>>> Probably what is happening is that the new database has some
>>>>> constraints that the old one doesn't. We have had some issues with
>>>>> hibernate in the past, and for certain databases that means that many of
>>>>> the constraints was not applied..
>>>>>
>>>>> If you look at the messages, you will see that e.g `shortName` must be
>>>>> unique for data elements.. the list of errors is not big though, probably
>>>>> you can go through the payload and manually update the objects which has
>>>>> issues?
>>>>>
>>>>> Another issue I see is that you are importing users, and those UIDs
>>>>> was only stabilized in 2.23.. so you might want to also manually update
>>>>> those (again, should not be a big job), this is not done as a startup
>>>>> routine.. just done when creating the admin user etc in a fresh database
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed <
>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Any thoughts please on the below approach. Let us know if any details
>>>>>> are required from our end around this.
>>>>>>
>>>>>> On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed <
>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>
>>>>>>> Hi Morten,
>>>>>>>
>>>>>>> We have our metadata configured on a DHIS 2.21 instance. We have
>>>>>>> sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the
>>>>>>> upgrade notes for each of them.
>>>>>>>
>>>>>>> We want to export the metadata from this upgraded 2.24 instance and
>>>>>>> import it to a fresh instance which is already on 2.24. We followed the
>>>>>>> below steps.
>>>>>>> 1) Export the metadata from the upgraded 2.24 instance using the new
>>>>>>> exporter
>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>> username:password http://xxxx:8080/api/24/metadata.json
>>>>>>> <http://xxxx:8080/api/24/metadata.json> > metadata.224.newapi.json *
>>>>>>>
>>>>>>> 2) Import the metadata using the new importer to a fresh instance
>>>>>>> which is already on 2.24
>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>>>>> @metadata.224.newapi.json -u username:password
>>>>>>> http://yyyy:8080/api/24/metadata <http://yyyy:8080/api/24/metadata> >
>>>>>>> output.txt*
>>>>>>>
>>>>>>> We see that none of the objects are getting imported. The complete
>>>>>>> response of the importer is attached below. Any thoughts please. Also let
>>>>>>> me know if any further details are required from our end.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Aamer.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


-- 
With Regards
ThoughtWorks Technologies
Hyderabad

--Stay Hungry Stay Foolish!!

* WARN  2016-09-01 15:12:15,380 Authentication event AuthenticationSuccessEvent: admin; details: org.springframework.security.web.authentication.WebAuthenticationDetails@b364: RemoteIpAddress: 0:0:0:0:0:0:0:1; SessionId: null (LoggerListener.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:15,561 Skipping unknown property 'date'. (DefaultRenderService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:16,831 (admin) Import:Start (DefaultMetadataImportService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:17,039 (admin) Import:Preheat[REFERENCE] took 0.20 seconds (DefaultPreheatService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:17,954 (admin) Import:Validation took 0.39 seconds (DefaultObjectBundleValidationService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:17,976 (admin) Creating 9 object(s) of type UserAuthorityGroup (DefaultObjectBundleService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:17,983 (admin) Updating 1 object(s) of type UserAuthorityGroup (DefaultObjectBundleService.java [qtp2119344747-17])
* INFO  2016-09-01 15:12:18,038 (admin) Creating 735 object(s) of type User (DefaultObjectBundleService.java [qtp2119344747-17])
PUT object : {"username":"twoca", "openId":"null", "password":"null", "passwordLastUpdated":"Mon Jul 25 16:38:53 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"yrB6vc5Ip3r", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Sep 01 11:31:57 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-mfp-assist", "openId":"null", "password":"null", "passwordLastUpdated":"Fri Jul 15 14:14:02 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"gmOkgYI46ny", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Fri Jul 15 14:14:02 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"maria.cristobal", "openId":"null", "password":"null", "passwordLastUpdated":"Fri Jul 15 14:04:00 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"eANTPuEwDDF", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Fri Jul 15 14:04:00 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-ecuador-emerg-online3", "openId":"null", "password":"null", "passwordLastUpdated":"Tue Jul 12 07:35:04 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"mnIBssgDVSC", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Tue Jul 12 07:38:48 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-ecuador-emerg-online2", "openId":"null", "password":"null", "passwordLastUpdated":"Tue Jul 12 07:34:05 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"mnIBssgDVSC", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Wed Jul 13 14:03:06 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-ecuador-emerg-online1", "openId":"null", "password":"null", "passwordLastUpdated":"Tue Jul 12 07:29:37 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"mnIBssgDVSC", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Tue Jul 12 22:26:55 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"bruno.raimbault", "openId":"null", "password":"null", "passwordLastUpdated":"Mon Jul 11 08:25:06 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"yrB6vc5Ip3r", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Fri Jul 22 06:42:03 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"merce.rocaspana", "openId":"null", "password":"null", "passwordLastUpdated":"Mon Jul 04 08:08:01 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"imjx3wfUjCJ", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Wed Jul 06 13:59:40 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-user15", "openId":"null", "password":"null", "passwordLastUpdated":"Thu Jun 30 11:03:25 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"N4dxeOVu7aN", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Jun 30 11:03:25 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-user14", "openId":"null", "password":"null", "passwordLastUpdated":"Thu Jun 30 11:02:40 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"N4dxeOVu7aN", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Jun 30 11:02:40 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-user13", "openId":"null", "password":"null", "passwordLastUpdated":"Thu Jun 30 11:01:55 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"N4dxeOVu7aN", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Jun 30 11:01:55 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-user12", "openId":"null", "password":"null", "passwordLastUpdated":"Thu Jun 30 11:01:10 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"N4dxeOVu7aN", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Jun 30 11:01:10 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpiry":"null", "selfRegistered":"false", "disabled":"false" }
PUT object : {"username":"msfe-kabala-user11", "openId":"null", "password":"null", "passwordLastUpdated":"Thu Jun 30 10:59:53 IST 2016", "userAuthorityGroups":"[{"class":"class org.hisp.dhis.user.UserAuthorityGroup", "id":"0", "uid":"N4dxeOVu7aN", "code":"null", "name":"null", "created":"null", "lastUpdated":"null" }]", "lastLogin":"Thu Jun 30 10:59:53 IST 2016", "restoreToken":"null", "restoreCode":"null", "restoreExpim��ם~���jب