dhis2-devs team mailing list archive

[Lars Helge Øverland <lars@xxxxxxxxx>]

Hi Vanya,

yes agree, now we typically embed the upgrade statements within DHIS 2
itself, as we have learnt that people often forget to run upgrade
statements which causes all kinds of problems.

regards,

Lars


On Tue, Oct 4, 2016 at 11:31 AM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx> wrote:

> Hi Lars
>
> Thanks for the response.
>
> But have we only introduced not null constraints over different version
> updates?
>
> But am still not sure how the constraints get applied if the data does not
> abide by the rules.
>
> What looks like a reasonable process to me is :
> - Update your database data to abide by the new constraints.
> - Deploy the new war.
>
> Thought?
>
> Regards
> Vanya
>
>
> On Tue, Oct 4, 2016 at 1:27 PM, Lars Helge Øverland <lars@xxxxxxxxx>
> wrote:
>
>> Hi Vanya,
>>
>> sure. We definitely try to avoid this. See previous email in thread. The
>> operand constraint has been removed. We will include not-null constraints
>> in the upgrade notes.
>>
>> For 2.25 <https://www.dhis2.org/225-upgrade> we will set the "created"
>> and "lastupdated" columns of tables of "identifiable objects" to not-null.
>>
>> best regards,
>>
>> Lars
>>
>>
>> On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
>> wrote:
>>
>>> Hi Morten and Lars
>>>
>>> There are certain key points highlighted in the above email. Whenever a
>>> new DHIS2 version comes out there are new integrity constraints added but
>>> thats not part of the release or the upgrade notes.  So, whats the right
>>> way to get information about these constraints?
>>>
>>> When we deploy the new war we assume that the TableAlteror will take
>>> care of modifying the schema and also take care of adding the new
>>> constraints. But in case the existing data does not abide by these
>>> constraints how do we apply them?
>>>
>>> So, principally after a successful redeploy of the war of a higher
>>> version can we say that the DHIS instance has been upgraded in its true
>>> sense.
>>> As of now it seems the war does get updated but the underlying database
>>> does not.
>>>
>>> Needless to say this breaks the export /import of metadata downstream.
>>>
>>> Any insights about this would be appreciated.
>>>
>>> Regards
>>> Vanya
>>>
>>>
>>> On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> For continuation of testing, we changed the metadata to ensure that a
>>>> Group can be part of only one GroupSet.
>>>> 1) We got the below issue.
>>>> {"status":"ERROR","stats":{"total":1,"created":0,"updated":0
>>>> ,"deleted":0,"ignored":1},"typeReports":[{"klass":"org.hisp.
>>>> dhis.indicator.Indicator","stats":{"total":1,"created":0,"up
>>>> dated":0,"deleted":0,"ignored":1},"objectReports":[{"klass":
>>>> "org.hisp.dhis.indicator.Indicator","index":277,"uid":"oMbEl
>>>> S3lhOO","errorReports":[{"message":"Missing required property
>>>> `numerator`.","mainKlass":"org.hisp.dhis.indicator.Indicator
>>>> ","errorKlass":"java.lang.String","errorCode":"E4000"}]}]}]}
>>>>
>>>> which means that Numerator is required field for an indicator. This was
>>>> not the case in 2.21. This restriction was introduced in some version after
>>>> 2.21. However, we modified the metadata for the indicator having this issue
>>>> and proceeded further.
>>>>
>>>> 2) Now we got the below issue.
>>>> {"httpStatus":"Internal Server Error","httpStatusCode":500,"s
>>>> tatus":"ERROR","message":"ERROR: duplicate key value violates unique
>>>> constraint \"dataelement_operand_unique_key\"\n  Detail: Key
>>>> (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists."}
>>>>
>>>> This means that you cannot have duplicate combination of
>>>> (dataelementid, categoryoptioncomboid) in dataelementoperand table. This
>>>> constraint was not present in 2.21 version.
>>>> When we see dataelementoperand table, there are lot of such duplicates.
>>>> We would like to understand more around this constraint and why it was
>>>> introduced.
>>>> *Is there any note which has the details about the constraints being
>>>> introduced in a DHIS version?*
>>>>
>>>> 3) Also this constraint (dataelement_operand_unique_key) is not
>>>> present in the instance which was upgraded from 2.21 to 2.24. But it is
>>>> present in the instance which is directly on 2.24. Any thoughts on this.
>>>> Did the constraint not get created in upgraded instance because the data
>>>> was not abiding to the constraint rules? What happens in a dhis upgrade if
>>>> there is any constraint being introduced and existing data does not confirm
>>>> to the constraint? Please suggest.
>>>>
>>>>
>>>> Thanks
>>>> Aamer.
>>>>
>>>>
>>>> On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland <lars@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Aamer,
>>>>>
>>>>> the problem here happens because the hibernate mapping between
>>>>> GroupSet and Group is one-to-many, ie. a Group can only be part of one
>>>>> GroupSet.
>>>>>
>>>>> There is a reason for this, but it is not very good ;) When we store
>>>>> favorites with group sets, we need to know which dimension (GroupSet) a
>>>>> dimension item (Group) is part of. If the Group is part of many GroupSets,
>>>>> we need another link object in between to represent which dimension it is
>>>>> part of. This can clearly be done and this is something we plan to fix, but
>>>>> it is a bit of work and it requires an upgrade routine to upgrade existing
>>>>> databases.
>>>>>
>>>>> So I am pretty sure that is why you see these duplicates: You have
>>>>> groups which are part of multiple group sets. In these cases, Hibernate
>>>>> will create duplicate GroupSets as a result of the database integrity issue.
>>>>>
>>>>> So short term solution is to change the metadata, long term we will
>>>>> fix this properly and allow for many-to-many between Group and GroupSet.
>>>>>
>>>>> regards,
>>>>>
>>>>> Lars
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Aamer
>>>>>>
>>>>>> Lars knows what is wrong, he will provide a fix shortly :)
>>>>>>
>>>>>> --
>>>>>> Morten Olav Hansen
>>>>>> Senior Engineer, DHIS 2
>>>>>> University of Oslo
>>>>>> http://www.dhis2.org
>>>>>>
>>>>>> On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed <
>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>
>>>>>>> Thanks Morten. Let us know if any other details are required from
>>>>>>> our end in this regard.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Aamer.
>>>>>>>
>>>>>>> On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <
>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> That's interesting.
>>>>>>>>
>>>>>>>> @Lars any suggestions why this happens? I guess there is something
>>>>>>>> wrong with our mapping. Will have a look tomorrow.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Morten Olav Hansen
>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>> University of Oslo
>>>>>>>> http://www.dhis2.org
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <
>>>>>>>> vgarciabnz@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi Morten, Sultan,
>>>>>>>>>
>>>>>>>>> I think we know the reason of duplicates: categoryOptionGroups are
>>>>>>>>> shared between categoryOptionGroupSets, for example, ">= 15" is
>>>>>>>>> used in three categoryOptionGroupSets (Paediatrics, HIV and general). In
>>>>>>>>> the API, categoryOptionGroups appear duplicated as many times as they are
>>>>>>>>> re-used.
>>>>>>>>>
>>>>>>>>> We don't know if sharing categoryOptionGroups is something to
>>>>>>>>> avoid, but it seems to be the cause of this issue.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Víctor
>>>>>>>>>
>>>>>>>>> On 12 September 2016 at 09:41, Morten Olav Hansen <
>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Hi again
>>>>>>>>>>
>>>>>>>>>> Would it be possible to share your database with me? I haven't
>>>>>>>>>> seen this
>>>>>>>>>> issues anywhere else.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> University of Oslo
>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>>>>>>>>>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>> > Hi Morten,
>>>>>>>>>> >
>>>>>>>>>> > This is what our table is showing up when we say \d
>>>>>>>>>> dataelement. I think
>>>>>>>>>> > there is nothing suspicious out here.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > ​
>>>>>>>>>> > Regards,
>>>>>>>>>> > Sultan Ahamar.
>>>>>>>>>> >
>>>>>>>>>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>> > wrote:
>>>>>>>>>> >
>>>>>>>>>> >> I'm not sure what is causing it, but it shouldn't be allowed
>>>>>>>>>> to begin
>>>>>>>>>> >> with. Is there any constraint on the UID column in your
>>>>>>>>>> database?
>>>>>>>>>> >>
>>>>>>>>>> >> [image: Inline image 1]
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> Morten Olav Hansen
>>>>>>>>>> >> Senior Engineer, DHIS 2
>>>>>>>>>> >> University of Oslo
>>>>>>>>>> >> http://www.dhis2.org
>>>>>>>>>> >>
>>>>>>>>>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>>>>>>>>>> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >>> Hi Morten,
>>>>>>>>>> >>>
>>>>>>>>>> >>> Hope you are doing good. Let us know your thoughts on the
>>>>>>>>>> above issue
>>>>>>>>>> >>> and please let us know if you need any information that you
>>>>>>>>>> might require.
>>>>>>>>>> >>>
>>>>>>>>>> >>> Regards,
>>>>>>>>>> >>> Sultan Ahamar.
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>>>>>>>>> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>>> Hi Morten,
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> We have pulled latest code from 2.24 and tried import /
>>>>>>>>>> export again.
>>>>>>>>>> >>>> We have some interesting observations
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> *DB snapshot:*
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> ​
>>>>>>>>>> >>>> There are no duplicate items in DB. We double checked.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> *Entity endpoint snapshot:*
>>>>>>>>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> ​Interestingly we found few duplicate entities when we try
>>>>>>>>>> to fetch
>>>>>>>>>> >>>> this data through the API. In Maintenance app, we are not
>>>>>>>>>> able to see these
>>>>>>>>>> >>>> duplicate entities. Does it also explain why we get
>>>>>>>>>> repeating entries (with
>>>>>>>>>> >>>> same UID's) in the full export JSON as well?  Let us know on
>>>>>>>>>> how to proceed
>>>>>>>>>> >>>> and if you need more input.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Thanks for all the help in advance.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Regards,
>>>>>>>>>> >>>> Sultan Ahamar.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>> >>>> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>> Hi Vanya
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> This should now have been fixed in master and 224. It will
>>>>>>>>>> now do a
>>>>>>>>>> >>>>> global UID check first, and report back any duplicates, if
>>>>>>>>>> it finds a
>>>>>>>>>> >>>>> duplicate it will remove them from the import and report
>>>>>>>>>> back an
>>>>>>>>>> >>>>> `ErrorReport` (as it would do with other validation
>>>>>>>>>> issues). So if it finds
>>>>>>>>>> >>>>> a duplicate, you will need to set atomic mode to NONE to
>>>>>>>>>> have it import and
>>>>>>>>>> >>>>> ignore the duplicates.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> (btw, there was several duplicates in your file, not sure
>>>>>>>>>> how that
>>>>>>>>>> >>>>> happened.. maybe something needs to be cleaned in your
>>>>>>>>>> source instance)
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> --
>>>>>>>>>> >>>>> Morten Olav Hansen
>>>>>>>>>> >>>>> Senior Engineer, DHIS 2
>>>>>>>>>> >>>>> University of Oslo
>>>>>>>>>> >>>>> http://www.dhis2.org
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>> Hi Vanya
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> This should have been caught by the importer... but there
>>>>>>>>>> are several
>>>>>>>>>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is
>>>>>>>>>> used in 3 category
>>>>>>>>>> >>>>>> option groups, same with oqeVQ71LCgY..
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> I will look into making the validation process more
>>>>>>>>>> robust.. but at
>>>>>>>>>> >>>>>> least you should know that this file have issues
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> --
>>>>>>>>>> >>>>>> Morten Olav Hansen
>>>>>>>>>> >>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> >>>>>> University of Oslo
>>>>>>>>>> >>>>>> http://www.dhis2.org
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>> >>>>>> wrote:
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>>> Ok, thanks Vanya, I'm looking into it now
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> --
>>>>>>>>>> >>>>>>> Morten Olav Hansen
>>>>>>>>>> >>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> >>>>>>> University of Oslo
>>>>>>>>>> >>>>>>> http://www.dhis2.org
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>>>>>>>>>> vanyas@xxxxxxxxxxxxxxxx>
>>>>>>>>>> >>>>>>> wrote:
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>>> Hi Morten
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Thanks for the response.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> We ran into another issue while trying to do a full
>>>>>>>>>> export and
>>>>>>>>>> >>>>>>>> import.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> As per the previous conversations we took care of all
>>>>>>>>>> data
>>>>>>>>>> >>>>>>>> migrations (as per the new constraints on the DB- pretty
>>>>>>>>>> much caught by the
>>>>>>>>>> >>>>>>>> Importer during the validation phase itself).
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Having done that we run into this issue:
>>>>>>>>>> >>>>>>>> The 'categoryoptiongroup' import is failing with
>>>>>>>>>> constraint
>>>>>>>>>> >>>>>>>> violation for the UID. We checked the data and there is
>>>>>>>>>> no repeating UID in
>>>>>>>>>> >>>>>>>> the  database.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> The error text is attached for your reference, as well
>>>>>>>>>> the payload
>>>>>>>>>> >>>>>>>> used for the import.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Thanks for all the help in advance.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Regards
>>>>>>>>>> >>>>>>>> Vanya
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>>>>>>>>> >>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now..
>>>>>>>>>> seems some
>>>>>>>>>> >>>>>>>>> objects don't get their deps properly attached..
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> --
>>>>>>>>>> >>>>>>>>> Morten Olav Hansen
>>>>>>>>>> >>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> >>>>>>>>> University of Oslo
>>>>>>>>>> >>>>>>>>> http://www.dhis2.org
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>>>>>>>> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Hi Morten,
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Please find the payload for below request
>>>>>>>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>>> >>>>>>>>>> username:password
>>>>>>>>>> >>>>>>>>>> "http://localhost:8888/api/24/
>>>>>>>>>> metadata?filter=created:gt:2016-07
>>>>>>>>>> >>>>>>>>>> <http://localhost:8888/api/24/
>>>>>>>>>> metadata?filter=created:gt:2016-07>" >
>>>>>>>>>> >>>>>>>>>> metadata.224.newapi.created.json*
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Import the payload in a fresh instance for import using
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST
>>>>>>>>>> --data
>>>>>>>>>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>>>>>>>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>>>>>>>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>”
>>>>>>>>>> >
>>>>>>>>>> >>>>>>>>>> output_created_besteffort.txt*
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am
>>>>>>>>>> getting
>>>>>>>>>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But
>>>>>>>>>> if it is run with
>>>>>>>>>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which
>>>>>>>>>> the complete stack
>>>>>>>>>> >>>>>>>>>> trace is attached earlier. filename: '
>>>>>>>>>> >>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Let me know if the complete payload or any other
>>>>>>>>>> details are
>>>>>>>>>> >>>>>>>>>> required from my end.
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>> >>>>>>>>>> Aamer.
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>>>>>>>> >>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> Hi Aamer
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> Could you please share the payload of the object
>>>>>>>>>> where this
>>>>>>>>>> >>>>>>>>>>> happen?
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> --
>>>>>>>>>> >>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>> >>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> >>>>>>>>>>> University of Oslo
>>>>>>>>>> >>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>>>>>>> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Hi Morten,
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the
>>>>>>>>>> payload is
>>>>>>>>>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary.
>>>>>>>>>> As per errors in
>>>>>>>>>> >>>>>>>>>>>> previous ImportSummary, I have done below.
>>>>>>>>>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each
>>>>>>>>>> of the
>>>>>>>>>> >>>>>>>>>>>> 'data elements' and 'data element group' as well
>>>>>>>>>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was
>>>>>>>>>> changed in
>>>>>>>>>> >>>>>>>>>>>> payload to be same to where it is getting imported.
>>>>>>>>>> >>>>>>>>>>>> I have manually resolved them and ran the importer
>>>>>>>>>> again with
>>>>>>>>>> >>>>>>>>>>>> atomicMode=ALL.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> I got exception as attached in trace file
>>>>>>>>>> >>>>>>>>>>>> 'output_besteffort_trace.txt'
>>>>>>>>>> >>>>>>>>>>>> Would these kind of issues be known only after the
>>>>>>>>>> import has
>>>>>>>>>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
>>>>>>>>>> as-and-when the importer
>>>>>>>>>> >>>>>>>>>>>> throws them? Is there any note which has the details
>>>>>>>>>> about the constraints
>>>>>>>>>> >>>>>>>>>>>> being introduced in a new DHIS version?
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Alternatively, If we export a filtered set of
>>>>>>>>>> metadata using '
>>>>>>>>>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and
>>>>>>>>>> run the
>>>>>>>>>> >>>>>>>>>>>> importer with atomicMode=NONE
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST
>>>>>>>>>> --data
>>>>>>>>>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>>>>>> >>>>>>>>>>>> “http://localhost:8080/api/24/
>>>>>>>>>> metadata?atomicMode=NONE
>>>>>>>>>> >>>>>>>>>>>> <http://localhost:8080/api/24/
>>>>>>>>>> metadata?atomicMode=NONE>” >
>>>>>>>>>> >>>>>>>>>>>> output_created_besteffort_trace.txt*
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Getting below exception in trace. Complete trace
>>>>>>>>>> also attached.
>>>>>>>>>> >>>>>>>>>>>> -----
>>>>>>>>>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57
>>>>>>>>>> object(s) of
>>>>>>>>>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>>>>>>>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectE
>>>>>>>>>> xception:
>>>>>>>>>> >>>>>>>>>>>> object references an unsaved transient instance -
>>>>>>>>>> save the transient
>>>>>>>>>> >>>>>>>>>>>> instance before flushing:
>>>>>>>>>> org.hisp.dhis.indicator.Indicator
>>>>>>>>>> >>>>>>>>>>>> ------
>>>>>>>>>> >>>>>>>>>>>> Any suggestions please.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Thanks
>>>>>>>>>> >>>>>>>>>>>> Aamer.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen
>>>>>>>>>> <
>>>>>>>>>> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> Hi Aamer
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> Probably what is happening is that the new database
>>>>>>>>>> has some
>>>>>>>>>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had
>>>>>>>>>> some issues with
>>>>>>>>>> >>>>>>>>>>>>> hibernate in the past, and for certain databases
>>>>>>>>>> that means that many of
>>>>>>>>>> >>>>>>>>>>>>> the constraints was not applied..
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
>>>>>>>>>> `shortName`
>>>>>>>>>> >>>>>>>>>>>>> must be unique for data elements.. the list of
>>>>>>>>>> errors is not big though,
>>>>>>>>>> >>>>>>>>>>>>> probably you can go through
>>>>>>>>>> [truncated for moderation]
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lars Helge Øverland
>>>>> Lead developer, DHIS 2
>>>>> University of Oslo
>>>>> Skype: larshelgeoverland
>>>>> lars@xxxxxxxxx
>>>>> http://www.dhis2.org <https://www.dhis2.org/>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> With Regards
>>> ThoughtWorks Technologies
>>> Hyderabad
>>>
>>> --Stay Hungry Stay Foolish!!
>>>
>>
>>
>>
>> --
>> Lars Helge Øverland
>> Lead developer, DHIS 2
>> University of Oslo
>> Skype: larshelgeoverland
>> lars@xxxxxxxxx
>> http://www.dhis2.org <https://www.dhis2.org/>
>>
>>
>
>
> --
> With Regards
> ThoughtWorks Technologies
> Hyderabad
>
> --Stay Hungry Stay Foolish!!
>



-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
lars@xxxxxxxxx
http://www.dhis2.org <https://www.dhis2.org/>