dhis2-devs team mailing list archive

[Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>]

Sure that makes sense.

But I am more curious to understand the following:

My existing Database (the data inside) does not follow the new constraints
that I am trying to add. Say I decide to make "shortName" unique by
altering the table to add this uniqueness constraint. But if the existing
short names column has repetitions then how does this get handled?

This is just one example but we have encountered so many such use cases.
The only part that I am unable to understand is "the new version war gets
deployed successfully" giving a false notion of version upgrade.

Thoughts?

Regards
Vanya

On Thu, Oct 6, 2016 at 6:34 PM, Lars Helge Øverland <lars@xxxxxxxxx> wrote:

> Hi Vanya,
>
> yes agree, now we typically embed the upgrade statements within DHIS 2
> itself, as we have learnt that people often forget to run upgrade
> statements which causes all kinds of problems.
>
> regards,
>
> Lars
>
>
> On Tue, Oct 4, 2016 at 11:31 AM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi Lars
>>
>> Thanks for the response.
>>
>> But have we only introduced not null constraints over different version
>> updates?
>>
>> But am still not sure how the constraints get applied if the data does
>> not abide by the rules.
>>
>> What looks like a reasonable process to me is :
>> - Update your database data to abide by the new constraints.
>> - Deploy the new war.
>>
>> Thought?
>>
>> Regards
>> Vanya
>>
>>
>> On Tue, Oct 4, 2016 at 1:27 PM, Lars Helge Øverland <lars@xxxxxxxxx>
>> wrote:
>>
>>> Hi Vanya,
>>>
>>> sure. We definitely try to avoid this. See previous email in thread. The
>>> operand constraint has been removed. We will include not-null constraints
>>> in the upgrade notes.
>>>
>>> For 2.25 <https://www.dhis2.org/225-upgrade> we will set the "created"
>>> and "lastupdated" columns of tables of "identifiable objects" to not-null.
>>>
>>> best regards,
>>>
>>> Lars
>>>
>>>
>>> On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth <vanyas@xxxxxxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Morten and Lars
>>>>
>>>> There are certain key points highlighted in the above email. Whenever a
>>>> new DHIS2 version comes out there are new integrity constraints added but
>>>> thats not part of the release or the upgrade notes.  So, whats the right
>>>> way to get information about these constraints?
>>>>
>>>> When we deploy the new war we assume that the TableAlteror will take
>>>> care of modifying the schema and also take care of adding the new
>>>> constraints. But in case the existing data does not abide by these
>>>> constraints how do we apply them?
>>>>
>>>> So, principally after a successful redeploy of the war of a higher
>>>> version can we say that the DHIS instance has been upgraded in its true
>>>> sense.
>>>> As of now it seems the war does get updated but the underlying database
>>>> does not.
>>>>
>>>> Needless to say this breaks the export /import of metadata downstream.
>>>>
>>>> Any insights about this would be appreciated.
>>>>
>>>> Regards
>>>> Vanya
>>>>
>>>>
>>>> On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed <
>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> For continuation of testing, we changed the metadata to ensure that a
>>>>> Group can be part of only one GroupSet.
>>>>> 1) We got the below issue.
>>>>> {"status":"ERROR","stats":{"total":1,"created":0,"updated":0
>>>>> ,"deleted":0,"ignored":1},"typeReports":[{"klass":"org.hisp.
>>>>> dhis.indicator.Indicator","stats":{"total":1,"created":0,"up
>>>>> dated":0,"deleted":0,"ignored":1},"objectReports":[{"klass":
>>>>> "org.hisp.dhis.indicator.Indicator","index":277,"uid":"oMbEl
>>>>> S3lhOO","errorReports":[{"message":"Missing required property
>>>>> `numerator`.","mainKlass":"org.hisp.dhis.indicator.Indicator
>>>>> ","errorKlass":"java.lang.String","errorCode":"E4000"}]}]}]}
>>>>>
>>>>> which means that Numerator is required field for an indicator. This
>>>>> was not the case in 2.21. This restriction was introduced in some version
>>>>> after 2.21. However, we modified the metadata for the indicator having this
>>>>> issue and proceeded further.
>>>>>
>>>>> 2) Now we got the below issue.
>>>>> {"httpStatus":"Internal Server Error","httpStatusCode":500,"s
>>>>> tatus":"ERROR","message":"ERROR: duplicate key value violates unique
>>>>> constraint \"dataelement_operand_unique_key\"\n  Detail: Key
>>>>> (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists."}
>>>>>
>>>>> This means that you cannot have duplicate combination of
>>>>> (dataelementid, categoryoptioncomboid) in dataelementoperand table. This
>>>>> constraint was not present in 2.21 version.
>>>>> When we see dataelementoperand table, there are lot of such
>>>>> duplicates. We would like to understand more around this constraint and why
>>>>> it was introduced.
>>>>> *Is there any note which has the details about the constraints being
>>>>> introduced in a DHIS version?*
>>>>>
>>>>> 3) Also this constraint (dataelement_operand_unique_key) is not
>>>>> present in the instance which was upgraded from 2.21 to 2.24. But it is
>>>>> present in the instance which is directly on 2.24. Any thoughts on this.
>>>>> Did the constraint not get created in upgraded instance because the data
>>>>> was not abiding to the constraint rules? What happens in a dhis upgrade if
>>>>> there is any constraint being introduced and existing data does not confirm
>>>>> to the constraint? Please suggest.
>>>>>
>>>>>
>>>>> Thanks
>>>>> Aamer.
>>>>>
>>>>>
>>>>> On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland <lars@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Aamer,
>>>>>>
>>>>>> the problem here happens because the hibernate mapping between
>>>>>> GroupSet and Group is one-to-many, ie. a Group can only be part of one
>>>>>> GroupSet.
>>>>>>
>>>>>> There is a reason for this, but it is not very good ;) When we store
>>>>>> favorites with group sets, we need to know which dimension (GroupSet) a
>>>>>> dimension item (Group) is part of. If the Group is part of many GroupSets,
>>>>>> we need another link object in between to represent which dimension it is
>>>>>> part of. This can clearly be done and this is something we plan to fix, but
>>>>>> it is a bit of work and it requires an upgrade routine to upgrade existing
>>>>>> databases.
>>>>>>
>>>>>> So I am pretty sure that is why you see these duplicates: You have
>>>>>> groups which are part of multiple group sets. In these cases, Hibernate
>>>>>> will create duplicate GroupSets as a result of the database integrity issue.
>>>>>>
>>>>>> So short term solution is to change the metadata, long term we will
>>>>>> fix this properly and allow for many-to-many between Group and GroupSet.
>>>>>>
>>>>>> regards,
>>>>>>
>>>>>> Lars
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen <morten@xxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Aamer
>>>>>>>
>>>>>>> Lars knows what is wrong, he will provide a fix shortly :)
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed <
>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Thanks Morten. Let us know if any other details are required from
>>>>>>>> our end in this regard.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Aamer.
>>>>>>>>
>>>>>>>> On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <
>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> That's interesting.
>>>>>>>>>
>>>>>>>>> @Lars any suggestions why this happens? I guess there is something
>>>>>>>>> wrong with our mapping. Will have a look tomorrow.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Morten Olav Hansen
>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>> University of Oslo
>>>>>>>>> http://www.dhis2.org
>>>>>>>>>
>>>>>>>>> On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <
>>>>>>>>> vgarciabnz@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Morten, Sultan,
>>>>>>>>>>
>>>>>>>>>> I think we know the reason of duplicates: categoryOptionGroups
>>>>>>>>>> are shared between categoryOptionGroupSets, for example, ">= 15"
>>>>>>>>>> is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In
>>>>>>>>>> the API, categoryOptionGroups appear duplicated as many times as they are
>>>>>>>>>> re-used.
>>>>>>>>>>
>>>>>>>>>> We don't know if sharing categoryOptionGroups is something to
>>>>>>>>>> avoid, but it seems to be the cause of this issue.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Víctor
>>>>>>>>>>
>>>>>>>>>> On 12 September 2016 at 09:41, Morten Olav Hansen <
>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi again
>>>>>>>>>>>
>>>>>>>>>>> Would it be possible to share your database with me? I haven't
>>>>>>>>>>> seen this
>>>>>>>>>>> issues anywhere else.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> University of Oslo
>>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>>>>>>>>>>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>> > Hi Morten,
>>>>>>>>>>> >
>>>>>>>>>>> > This is what our table is showing up when we say \d
>>>>>>>>>>> dataelement. I think
>>>>>>>>>>> > there is nothing suspicious out here.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > ​
>>>>>>>>>>> > Regards,
>>>>>>>>>>> > Sultan Ahamar.
>>>>>>>>>>> >
>>>>>>>>>>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <
>>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>>> > wrote:
>>>>>>>>>>> >
>>>>>>>>>>> >> I'm not sure what is causing it, but it shouldn't be allowed
>>>>>>>>>>> to begin
>>>>>>>>>>> >> with. Is there any constraint on the UID column in your
>>>>>>>>>>> database?
>>>>>>>>>>> >>
>>>>>>>>>>> >> [image: Inline image 1]
>>>>>>>>>>> >>
>>>>>>>>>>> >> --
>>>>>>>>>>> >> Morten Olav Hansen
>>>>>>>>>>> >> Senior Engineer, DHIS 2
>>>>>>>>>>> >> University of Oslo
>>>>>>>>>>> >> http://www.dhis2.org
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>>>>>>>>>>> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >>> Hi Morten,
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Hope you are doing good. Let us know your thoughts on the
>>>>>>>>>>> above issue
>>>>>>>>>>> >>> and please let us know if you need any information that you
>>>>>>>>>>> might require.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Regards,
>>>>>>>>>>> >>> Sultan Ahamar.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>>>>>>>>>> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>> Hi Morten,
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> We have pulled latest code from 2.24 and tried import /
>>>>>>>>>>> export again.
>>>>>>>>>>> >>>> We have some interesting observations
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> *DB snapshot:*
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> ​
>>>>>>>>>>> >>>> There are no duplicate items in DB. We double checked.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> *Entity endpoint snapshot:*
>>>>>>>>>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> ​Interestingly we found few duplicate entities when we try
>>>>>>>>>>> to fetch
>>>>>>>>>>> >>>> this data through the API. In Maintenance app, we are not
>>>>>>>>>>> able to see these
>>>>>>>>>>> >>>> duplicate entities. Does it also explain why we get
>>>>>>>>>>> repeating entries (with
>>>>>>>>>>> >>>> same UID's) in the full export JSON as well?  Let us know
>>>>>>>>>>> on how to proceed
>>>>>>>>>>> >>>> and if you need more input.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Thanks for all the help in advance.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Regards,
>>>>>>>>>>> >>>> Sultan Ahamar.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <
>>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>>> >>>> wrote:
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>> Hi Vanya
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> This should now have been fixed in master and 224. It will
>>>>>>>>>>> now do a
>>>>>>>>>>> >>>>> global UID check first, and report back any duplicates, if
>>>>>>>>>>> it finds a
>>>>>>>>>>> >>>>> duplicate it will remove them from the import and report
>>>>>>>>>>> back an
>>>>>>>>>>> >>>>> `ErrorReport` (as it would do with other validation
>>>>>>>>>>> issues). So if it finds
>>>>>>>>>>> >>>>> a duplicate, you will need to set atomic mode to NONE to
>>>>>>>>>>> have it import and
>>>>>>>>>>> >>>>> ignore the duplicates.
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> (btw, there was several duplicates in your file, not sure
>>>>>>>>>>> how that
>>>>>>>>>>> >>>>> happened.. maybe something needs to be cleaned in your
>>>>>>>>>>> source instance)
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> --
>>>>>>>>>>> >>>>> Morten Olav Hansen
>>>>>>>>>>> >>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> >>>>> University of Oslo
>>>>>>>>>>> >>>>> http://www.dhis2.org
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>>> Hi Vanya
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> This should have been caught by the importer... but there
>>>>>>>>>>> are several
>>>>>>>>>>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is
>>>>>>>>>>> used in 3 category
>>>>>>>>>>> >>>>>> option groups, same with oqeVQ71LCgY..
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> I will look into making the validation process more
>>>>>>>>>>> robust.. but at
>>>>>>>>>>> >>>>>> least you should know that this file have issues
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> --
>>>>>>>>>>> >>>>>> Morten Olav Hansen
>>>>>>>>>>> >>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> >>>>>> University of Oslo
>>>>>>>>>>> >>>>>> http://www.dhis2.org
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>>>>>>>>>>> morten@xxxxxxxxx>
>>>>>>>>>>> >>>>>> wrote:
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>>> Ok, thanks Vanya, I'm looking into it now
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> --
>>>>>>>>>>> >>>>>>> Morten Olav Hansen
>>>>>>>>>>> >>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> >>>>>>> University of Oslo
>>>>>>>>>>> >>>>>>> http://www.dhis2.org
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>>>>>>>>>>> vanyas@xxxxxxxxxxxxxxxx>
>>>>>>>>>>> >>>>>>> wrote:
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>>> Hi Morten
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Thanks for the response.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> We ran into another issue while trying to do a full
>>>>>>>>>>> export and
>>>>>>>>>>> >>>>>>>> import.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> As per the previous conversations we took care of all
>>>>>>>>>>> data
>>>>>>>>>>> >>>>>>>> migrations (as per the new constraints on the DB-
>>>>>>>>>>> pretty much caught by the
>>>>>>>>>>> >>>>>>>> Importer during the validation phase itself).
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Having done that we run into this issue:
>>>>>>>>>>> >>>>>>>> The 'categoryoptiongroup' import is failing with
>>>>>>>>>>> constraint
>>>>>>>>>>> >>>>>>>> violation for the UID. We checked the data and there is
>>>>>>>>>>> no repeating UID in
>>>>>>>>>>> >>>>>>>> the  database.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> The error text is attached for your reference, as well
>>>>>>>>>>> the payload
>>>>>>>>>>> >>>>>>>> used for the import.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Thanks for all the help in advance.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Regards
>>>>>>>>>>> >>>>>>>> Vanya
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>>>>>>>>>> >>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now..
>>>>>>>>>>> seems some
>>>>>>>>>>> >>>>>>>>> objects don't get their deps properly attached..
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> --
>>>>>>>>>>> >>>>>>>>> Morten Olav Hansen
>>>>>>>>>>> >>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> >>>>>>>>> University of Oslo
>>>>>>>>>>> >>>>>>>>> http://www.dhis2.org
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>>>>>>>>> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Hi Morten,
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Please find the payload for below request
>>>>>>>>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>>>>>>>>> >>>>>>>>>> username:password
>>>>>>>>>>> >>>>>>>>>> "http://localhost:8888/api/24/
>>>>>>>>>>> metadata?filter=created:gt:2016-07
>>>>>>>>>>> >>>>>>>>>> <http://localhost:8888/api/24/
>>>>>>>>>>> metadata?filter=created:gt:2016-07>" >
>>>>>>>>>>> >>>>>>>>>> metadata.224.newapi.created.json*
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Import the payload in a fresh instance for import
>>>>>>>>>>> using
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST
>>>>>>>>>>> --data
>>>>>>>>>>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>>>>>>>>> >>>>>>>>>> “http://localhost:8080/api/24/
>>>>>>>>>>> metadata?atomicMode=NONE
>>>>>>>>>>> >>>>>>>>>> <http://localhost:8080/api/24/
>>>>>>>>>>> metadata?atomicMode=NONE>” >
>>>>>>>>>>> >>>>>>>>>> output_created_besteffort.txt*
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am
>>>>>>>>>>> getting
>>>>>>>>>>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But
>>>>>>>>>>> if it is run with
>>>>>>>>>>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which
>>>>>>>>>>> the complete stack
>>>>>>>>>>> >>>>>>>>>> trace is attached earlier. filename: '
>>>>>>>>>>> >>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Let me know if the complete payload or any other
>>>>>>>>>>> details are
>>>>>>>>>>> >>>>>>>>>> required from my end.
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>> >>>>>>>>>> Aamer.
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>>>>>>>>> >>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> Hi Aamer
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> Could you please share the payload of the object
>>>>>>>>>>> where this
>>>>>>>>>>> >>>>>>>>>>> happen?
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> --
>>>>>>>>>>> >>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>>> >>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>>> >>>>>>>>>>> University of Oslo
>>>>>>>>>>> >>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>>>>>>>>> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Hi Morten,
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though
>>>>>>>>>>> the payload is
>>>>>>>>>>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary.
>>>>>>>>>>> As per errors in
>>>>>>>>>>> >>>>>>>>>>>> previous ImportSummary, I have done below.
>>>>>>>>>>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for
>>>>>>>>>>> each of the
>>>>>>>>>>> >>>>>>>>>>>> 'data elements' and 'data element group' as well
>>>>>>>>>>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was
>>>>>>>>>>> changed in
>>>>>>>>>>> >>>>>>>>>>>> payload to be same to where it is getting imported.
>>>>>>>>>>> >>>>>>>>>>>> I have manually resolved them and ran the importer
>>>>>>>>>>> again with
>>>>>>>>>>> >>>>>>>>>>>> atomicMode=ALL.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> I got exception as attached in trace file
>>>>>>>>>>> >>>>>>>>>>>> 'output_besteffort_trace.txt'
>>>>>>>>>>> >>>>>>>>>>>> Would these kind of issues be known only after the
>>>>>>>>>>> import has
>>>>>>>>>>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
>>>>>>>>>>> as-and-when the importer
>>>>>>>>>>> >>>>>>>>>>>> throws them? Is there any note which has the
>>>>>>>>>>> details about the constraints
>>>>>>>>>>> >>>>>>>>>>>> being introduced in a new DHIS version?
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Alternatively, If we export a filtered set of
>>>>>>>>>>> metadata using '
>>>>>>>>>>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05'
>>>>>>>>>>> and run the
>>>>>>>>>>> >>>>>>>>>>>> importer with atomicMode=NONE
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST
>>>>>>>>>>> --data
>>>>>>>>>>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>>>>>>>>> >>>>>>>>>>>> “http://localhost:8080/api/24/
>>>>>>>>>>> metadata?atomicMode=NONE
>>>>>>>>>>> >>>>>>>>>>>> <http://localhost:8080/api/24/
>>>>>>>>>>> metadata?atomicMode=NONE>” >
>>>>>>>>>>> >>>>>>>>>>>> output_created_besteffort_trace.txt*
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Getting below exception in trace. Complete trace
>>>>>>>>>>> also attached.
>>>>>>>>>>> >>>>>>>>>>>> -----
>>>>>>>>>>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57
>>>>>>>>>>> object(s) of
>>>>>>>>>>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>>>>>>>>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectE
>>>>>>>>>>> xception:
>>>>>>>>>>> >>>>>>>>>>>> object references an unsaved transient instance -
>>>>>>>>>>> save the transient
>>>>>>>>>>> >>>>>>>>>>>> instance before flushing:
>>>>>>>>>>> org.hisp.dhis.indicator.Indicator
>>>>>>>>>>> >>>>>>>>>>>> ------
>>>>>>>>>>> >>>>>>>>>>>> Any suggestions please.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Thanks
>>>>>>>>>>> >>>>>>>>>>>> Aamer.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav
>>>>>>>>>>> Hansen <
>>>>>>>>>>> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> Hi Aamer
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> Probably what is happening is that the new
>>>>>>>>>>> database has some
>>>>>>>>>>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had
>>>>>>>>>>> some issues with
>>>>>>>>>>> >>>>>>>>>>>>> hibernate in the past, and for certain databases
>>>>>>>>>>> that means that many of
>>>>>>>>>>> >>>>>>>>>>>>> the constraints was not applied..
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
>>>>>>>>>>> `shortName`
>>>>>>>>>>> >>>>>>>>>>>>> must be unique for data elements.. the list of
>>>>>>>>>>> errors is not big though,
>>>>>>>>>>> >>>>>>>>>>>>> probably you can go through
>>>>>>>>>>> [truncated for moderation]
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lars Helge Øverland
>>>>>> Lead developer, DHIS 2
>>>>>> University of Oslo
>>>>>> Skype: larshelgeoverland
>>>>>> lars@xxxxxxxxx
>>>>>> http://www.dhis2.org <https://www.dhis2.org/>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> With Regards
>>>> ThoughtWorks Technologies
>>>> Hyderabad
>>>>
>>>> --Stay Hungry Stay Foolish!!
>>>>
>>>
>>>
>>>
>>> --
>>> Lars Helge Øverland
>>> Lead developer, DHIS 2
>>> University of Oslo
>>> Skype: larshelgeoverland
>>> lars@xxxxxxxxx
>>> http://www.dhis2.org <https://www.dhis2.org/>
>>>
>>>
>>
>>
>> --
>> With Regards
>> ThoughtWorks Technologies
>> Hyderabad
>>
>> --Stay Hungry Stay Foolish!!
>>
>
>
>
> --
> Lars Helge Øverland
> Lead developer, DHIS 2
> University of Oslo
> Skype: larshelgeoverland
> lars@xxxxxxxxx
> http://www.dhis2.org <https://www.dhis2.org/>
>
>


-- 
With Regards
ThoughtWorks Technologies
Hyderabad

--Stay Hungry Stay Foolish!!