dhis2-devs team mailing list archive

[Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx>]

Thanks Morten. Let us know if any other details are required from our end
in this regard.

Thanks
Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <morten@xxxxxxxxx>
wrote:

> That's interesting.
>
> @Lars any suggestions why this happens? I guess there is something wrong
> with our mapping. Will have a look tomorrow.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <vgarciabnz@xxxxxxxxx>
> wrote:
>
>> Hi Morten, Sultan,
>>
>> I think we know the reason of duplicates: categoryOptionGroups are shared
>> between categoryOptionGroupSets, for example, ">= 15" is used in three
>> categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
>> categoryOptionGroups appear duplicated as many times as they are re-used.
>>
>> We don't know if sharing categoryOptionGroups is something to avoid, but
>> it seems to be the cause of this issue.
>>
>> Regards,
>>
>> Víctor
>>
>> On 12 September 2016 at 09:41, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> Hi again
>>>
>>> Would it be possible to share your database with me? I haven't seen this
>>> issues anywhere else.
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>
>>> > Hi Morten,
>>> >
>>> > This is what our table is showing up when we say \d dataelement. I
>>> think
>>> > there is nothing suspicious out here.
>>> >
>>> >
>>> > ​
>>> > Regards,
>>> > Sultan Ahamar.
>>> >
>>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>> > wrote:
>>> >
>>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
>>> >> with. Is there any constraint on the UID column in your database?
>>> >>
>>> >> [image: Inline image 1]
>>> >>
>>> >> --
>>> >> Morten Olav Hansen
>>> >> Senior Engineer, DHIS 2
>>> >> University of Oslo
>>> >> http://www.dhis2.org
>>> >>
>>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>>> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>> >>
>>> >>> Hi Morten,
>>> >>>
>>> >>> Hope you are doing good. Let us know your thoughts on the above issue
>>> >>> and please let us know if you need any information that you might
>>> require.
>>> >>>
>>> >>> Regards,
>>> >>> Sultan Ahamar.
>>> >>>
>>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>> >>>
>>> >>>> Hi Morten,
>>> >>>>
>>> >>>> We have pulled latest code from 2.24 and tried import /  export
>>> again.
>>> >>>> We have some interesting observations
>>> >>>>
>>> >>>> *DB snapshot:*
>>> >>>>
>>> >>>>
>>> >>>> ​
>>> >>>> There are no duplicate items in DB. We double checked.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> *Entity endpoint snapshot:*
>>>
>>> >>>>
>>> >>>> ​Interestingly we found few duplicate entities when we try to fetch
>>> >>>> this data through the API. In Maintenance app, we are not able to
>>> see these
>>> >>>> duplicate entities. Does it also explain why we get repeating
>>> entries (with
>>> >>>> same UID's) in the full export JSON as well?  Let us know on how to
>>> proceed
>>> >>>> and if you need more input.
>>> >>>>
>>> >>>> Thanks for all the help in advance.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Sultan Ahamar.
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <
>>> morten@xxxxxxxxx>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi Vanya
>>> >>>>>
>>> >>>>> This should now have been fixed in master and 224. It will now do a
>>> >>>>> global UID check first, and report back any duplicates, if it
>>> finds a
>>> >>>>> duplicate it will remove them from the import and report back an
>>> >>>>> `ErrorReport` (as it would do with other validation issues). So if
>>> it finds
>>> >>>>> a duplicate, you will need to set atomic mode to NONE to have it
>>> import and
>>> >>>>> ignore the duplicates.
>>> >>>>>
>>> >>>>> (btw, there was several duplicates in your file, not sure how that
>>> >>>>> happened.. maybe something needs to be cleaned in your source
>>> instance)
>>> >>>>>
>>> >>>>> --
>>> >>>>> Morten Olav Hansen
>>> >>>>> Senior Engineer, DHIS 2
>>> >>>>> University of Oslo
>>> >>>>> http://www.dhis2.org
>>> >>>>>
>>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>>> morten@xxxxxxxxx>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Hi Vanya
>>> >>>>>>
>>> >>>>>> This should have been caught by the importer... but there are
>>> several
>>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3
>>> category
>>> >>>>>> option groups, same with oqeVQ71LCgY..
>>> >>>>>>
>>> >>>>>> I will look into making the validation process more robust.. but
>>> at
>>> >>>>>> least you should know that this file have issues
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Morten Olav Hansen
>>> >>>>>> Senior Engineer, DHIS 2
>>> >>>>>> University of Oslo
>>> >>>>>> http://www.dhis2.org
>>> >>>>>>
>>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>>> morten@xxxxxxxxx>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Ok, thanks Vanya, I'm looking into it now
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Morten Olav Hansen
>>> >>>>>>> Senior Engineer, DHIS 2
>>> >>>>>>> University of Oslo
>>> >>>>>>> http://www.dhis2.org
>>> >>>>>>>
>>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>>> vanyas@xxxxxxxxxxxxxxxx>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> Hi Morten
>>> >>>>>>>>
>>> >>>>>>>> Thanks for the response.
>>> >>>>>>>>
>>> >>>>>>>> We ran into another issue while trying to do a full export and
>>> >>>>>>>> import.
>>> >>>>>>>>
>>> >>>>>>>> As per the previous conversations we took care of all data
>>> >>>>>>>> migrations (as per the new constraints on the DB- pretty much
>>> caught by the
>>> >>>>>>>> Importer during the validation phase itself).
>>> >>>>>>>>
>>> >>>>>>>> Having done that we run into this issue:
>>> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint
>>> >>>>>>>> violation for the UID. We checked the data and there is no
>>> repeating UID in
>>> >>>>>>>> the  database.
>>> >>>>>>>>
>>> >>>>>>>> The error text is attached for your reference, as well the
>>> payload
>>> >>>>>>>> used for the import.
>>> >>>>>>>>
>>> >>>>>>>> Thanks for all the help in advance.
>>> >>>>>>>>
>>> >>>>>>>> Regards
>>> >>>>>>>> Vanya
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>> >>>>>>>> morten@xxxxxxxxx> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems
>>> some
>>> >>>>>>>>> objects don't get their deps properly attached..
>>> >>>>>>>>>
>>> >>>>>>>>> --
>>> >>>>>>>>> Morten Olav Hansen
>>> >>>>>>>>> Senior Engineer, DHIS 2
>>> >>>>>>>>> University of Oslo
>>> >>>>>>>>> http://www.dhis2.org
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Hi Morten,
>>> >>>>>>>>>>
>>> >>>>>>>>>> Please find the payload for below request
>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>> >>>>>>>>>> username:password
>>> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:201
>>> 6-07
>>> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:201
>>> 6-07>" >
>>> >>>>>>>>>> metadata.224.newapi.created.json*
>>> >>>>>>>>>>
>>> >>>>>>>>>> Import the payload in a fresh instance for import using
>>> >>>>>>>>>>
>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>> >>>>>>>>>> output_created_besteffort.txt*
>>> >>>>>>>>>>
>>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is
>>> run with
>>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the
>>> complete stack
>>> >>>>>>>>>> trace is attached earlier. filename: '
>>> >>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> Let me know if the complete payload or any other details are
>>> >>>>>>>>>> required from my end.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks
>>> >>>>>>>>>> Aamer.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>> >>>>>>>>>> morten@xxxxxxxxx> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Hi Aamer
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Could you please share the payload of the object where this
>>> >>>>>>>>>>> happen?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> --
>>> >>>>>>>>>>> Morten Olav Hansen
>>> >>>>>>>>>>> Senior Engineer, DHIS 2
>>> >>>>>>>>>>> University of Oslo
>>> >>>>>>>>>>> http://www.dhis2.org
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Hi Morten,
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the
>>> payload is
>>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per
>>> errors in
>>> >>>>>>>>>>>> previous ImportSummary, I have done below.
>>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
>>> >>>>>>>>>>>> 'data elements' and 'data element group' as well
>>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>>> >>>>>>>>>>>> payload to be same to where it is getting imported.
>>> >>>>>>>>>>>> I have manually resolved them and ran the importer again
>>> with
>>> >>>>>>>>>>>> atomicMode=ALL.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I got exception as attached in trace file
>>> >>>>>>>>>>>> 'output_besteffort_trace.txt'
>>> >>>>>>>>>>>> Would these kind of issues be known only after the import
>>> has
>>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
>>> as-and-when the importer
>>> >>>>>>>>>>>> throws them? Is there any note which has the details about
>>> the constraints
>>> >>>>>>>>>>>> being introduced in a new DHIS version?
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata
>>> using '
>>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
>>> >>>>>>>>>>>> importer with atomicMode=NONE
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>> >>>>>>>>>>>> output_created_besteffort_trace.txt*
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Getting below exception in trace. Complete trace also
>>> attached.
>>> >>>>>>>>>>>> -----
>>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57
>>> object(s) of
>>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>>> >>>>>>>>>>>> object references an unsaved transient instance - save the
>>> transient
>>> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
>>> >>>>>>>>>>>> ------
>>> >>>>>>>>>>>> Any suggestions please.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Thanks
>>> >>>>>>>>>>>> Aamer.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> Hi Aamer
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Probably what is happening is that the new database has
>>> some
>>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some
>>> issues with
>>> >>>>>>>>>>>>> hibernate in the past, and for certain databases that
>>> means that many of
>>> >>>>>>>>>>>>> the constraints was not applied..
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
>>> `shortName`
>>> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is
>>> not big though,
>>> >>>>>>>>>>>>> probably you can go through
>>> [truncated for moderation]
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>