dhis2-devs team mailing list archive

[Morten Olav Hansen <morten@xxxxxxxxx>]

That's interesting.

@Lars any suggestions why this happens? I guess there is something wrong
with our mapping. Will have a look tomorrow.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <vgarciabnz@xxxxxxxxx> wrote:

> Hi Morten, Sultan,
>
> I think we know the reason of duplicates: categoryOptionGroups are shared
> between categoryOptionGroupSets, for example, ">= 15" is used in three
> categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
> categoryOptionGroups appear duplicated as many times as they are re-used.
>
> We don't know if sharing categoryOptionGroups is something to avoid, but
> it seems to be the cause of this issue.
>
> Regards,
>
> Víctor
>
> On 12 September 2016 at 09:41, Morten Olav Hansen <morten@xxxxxxxxx>
> wrote:
>
>> Hi again
>>
>> Would it be possible to share your database with me? I haven't seen this
>> issues anywhere else.
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>
>> > Hi Morten,
>> >
>> > This is what our table is showing up when we say \d dataelement. I think
>> > there is nothing suspicious out here.
>> >
>> >
>> > ​
>> > Regards,
>> > Sultan Ahamar.
>> >
>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>> > wrote:
>> >
>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
>> >> with. Is there any constraint on the UID column in your database?
>> >>
>> >> [image: Inline image 1]
>> >>
>> >> --
>> >> Morten Olav Hansen
>> >> Senior Engineer, DHIS 2
>> >> University of Oslo
>> >> http://www.dhis2.org
>> >>
>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
>> >>
>> >>> Hi Morten,
>> >>>
>> >>> Hope you are doing good. Let us know your thoughts on the above issue
>> >>> and please let us know if you need any information that you might
>> require.
>> >>>
>> >>> Regards,
>> >>> Sultan Ahamar.
>> >>>
>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>> >>>
>> >>>> Hi Morten,
>> >>>>
>> >>>> We have pulled latest code from 2.24 and tried import /  export
>> again.
>> >>>> We have some interesting observations
>> >>>>
>> >>>> *DB snapshot:*
>> >>>>
>> >>>>
>> >>>> ​
>> >>>> There are no duplicate items in DB. We double checked.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> *Entity endpoint snapshot:*
>>
>> >>>>
>> >>>> ​Interestingly we found few duplicate entities when we try to fetch
>> >>>> this data through the API. In Maintenance app, we are not able to
>> see these
>> >>>> duplicate entities. Does it also explain why we get repeating
>> entries (with
>> >>>> same UID's) in the full export JSON as well?  Let us know on how to
>> proceed
>> >>>> and if you need more input.
>> >>>>
>> >>>> Thanks for all the help in advance.
>> >>>>
>> >>>> Regards,
>> >>>> Sultan Ahamar.
>> >>>>
>> >>>>
>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@xxxxxxxxx
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> Hi Vanya
>> >>>>>
>> >>>>> This should now have been fixed in master and 224. It will now do a
>> >>>>> global UID check first, and report back any duplicates, if it finds
>> a
>> >>>>> duplicate it will remove them from the import and report back an
>> >>>>> `ErrorReport` (as it would do with other validation issues). So if
>> it finds
>> >>>>> a duplicate, you will need to set atomic mode to NONE to have it
>> import and
>> >>>>> ignore the duplicates.
>> >>>>>
>> >>>>> (btw, there was several duplicates in your file, not sure how that
>> >>>>> happened.. maybe something needs to be cleaned in your source
>> instance)
>> >>>>>
>> >>>>> --
>> >>>>> Morten Olav Hansen
>> >>>>> Senior Engineer, DHIS 2
>> >>>>> University of Oslo
>> >>>>> http://www.dhis2.org
>> >>>>>
>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>> morten@xxxxxxxxx>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi Vanya
>> >>>>>>
>> >>>>>> This should have been caught by the importer... but there are
>> several
>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3
>> category
>> >>>>>> option groups, same with oqeVQ71LCgY..
>> >>>>>>
>> >>>>>> I will look into making the validation process more robust.. but at
>> >>>>>> least you should know that this file have issues
>> >>>>>>
>> >>>>>> --
>> >>>>>> Morten Olav Hansen
>> >>>>>> Senior Engineer, DHIS 2
>> >>>>>> University of Oslo
>> >>>>>> http://www.dhis2.org
>> >>>>>>
>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>> morten@xxxxxxxxx>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Ok, thanks Vanya, I'm looking into it now
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Morten Olav Hansen
>> >>>>>>> Senior Engineer, DHIS 2
>> >>>>>>> University of Oslo
>> >>>>>>> http://www.dhis2.org
>> >>>>>>>
>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>> vanyas@xxxxxxxxxxxxxxxx>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Morten
>> >>>>>>>>
>> >>>>>>>> Thanks for the response.
>> >>>>>>>>
>> >>>>>>>> We ran into another issue while trying to do a full export and
>> >>>>>>>> import.
>> >>>>>>>>
>> >>>>>>>> As per the previous conversations we took care of all data
>> >>>>>>>> migrations (as per the new constraints on the DB- pretty much
>> caught by the
>> >>>>>>>> Importer during the validation phase itself).
>> >>>>>>>>
>> >>>>>>>> Having done that we run into this issue:
>> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint
>> >>>>>>>> violation for the UID. We checked the data and there is no
>> repeating UID in
>> >>>>>>>> the  database.
>> >>>>>>>>
>> >>>>>>>> The error text is attached for your reference, as well the
>> payload
>> >>>>>>>> used for the import.
>> >>>>>>>>
>> >>>>>>>> Thanks for all the help in advance.
>> >>>>>>>>
>> >>>>>>>> Regards
>> >>>>>>>> Vanya
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>> >>>>>>>> morten@xxxxxxxxx> wrote:
>> >>>>>>>>
>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems
>> some
>> >>>>>>>>> objects don't get their deps properly attached..
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Morten Olav Hansen
>> >>>>>>>>> Senior Engineer, DHIS 2
>> >>>>>>>>> University of Oslo
>> >>>>>>>>> http://www.dhis2.org
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi Morten,
>> >>>>>>>>>>
>> >>>>>>>>>> Please find the payload for below request
>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>> >>>>>>>>>> username:password
>> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:201
>> 6-07
>> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:201
>> 6-07>" >
>> >>>>>>>>>> metadata.224.newapi.created.json*
>> >>>>>>>>>>
>> >>>>>>>>>> Import the payload in a fresh instance for import using
>> >>>>>>>>>>
>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>> >>>>>>>>>> output_created_besteffort.txt*
>> >>>>>>>>>>
>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is
>> run with
>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the
>> complete stack
>> >>>>>>>>>> trace is attached earlier. filename: '
>> >>>>>>>>>> *output_created_besteffort_trace.txt'*
>>
>> >>>>>>>>>>
>> >>>>>>>>>> Let me know if the complete payload or any other details are
>> >>>>>>>>>> required from my end.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks
>> >>>>>>>>>> Aamer.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>> >>>>>>>>>> morten@xxxxxxxxx> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hi Aamer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Could you please share the payload of the object where this
>> >>>>>>>>>>> happen?
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Morten Olav Hansen
>> >>>>>>>>>>> Senior Engineer, DHIS 2
>> >>>>>>>>>>> University of Oslo
>> >>>>>>>>>>> http://www.dhis2.org
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi Morten,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload
>> is
>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per
>> errors in
>> >>>>>>>>>>>> previous ImportSummary, I have done below.
>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
>> >>>>>>>>>>>> 'data elements' and 'data element group' as well
>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>> >>>>>>>>>>>> payload to be same to where it is getting imported.
>> >>>>>>>>>>>> I have manually resolved them and ran the importer again with
>> >>>>>>>>>>>> atomicMode=ALL.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I got exception as attached in trace file
>> >>>>>>>>>>>> 'output_besteffort_trace.txt'
>> >>>>>>>>>>>> Would these kind of issues be known only after the import has
>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
>> as-and-when the importer
>> >>>>>>>>>>>> throws them? Is there any note which has the details about
>> the constraints
>> >>>>>>>>>>>> being introduced in a new DHIS version?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata using
>> '
>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
>> >>>>>>>>>>>> importer with atomicMode=NONE
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>> >>>>>>>>>>>> output_created_besteffort_trace.txt*
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Getting below exception in trace. Complete trace also
>> attached.
>> >>>>>>>>>>>> -----
>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s)
>> of
>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>> >>>>>>>>>>>> object references an unsaved transient instance - save the
>> transient
>> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
>> >>>>>>>>>>>> ------
>> >>>>>>>>>>>> Any suggestions please.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks
>> >>>>>>>>>>>> Aamer.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi Aamer
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Probably what is happening is that the new database has some
>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some
>> issues with
>> >>>>>>>>>>>>> hibernate in the past, and for certain databases that means
>> that many of
>> >>>>>>>>>>>>> the constraints was not applied..
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
>> `shortName`
>> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is
>> not big though,
>> >>>>>>>>>>>>> probably you can go through
>> [truncated for moderation]
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>