dhis2-devs team mailing list archive

[Victor Garcia <vgarciabnz@xxxxxxxxx>]

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared
between categoryOptionGroupSets, for example, ">= 15" is used in three
categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
categoryOptionGroups appear duplicated as many times as they are re-used.

We don't know if sharing categoryOptionGroups is something to avoid, but it
seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen <morten@xxxxxxxxx> wrote:

> Hi again
>
> Would it be possible to share your database with me? I haven't seen this
> issues anywhere else.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
> sultanm@xxxxxxxxxxxxxxxx> wrote:
>
> > Hi Morten,
> >
> > This is what our table is showing up when we say \d dataelement. I think
> > there is nothing suspicious out here.
> >
> >
> > ​
> > Regards,
> > Sultan Ahamar.
> >
> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx>
> > wrote:
> >
> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
> >> with. Is there any constraint on the UID column in your database?
> >>
> >> [image: Inline image 1]
> >>
> >> --
> >> Morten Olav Hansen
> >> Senior Engineer, DHIS 2
> >> University of Oslo
> >> http://www.dhis2.org
> >>
> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
> >>
> >>> Hi Morten,
> >>>
> >>> Hope you are doing good. Let us know your thoughts on the above issue
> >>> and please let us know if you need any information that you might
> require.
> >>>
> >>> Regards,
> >>> Sultan Ahamar.
> >>>
> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
> >>>
> >>>> Hi Morten,
> >>>>
> >>>> We have pulled latest code from 2.24 and tried import /  export again.
> >>>> We have some interesting observations
> >>>>
> >>>> *DB snapshot:*
> >>>>
> >>>>
> >>>> ​
> >>>> There are no duplicate items in DB. We double checked.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> *Entity endpoint snapshot:*
> >>>>
> >>>> ​Interestingly we found few duplicate entities when we try to fetch
> >>>> this data through the API. In Maintenance app, we are not able to see
> these
> >>>> duplicate entities. Does it also explain why we get repeating entries
> (with
> >>>> same UID's) in the full export JSON as well?  Let us know on how to
> proceed
> >>>> and if you need more input.
> >>>>
> >>>> Thanks for all the help in advance.
> >>>>
> >>>> Regards,
> >>>> Sultan Ahamar.
> >>>>
> >>>>
> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@xxxxxxxxx>
> >>>> wrote:
> >>>>
> >>>>> Hi Vanya
> >>>>>
> >>>>> This should now have been fixed in master and 224. It will now do a
> >>>>> global UID check first, and report back any duplicates, if it finds a
> >>>>> duplicate it will remove them from the import and report back an
> >>>>> `ErrorReport` (as it would do with other validation issues). So if
> it finds
> >>>>> a duplicate, you will need to set atomic mode to NONE to have it
> import and
> >>>>> ignore the duplicates.
> >>>>>
> >>>>> (btw, there was several duplicates in your file, not sure how that
> >>>>> happened.. maybe something needs to be cleaned in your source
> instance)
> >>>>>
> >>>>> --
> >>>>> Morten Olav Hansen
> >>>>> Senior Engineer, DHIS 2
> >>>>> University of Oslo
> >>>>> http://www.dhis2.org
> >>>>>
> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@xxxxxxxxx
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Vanya
> >>>>>>
> >>>>>> This should have been caught by the importer... but there are
> several
> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3
> category
> >>>>>> option groups, same with oqeVQ71LCgY..
> >>>>>>
> >>>>>> I will look into making the validation process more robust.. but at
> >>>>>> least you should know that this file have issues
> >>>>>>
> >>>>>> --
> >>>>>> Morten Olav Hansen
> >>>>>> Senior Engineer, DHIS 2
> >>>>>> University of Oslo
> >>>>>> http://www.dhis2.org
> >>>>>>
> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
> morten@xxxxxxxxx>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Ok, thanks Vanya, I'm looking into it now
> >>>>>>>
> >>>>>>> --
> >>>>>>> Morten Olav Hansen
> >>>>>>> Senior Engineer, DHIS 2
> >>>>>>> University of Oslo
> >>>>>>> http://www.dhis2.org
> >>>>>>>
> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
> vanyas@xxxxxxxxxxxxxxxx>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Morten
> >>>>>>>>
> >>>>>>>> Thanks for the response.
> >>>>>>>>
> >>>>>>>> We ran into another issue while trying to do a full export and
> >>>>>>>> import.
> >>>>>>>>
> >>>>>>>> As per the previous conversations we took care of all data
> >>>>>>>> migrations (as per the new constraints on the DB- pretty much
> caught by the
> >>>>>>>> Importer during the validation phase itself).
> >>>>>>>>
> >>>>>>>> Having done that we run into this issue:
> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint
> >>>>>>>> violation for the UID. We checked the data and there is no
> repeating UID in
> >>>>>>>> the  database.
> >>>>>>>>
> >>>>>>>> The error text is attached for your reference, as well the payload
> >>>>>>>> used for the import.
> >>>>>>>>
> >>>>>>>> Thanks for all the help in advance.
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>> Vanya
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
> >>>>>>>> morten@xxxxxxxxx> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some
> >>>>>>>>> objects don't get their deps properly attached..
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Morten Olav Hansen
> >>>>>>>>> Senior Engineer, DHIS 2
> >>>>>>>>> University of Oslo
> >>>>>>>>> http://www.dhis2.org
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Morten,
> >>>>>>>>>>
> >>>>>>>>>> Please find the payload for below request
> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
> >>>>>>>>>> username:password
> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:
> 2016-07
> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:
> 2016-07>" >
> >>>>>>>>>> metadata.224.newapi.created.json*
> >>>>>>>>>>
> >>>>>>>>>> Import the payload in a fresh instance for import using
> >>>>>>>>>>
> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
> >>>>>>>>>> output_created_besteffort.txt*
> >>>>>>>>>>
> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is
> run with
> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the
> complete stack
> >>>>>>>>>> trace is attached earlier. filename: '
> >>>>>>>>>> *output_created_besteffort_trace.txt'*
> >>>>>>>>>>
> >>>>>>>>>> Let me know if the complete payload or any other details are
> >>>>>>>>>> required from my end.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>> Aamer.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
> >>>>>>>>>> morten@xxxxxxxxx> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Aamer
> >>>>>>>>>>>
> >>>>>>>>>>> Could you please share the payload of the object where this
> >>>>>>>>>>> happen?
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Morten Olav Hansen
> >>>>>>>>>>> Senior Engineer, DHIS 2
> >>>>>>>>>>> University of Oslo
> >>>>>>>>>>> http://www.dhis2.org
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Morten,
> >>>>>>>>>>>>
> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload
> is
> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per
> errors in
> >>>>>>>>>>>> previous ImportSummary, I have done below.
> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
> >>>>>>>>>>>> 'data elements' and 'data element group' as well
> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
> >>>>>>>>>>>> payload to be same to where it is getting imported.
> >>>>>>>>>>>> I have manually resolved them and ran the importer again with
> >>>>>>>>>>>> atomicMode=ALL.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I got exception as attached in trace file
> >>>>>>>>>>>> 'output_besteffort_trace.txt'
> >>>>>>>>>>>> Would these kind of issues be known only after the import has
> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
> as-and-when the importer
> >>>>>>>>>>>> throws them? Is there any note which has the details about
> the constraints
> >>>>>>>>>>>> being introduced in a new DHIS version?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata using '
> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
> >>>>>>>>>>>> importer with atomicMode=NONE
> >>>>>>>>>>>>
> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
> >>>>>>>>>>>> output_created_besteffort_trace.txt*
> >>>>>>>>>>>>
> >>>>>>>>>>>> Getting below exception in trace. Complete trace also
> attached.
> >>>>>>>>>>>> -----
> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s)
> of
> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
> >>>>>>>>>>>> object references an unsaved transient instance - save the
> transient
> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator
> >>>>>>>>>>>> ------
> >>>>>>>>>>>> Any suggestions please.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>> Aamer.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Aamer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Probably what is happening is that the new database has some
> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some
> issues with
> >>>>>>>>>>>>> hibernate in the past, and for certain databases that means
> that many of
> >>>>>>>>>>>>> the constraints was not applied..
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
> `shortName`
> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is not
> big though,
> >>>>>>>>>>>>> probably you can go through
> [truncated for moderation]
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>