dhis2-devs team mailing list archive

[Morten Olav Hansen <morten@xxxxxxxxx>]

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :)

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed <aamerm@xxxxxxxxxxxxxxxx>
wrote:

> Thanks Morten. Let us know if any other details are required from our end
> in this regard.
>
> Thanks
> Aamer.
>
> On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <morten@xxxxxxxxx>
> wrote:
>
>> That's interesting.
>>
>> @Lars any suggestions why this happens? I guess there is something wrong
>> with our mapping. Will have a look tomorrow.
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <vgarciabnz@xxxxxxxxx>
>> wrote:
>>
>>> Hi Morten, Sultan,
>>>
>>> I think we know the reason of duplicates: categoryOptionGroups are
>>> shared between categoryOptionGroupSets, for example, ">= 15" is used in
>>> three categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
>>> categoryOptionGroups appear duplicated as many times as they are re-used.
>>>
>>> We don't know if sharing categoryOptionGroups is something to avoid, but
>>> it seems to be the cause of this issue.
>>>
>>> Regards,
>>>
>>> Víctor
>>>
>>> On 12 September 2016 at 09:41, Morten Olav Hansen <morten@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi again
>>>>
>>>> Would it be possible to share your database with me? I haven't seen this
>>>> issues anywhere else.
>>>>
>>>> --
>>>> Morten Olav Hansen
>>>> Senior Engineer, DHIS 2
>>>> University of Oslo
>>>> http://www.dhis2.org
>>>>
>>>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>>>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> > Hi Morten,
>>>> >
>>>> > This is what our table is showing up when we say \d dataelement. I
>>>> think
>>>> > there is nothing suspicious out here.
>>>> >
>>>> >
>>>> > ​
>>>> > Regards,
>>>> > Sultan Ahamar.
>>>> >
>>>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> > wrote:
>>>> >
>>>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
>>>> >> with. Is there any constraint on the UID column in your database?
>>>> >>
>>>> >> [image: Inline image 1]
>>>> >>
>>>> >> --
>>>> >> Morten Olav Hansen
>>>> >> Senior Engineer, DHIS 2
>>>> >> University of Oslo
>>>> >> http://www.dhis2.org
>>>> >>
>>>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>>>> >> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>> >>
>>>> >>> Hi Morten,
>>>> >>>
>>>> >>> Hope you are doing good. Let us know your thoughts on the above
>>>> issue
>>>> >>> and please let us know if you need any information that you might
>>>> require.
>>>> >>>
>>>> >>> Regards,
>>>> >>> Sultan Ahamar.
>>>> >>>
>>>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>>> >>> sultanm@xxxxxxxxxxxxxxxx> wrote:
>>>> >>>
>>>> >>>> Hi Morten,
>>>> >>>>
>>>> >>>> We have pulled latest code from 2.24 and tried import /  export
>>>> again.
>>>> >>>> We have some interesting observations
>>>> >>>>
>>>> >>>> *DB snapshot:*
>>>> >>>>
>>>> >>>>
>>>> >>>> ​
>>>> >>>> There are no duplicate items in DB. We double checked.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> *Entity endpoint snapshot:*
>>>>
>>>> >>>>
>>>> >>>> ​Interestingly we found few duplicate entities when we try to fetch
>>>> >>>> this data through the API. In Maintenance app, we are not able to
>>>> see these
>>>> >>>> duplicate entities. Does it also explain why we get repeating
>>>> entries (with
>>>> >>>> same UID's) in the full export JSON as well?  Let us know on how
>>>> to proceed
>>>> >>>> and if you need more input.
>>>> >>>>
>>>> >>>> Thanks for all the help in advance.
>>>> >>>>
>>>> >>>> Regards,
>>>> >>>> Sultan Ahamar.
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <
>>>> morten@xxxxxxxxx>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Hi Vanya
>>>> >>>>>
>>>> >>>>> This should now have been fixed in master and 224. It will now do
>>>> a
>>>> >>>>> global UID check first, and report back any duplicates, if it
>>>> finds a
>>>> >>>>> duplicate it will remove them from the import and report back an
>>>> >>>>> `ErrorReport` (as it would do with other validation issues). So
>>>> if it finds
>>>> >>>>> a duplicate, you will need to set atomic mode to NONE to have it
>>>> import and
>>>> >>>>> ignore the duplicates.
>>>> >>>>>
>>>> >>>>> (btw, there was several duplicates in your file, not sure how that
>>>> >>>>> happened.. maybe something needs to be cleaned in your source
>>>> instance)
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> Morten Olav Hansen
>>>> >>>>> Senior Engineer, DHIS 2
>>>> >>>>> University of Oslo
>>>> >>>>> http://www.dhis2.org
>>>> >>>>>
>>>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>>>> morten@xxxxxxxxx>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>>> Hi Vanya
>>>> >>>>>>
>>>> >>>>>> This should have been caught by the importer... but there are
>>>> several
>>>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in
>>>> 3 category
>>>> >>>>>> option groups, same with oqeVQ71LCgY..
>>>> >>>>>>
>>>> >>>>>> I will look into making the validation process more robust.. but
>>>> at
>>>> >>>>>> least you should know that this file have issues
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> Morten Olav Hansen
>>>> >>>>>> Senior Engineer, DHIS 2
>>>> >>>>>> University of Oslo
>>>> >>>>>> http://www.dhis2.org
>>>> >>>>>>
>>>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>>>> morten@xxxxxxxxx>
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>> Ok, thanks Vanya, I'm looking into it now
>>>> >>>>>>>
>>>> >>>>>>> --
>>>> >>>>>>> Morten Olav Hansen
>>>> >>>>>>> Senior Engineer, DHIS 2
>>>> >>>>>>> University of Oslo
>>>> >>>>>>> http://www.dhis2.org
>>>> >>>>>>>
>>>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>>>> vanyas@xxxxxxxxxxxxxxxx>
>>>> >>>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>>> Hi Morten
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks for the response.
>>>> >>>>>>>>
>>>> >>>>>>>> We ran into another issue while trying to do a full export and
>>>> >>>>>>>> import.
>>>> >>>>>>>>
>>>> >>>>>>>> As per the previous conversations we took care of all data
>>>> >>>>>>>> migrations (as per the new constraints on the DB- pretty much
>>>> caught by the
>>>> >>>>>>>> Importer during the validation phase itself).
>>>> >>>>>>>>
>>>> >>>>>>>> Having done that we run into this issue:
>>>> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint
>>>> >>>>>>>> violation for the UID. We checked the data and there is no
>>>> repeating UID in
>>>> >>>>>>>> the  database.
>>>> >>>>>>>>
>>>> >>>>>>>> The error text is attached for your reference, as well the
>>>> payload
>>>> >>>>>>>> used for the import.
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks for all the help in advance.
>>>> >>>>>>>>
>>>> >>>>>>>> Regards
>>>> >>>>>>>> Vanya
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>>> >>>>>>>> morten@xxxxxxxxx> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems
>>>> some
>>>> >>>>>>>>> objects don't get their deps properly attached..
>>>> >>>>>>>>>
>>>> >>>>>>>>> --
>>>> >>>>>>>>> Morten Olav Hansen
>>>> >>>>>>>>> Senior Engineer, DHIS 2
>>>> >>>>>>>>> University of Oslo
>>>> >>>>>>>>> http://www.dhis2.org
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
>>>> >>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>>> Hi Morten,
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Please find the payload for below request
>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u
>>>> >>>>>>>>>> username:password
>>>> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:201
>>>> 6-07
>>>> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:201
>>>> 6-07>" >
>>>> >>>>>>>>>> metadata.224.newapi.created.json*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Import the payload in a fresh instance for import using
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district
>>>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>> >>>>>>>>>> output_created_besteffort.txt*
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting
>>>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it
>>>> is run with
>>>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the
>>>> complete stack
>>>> >>>>>>>>>> trace is attached earlier. filename: '
>>>> >>>>>>>>>> *output_created_besteffort_trace.txt'*
>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Let me know if the complete payload or any other details are
>>>> >>>>>>>>>> required from my end.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Thanks
>>>> >>>>>>>>>> Aamer.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <
>>>> >>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>> Hi Aamer
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Could you please share the payload of the object where this
>>>> >>>>>>>>>>> happen?
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> --
>>>> >>>>>>>>>>> Morten Olav Hansen
>>>> >>>>>>>>>>> Senior Engineer, DHIS 2
>>>> >>>>>>>>>>> University of Oslo
>>>> >>>>>>>>>>> http://www.dhis2.org
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <
>>>> >>>>>>>>>>> aamerm@xxxxxxxxxxxxxxxx> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>> Hi Morten,
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the
>>>> payload is
>>>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per
>>>> errors in
>>>> >>>>>>>>>>>> previous ImportSummary, I have done below.
>>>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the
>>>> >>>>>>>>>>>> 'data elements' and 'data element group' as well
>>>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in
>>>> >>>>>>>>>>>> payload to be same to where it is getting imported.
>>>> >>>>>>>>>>>> I have manually resolved them and ran the importer again
>>>> with
>>>> >>>>>>>>>>>> atomicMode=ALL.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> I got exception as attached in trace file
>>>> >>>>>>>>>>>> 'output_besteffort_trace.txt'
>>>> >>>>>>>>>>>> Would these kind of issues be known only after the import
>>>> has
>>>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and
>>>> as-and-when the importer
>>>> >>>>>>>>>>>> throws them? Is there any note which has the details about
>>>> the constraints
>>>> >>>>>>>>>>>> being introduced in a new DHIS version?
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata
>>>> using '
>>>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run
>>>> the
>>>> >>>>>>>>>>>> importer with atomicMode=NONE
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data
>>>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district
>>>> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE
>>>> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
>>>> >>>>>>>>>>>> output_created_besteffort_trace.txt*
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Getting below exception in trace. Complete trace also
>>>> attached.
>>>> >>>>>>>>>>>> -----
>>>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57
>>>> object(s) of
>>>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java
>>>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException:
>>>> >>>>>>>>>>>> object references an unsaved transient instance - save the
>>>> transient
>>>> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indica
>>>> tor
>>>> >>>>>>>>>>>> ------
>>>> >>>>>>>>>>>> Any suggestions please.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Thanks
>>>> >>>>>>>>>>>> Aamer.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <
>>>> >>>>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Hi Aamer
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Probably what is happening is that the new database has
>>>> some
>>>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some
>>>> issues with
>>>> >>>>>>>>>>>>> hibernate in the past, and for certain databases that
>>>> means that many of
>>>> >>>>>>>>>>>>> the constraints was not applied..
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g
>>>> `shortName`
>>>> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is
>>>> not big though,
>>>> >>>>>>>>>>>>> probably you can go through
>>>> [truncated for moderation]
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>