← Back to team overview

dhis2-devs team mailing list archive

Re: scaling to many orgunits ...

 

Ok
On Dec 15, 2012 10:12 PM, "Brajesh Murari" <brajesh2murari@xxxxxxxxx> wrote:

> Hi Morten,
>
> I am using java version with info below
>
> brajesh.murari@mohlapbrajesh /cygdrive/d/src/dhis2/dhis-2
> $ java -version
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode)
>
> After your fix, its now working, with same above java version. If you do
> any
> java version up-gradation in scm-dhis2, don't forget to update its info
> here as well on
>
> http://www.dhis2.org/development
>
> I will upgrade my java version on 24th December in morning at 7:30 AM IST
> after
> taking fresh shower with cold/child water. I thought, may be there would
> be any real time idea implementation with your commit 9242 without
> blueprint or any bug fixing after Bob's universal declaration ie. "Been
> looking at the case of a certain West African country with *40000 orgunits
> *. *Which is actually not a huge number of objects and we could very
> likely be looking at orgunit tables bigger than this as villages and even
> households start finding their way into the tree*."
>
> On Sat, Dec 15, 2012 at 10:58 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx>wrote:
>
>> If you have an old version of Java, the commit I just did should probably
>> fix it. But I would suggest that you upgrade.
>>
>> --
>> Morten
>>
>>
>>
>> On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx>wrote:
>>
>>> There should not be any issues with that code, and as bob said our ci
>>> server seems to compile it fine (also, no other dev has complained) .
>>> Update, clean, install and if it fails update java.
>>>
>> *>>>*Developers usually don't take fresh checkout nor they like to do
> some crazy white box testing all the time, that's why perhaps they may
> not complain usually, but there are some developers who don't even relay
> on CI nightly build status update, that's why they complain about build
> failure...which was perhaps true.
> *:-)*
>
>
>>  On Dec 15, 2012 2:50 AM, "Bob Jolliffe" <bobjolliffe@xxxxxxxxx> wrote:
>>>
>>>> Brajesh I am not sure I understand the tone of your email nor the
>>>> impressive cc list.  But anyway ...
>>>>
>>>> If you have a build failure then please describe that simply on the
>>>> list and perhaps people can help you.  It is worth noting that the
>>>> continuous integration server http://apps.dhis2.org/ci/ is building
>>>> fine so at least the current trunk is fine.
>>>>
>>>> In general if there was a compilation failure on trunk (and I am not
>>>> sure I can verify because I don't think I built 9242 but you can look back
>>>> through the emails and see) an alert is sent to the devs list and it is
>>>> resolved very quickly.  Never more than a few hours.  So I think you are
>>>> safe regarding the issue being resolved before December 24.
>>>>
>>>> Please verify that you have updated to latest trunk and see if the
>>>> problem persists.  If so then give a detailed report to the list.  But
>>>> please do so in a separate thread.
>>>>
>>>> Regards
>>>> Bob
>>>>
>>>>
>>>> On 14 December 2012 21:31, Brajesh Murari <brajesh2murari@xxxxxxxxx>wrote:
>>>>
>>>>> Hello Bob,
>>>>>
>>>>> Wishing you and everyone on dhis2-dev list, a joyous Christmas season.
>>>>>
>>>>> I am getting build failure in main trunk build with info as given
>>>>> below,
>>>>>
>>>>> [INFO] -------------------------------------------------------------
>>>>> [ERROR] COMPILATION ERROR :
>>>>> [INFO] -------------------------------------------------------------
>>>>> [ERROR]
>>>>> \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22]
>>>>> cannot find symbol
>>>>> symbol  : method compare(int,int)
>>>>> location: class java.lang.Integer
>>>>> [INFO] 1 error
>>>>>
>>>>> On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>wrote:
>>>>>
>>>>>> Just sharing some thoughts before I lose them ...
>>>>>>
>>>>>> Been looking at the case of a certain West African country with 40000
>>>>>> orgunits.   Which is actually not a huge number of objects and we could
>>>>>> very likely be looking at orgunit tables bigger than this as villages and
>>>>>> even households start finding their way into the tree.
>>>>>>
>>>>>>
>>>>> One day later after your this email, Morten Olav Hansen did
>>>>> a miner change
>>>>> in one custom comparator ie. OrganisationUnitByLevelComparator.java<http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/trunk/revision/9242/dhis-2/dhis-api/src/main/java/org/hisp/dhis/organisationunit/comparator/OrganisationUnitByLevelComparator.java>
>>>>>  .
>>>>> in lp:dhis2 branch with revision number 9242. Before 9242, this
>>>>> comparator
>>>>> was looks like this given below
>>>>>
>>>>> @Override
>>>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>>>          return o1.getLevel() - o2.getLevel();
>>>>>     }
>>>>> }
>>>>>
>>>>> Now, after 9242, it has been like
>>>>>
>>>>> @Override
>>>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>>>          return Integer.compare( o1.getOrganisationUnitLevel(),
>>>>> o2.getOrganisationUnitLevel() );
>>>>>     }
>>>>> }
>>>>>
>>>>> After doing this change, committer left a log as "Updated
>>>>> OrgUnitByLevelComparator to not rely on pre-populated level field, Added
>>>>> 'level Sorted' parameter to OrgUnitController, set to true if you want
>>>>> orgUnits sored by level". Could you let us know why
>>>>> this pretty change results with DHIS2 build failure !!! I am not sure but
>>>>> is there any link in between pretty cool commit 9242 and your this email
>>>>> !!!  Pls elaborate and provide us your expert view on this issue, i am
>>>>> expecting, we should do pretty cool hand around this issue and you let us
>>>>> know how we should comes out from this
>>>>> "Build Failure" !!! I wish this issue should be closed before 23rd
>>>>> December.
>>>>>
>>>>>  Our orgunit tree (the one-true-tree) is maintained using parentid
>>>>>> pointers.  This is simple enough to maintain and updates and insertions are
>>>>>> efficient.
>>>>>>
>>>>>> Of course updates and insertions are relatively rare.  What we need
>>>>>> to do much more frequently is selecting various subtrees.  All the
>>>>>> facilities in a district, all the districts in a province etc.
>>>>>>
>>>>>> Also the depth of our trees are relatively shallow.  Most places seem
>>>>>> to have around 5 levels.  There is some trend that this starts to increase,
>>>>>> but not exponentially. We might conceive one day of 8 or even 10 levels but
>>>>>> not 100s or 1000s of levels.  Its this shallowness which makes the
>>>>>> _orgunitstructure table viable as the number of columns in that table will
>>>>>> always be within a practical limit..
>>>>>>
>>>>>> Selecting for tree structures (traversal) which are built using
>>>>>> 'parentid' is not very efficient.  Postgres offers 'WITH RECURSIVE' which
>>>>>> is pretty cool if you can get your head around it, but not supported in
>>>>>> mysql (and not necessarily fast either).
>>>>>>
>>>>>> So when selecting subtrees the _orgunitstructure table is our best
>>>>>> friend.  I use it for the mydatamart aggregatedXXvalue queries and it looks
>>>>>> like Lars uses it in the scheduled datamart job as well as well.   And it
>>>>>> is the obvious table to use when implementing filtering as I am now looking
>>>>>> at re mydatamart export.
>>>>>>
>>>>>> The problem is (with this and the other resource tables) how to
>>>>>> maintain integrity.  Currently we generate this table on demand (from the
>>>>>> user) from the parentid pointers.  If users forget to do it then all sorts
>>>>>> of things fail.  There has been some discussion on list of generating this,
>>>>>> at least nightly which would be an improvement.  There would be some
>>>>>> benefit in maintaining it dynamically, ie "triggered" during those
>>>>>> relatively rare updates and insertions of new orgunits.  One possible
>>>>>> consequence of this approach would be that the parentid on the orgunit
>>>>>> becomes effectively redundant. It could be argued that its *only* current
>>>>>> use is to generate the _orgunitstructure table.
>>>>>>
>>>>>> A consequence of the parentid being redundant is that hierarchy is
>>>>>> maintained in a separate table to the orgunits themselves.  And there is no
>>>>>> reason why there should be only one _orgunitstructure table.  There could
>>>>>> be any number, limited only by the number of hierarchies we needed to
>>>>>> maintain.  There's a thought ...
>>>>>>
>>>>>> Of course the other use for the parentid is when you are exporting a
>>>>>> bundle of orgunits in a way which reflects the (or a) hierarchy.  In which
>>>>>> case it is really kind to clients to serialize this tree as a breadth-first
>>>>>> traversal ie. they come out in order which is quick and easy to rebuild on
>>>>>> the client.  That is what I am looking at doing now to try and help out our
>>>>>> mydatamart metadata export.  And I will of course use the _orgunitstructure
>>>>>> table to make this trivial on the server side.
>>>>>>
>>>>>> But some principles derived from the above discussion:
>>>>>> 1.  try to avoid using parentid directly in code ... always link to
>>>>>> the orgunitstructuretable. This might ease its eventual demise
>>>>>> 2.  all places where we export collections of orgunits should be in a
>>>>>>  proper traversal - breadth first or depth firt shouldn't matter really but
>>>>>> I'm going to do the latter.
>>>>>>
>>>>>> Bob
>>>>>>
>>>>>
>>>>> Merry Christmas
>>>>> :-)
>>>>>
>>>>> Regards,
>>>>> Brajesh
>>>>> --
>>>>>
>>>>
> Any way, build procedure is working fine now in my system.
> thanks for informing us.
> :-)
>
> --
> Regards,
> Brajesh
>
>
>

Follow ups

References