← Back to team overview

dhis2-devs team mailing list archive

Re: scaling to many orgunits ...

 

If you have an old version of Java, the commit I just did should probably
fix it. But I would suggest that you upgrade.

--
Morten



On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx>wrote:

> There should not be any issues with that code, and as bob said our ci
> server seems to compile it fine (also, no other dev has complained) .
> Update, clean, install and if it fails update java.
> On Dec 15, 2012 2:50 AM, "Bob Jolliffe" <bobjolliffe@xxxxxxxxx> wrote:
>
>> Brajesh I am not sure I understand the tone of your email nor the
>> impressive cc list.  But anyway ...
>>
>> If you have a build failure then please describe that simply on the list
>> and perhaps people can help you.  It is worth noting that the continuous
>> integration server http://apps.dhis2.org/ci/ is building fine so at
>> least the current trunk is fine.
>>
>> In general if there was a compilation failure on trunk (and I am not sure
>> I can verify because I don't think I built 9242 but you can look back
>> through the emails and see) an alert is sent to the devs list and it is
>> resolved very quickly.  Never more than a few hours.  So I think you are
>> safe regarding the issue being resolved before December 24.
>>
>> Please verify that you have updated to latest trunk and see if the
>> problem persists.  If so then give a detailed report to the list.  But
>> please do so in a separate thread.
>>
>> Regards
>> Bob
>>
>>
>> On 14 December 2012 21:31, Brajesh Murari <brajesh2murari@xxxxxxxxx>wrote:
>>
>>> Hello Bob,
>>>
>>> Wishing you and everyone on dhis2-dev list, a joyous Christmas season.
>>>
>>> I am getting build failure in main trunk build with info as given below,
>>>
>>> [INFO] -------------------------------------------------------------
>>> [ERROR] COMPILATION ERROR :
>>> [INFO] -------------------------------------------------------------
>>> [ERROR]
>>> \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22]
>>> cannot find symbol
>>> symbol  : method compare(int,int)
>>> location: class java.lang.Integer
>>> [INFO] 1 error
>>>
>>> On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>wrote:
>>>
>>>> Just sharing some thoughts before I lose them ...
>>>>
>>>> Been looking at the case of a certain West African country with 40000
>>>> orgunits.   Which is actually not a huge number of objects and we could
>>>> very likely be looking at orgunit tables bigger than this as villages and
>>>> even households start finding their way into the tree.
>>>>
>>>>
>>> One day later after your this email, Morten Olav Hansen did
>>> a miner change
>>> in one custom comparator ie. OrganisationUnitByLevelComparator.java<http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/trunk/revision/9242/dhis-2/dhis-api/src/main/java/org/hisp/dhis/organisationunit/comparator/OrganisationUnitByLevelComparator.java>
>>>  .
>>> in lp:dhis2 branch with revision number 9242. Before 9242, this
>>> comparator
>>> was looks like this given below
>>>
>>> @Override
>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>          return o1.getLevel() - o2.getLevel();
>>>     }
>>> }
>>>
>>> Now, after 9242, it has been like
>>>
>>> @Override
>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>          return Integer.compare( o1.getOrganisationUnitLevel(),
>>> o2.getOrganisationUnitLevel() );
>>>     }
>>> }
>>>
>>> After doing this change, committer left a log as "Updated
>>> OrgUnitByLevelComparator to not rely on pre-populated level field, Added
>>> 'level Sorted' parameter to OrgUnitController, set to true if you want
>>> orgUnits sored by level". Could you let us know why this pretty change
>>> results with DHIS2 build failure !!! I am not sure but is there any link in
>>> between pretty cool commit 9242 and your this email !!!  Pls elaborate and
>>> provide us your expert view on this issue, i am expecting, we should do
>>> pretty cool hand around this issue and you let us know how we should comes
>>> out from this
>>> "Build Failure" !!! I wish this issue should be closed before 23rd
>>> December.
>>>
>>>  Our orgunit tree (the one-true-tree) is maintained using parentid
>>>> pointers.  This is simple enough to maintain and updates and insertions are
>>>> efficient.
>>>>
>>>> Of course updates and insertions are relatively rare.  What we need to
>>>> do much more frequently is selecting various subtrees.  All the facilities
>>>> in a district, all the districts in a province etc.
>>>>
>>>> Also the depth of our trees are relatively shallow.  Most places seem
>>>> to have around 5 levels.  There is some trend that this starts to increase,
>>>> but not exponentially. We might conceive one day of 8 or even 10 levels but
>>>> not 100s or 1000s of levels.  Its this shallowness which makes the
>>>> _orgunitstructure table viable as the number of columns in that table will
>>>> always be within a practical limit..
>>>>
>>>> Selecting for tree structures (traversal) which are built using
>>>> 'parentid' is not very efficient.  Postgres offers 'WITH RECURSIVE' which
>>>> is pretty cool if you can get your head around it, but not supported in
>>>> mysql (and not necessarily fast either).
>>>>
>>>> So when selecting subtrees the _orgunitstructure table is our best
>>>> friend.  I use it for the mydatamart aggregatedXXvalue queries and it looks
>>>> like Lars uses it in the scheduled datamart job as well as well.   And it
>>>> is the obvious table to use when implementing filtering as I am now looking
>>>> at re mydatamart export.
>>>>
>>>> The problem is (with this and the other resource tables) how to
>>>> maintain integrity.  Currently we generate this table on demand (from the
>>>> user) from the parentid pointers.  If users forget to do it then all sorts
>>>> of things fail.  There has been some discussion on list of generating this,
>>>> at least nightly which would be an improvement.  There would be some
>>>> benefit in maintaining it dynamically, ie "triggered" during those
>>>> relatively rare updates and insertions of new orgunits.  One possible
>>>> consequence of this approach would be that the parentid on the orgunit
>>>> becomes effectively redundant. It could be argued that its *only* current
>>>> use is to generate the _orgunitstructure table.
>>>>
>>>> A consequence of the parentid being redundant is that hierarchy is
>>>> maintained in a separate table to the orgunits themselves.  And there is no
>>>> reason why there should be only one _orgunitstructure table.  There could
>>>> be any number, limited only by the number of hierarchies we needed to
>>>> maintain.  There's a thought ...
>>>>
>>>> Of course the other use for the parentid is when you are exporting a
>>>> bundle of orgunits in a way which reflects the (or a) hierarchy.  In which
>>>> case it is really kind to clients to serialize this tree as a breadth-first
>>>> traversal ie. they come out in order which is quick and easy to rebuild on
>>>> the client.  That is what I am looking at doing now to try and help out our
>>>> mydatamart metadata export.  And I will of course use the _orgunitstructure
>>>> table to make this trivial on the server side.
>>>>
>>>> But some principles derived from the above discussion:
>>>> 1.  try to avoid using parentid directly in code ... always link to the
>>>> orgunitstructuretable. This might ease its eventual demise
>>>> 2.  all places where we export collections of orgunits should be in a
>>>>  proper traversal - breadth first or depth firt shouldn't matter really but
>>>> I'm going to do the latter.
>>>>
>>>> Bob
>>>>
>>>
>>> Merry Christmas
>>> :-)
>>>
>>> Regards,
>>> Brajesh
>>> --
>>> Regards,
>>> Brajesh
>>>
>>>
>>>
>>

Follow ups

References