← Back to team overview

dhis2-devs team mailing list archive

Re: scaling to many orgunits ...

 

There should not be any issues with that code, and as bob said our ci
server seems to compile it fine (also, no other dev has complained) .
Update, clean, install and if it fails update java.
On Dec 15, 2012 2:50 AM, "Bob Jolliffe" <bobjolliffe@xxxxxxxxx> wrote:

> Brajesh I am not sure I understand the tone of your email nor the
> impressive cc list.  But anyway ...
>
> If you have a build failure then please describe that simply on the list
> and perhaps people can help you.  It is worth noting that the continuous
> integration server http://apps.dhis2.org/ci/ is building fine so at least
> the current trunk is fine.
>
> In general if there was a compilation failure on trunk (and I am not sure
> I can verify because I don't think I built 9242 but you can look back
> through the emails and see) an alert is sent to the devs list and it is
> resolved very quickly.  Never more than a few hours.  So I think you are
> safe regarding the issue being resolved before December 24.
>
> Please verify that you have updated to latest trunk and see if the problem
> persists.  If so then give a detailed report to the list.  But please do so
> in a separate thread.
>
> Regards
> Bob
>
>
> On 14 December 2012 21:31, Brajesh Murari <brajesh2murari@xxxxxxxxx>wrote:
>
>> Hello Bob,
>>
>> Wishing you and everyone on dhis2-dev list, a joyous Christmas season.
>>
>> I am getting build failure in main trunk build with info as given below,
>>
>> [INFO] -------------------------------------------------------------
>> [ERROR] COMPILATION ERROR :
>> [INFO] -------------------------------------------------------------
>> [ERROR]
>> \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22]
>> cannot find symbol
>> symbol  : method compare(int,int)
>> location: class java.lang.Integer
>> [INFO] 1 error
>>
>> On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>wrote:
>>
>>> Just sharing some thoughts before I lose them ...
>>>
>>> Been looking at the case of a certain West African country with 40000
>>> orgunits.   Which is actually not a huge number of objects and we could
>>> very likely be looking at orgunit tables bigger than this as villages and
>>> even households start finding their way into the tree.
>>>
>>>
>> One day later after your this email, Morten Olav Hansen did a miner change
>> in one custom comparator ie. OrganisationUnitByLevelComparator.java<http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/trunk/revision/9242/dhis-2/dhis-api/src/main/java/org/hisp/dhis/organisationunit/comparator/OrganisationUnitByLevelComparator.java>
>>  .
>> in lp:dhis2 branch with revision number 9242. Before 9242, this
>> comparator
>> was looks like this given below
>>
>> @Override
>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>          return o1.getLevel() - o2.getLevel();
>>     }
>> }
>>
>> Now, after 9242, it has been like
>>
>> @Override
>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>          return Integer.compare( o1.getOrganisationUnitLevel(),
>> o2.getOrganisationUnitLevel() );
>>     }
>> }
>>
>> After doing this change, committer left a log as "Updated
>> OrgUnitByLevelComparator to not rely on pre-populated level field, Added
>> 'level Sorted' parameter to OrgUnitController, set to true if you want
>> orgUnits sored by level". Could you let us know why this pretty change
>> results with DHIS2 build failure !!! I am not sure but is there any link in
>> between pretty cool commit 9242 and your this email !!!  Pls elaborate and
>> provide us your expert view on this issue, i am expecting, we should do
>> pretty cool hand around this issue and you let us know how we should comes
>> out from this
>> "Build Failure" !!! I wish this issue should be closed before 23rd
>> December.
>>
>>  Our orgunit tree (the one-true-tree) is maintained using parentid
>>> pointers.  This is simple enough to maintain and updates and insertions are
>>> efficient.
>>>
>>> Of course updates and insertions are relatively rare.  What we need to
>>> do much more frequently is selecting various subtrees.  All the facilities
>>> in a district, all the districts in a province etc.
>>>
>>> Also the depth of our trees are relatively shallow.  Most places seem to
>>> have around 5 levels.  There is some trend that this starts to increase,
>>> but not exponentially. We might conceive one day of 8 or even 10 levels but
>>> not 100s or 1000s of levels.  Its this shallowness which makes the
>>> _orgunitstructure table viable as the number of columns in that table will
>>> always be within a practical limit..
>>>
>>> Selecting for tree structures (traversal) which are built using
>>> 'parentid' is not very efficient.  Postgres offers 'WITH RECURSIVE' which
>>> is pretty cool if you can get your head around it, but not supported in
>>> mysql (and not necessarily fast either).
>>>
>>> So when selecting subtrees the _orgunitstructure table is our best
>>> friend.  I use it for the mydatamart aggregatedXXvalue queries and it looks
>>> like Lars uses it in the scheduled datamart job as well as well.   And it
>>> is the obvious table to use when implementing filtering as I am now looking
>>> at re mydatamart export.
>>>
>>> The problem is (with this and the other resource tables) how to maintain
>>> integrity.  Currently we generate this table on demand (from the user) from
>>> the parentid pointers.  If users forget to do it then all sorts of things
>>> fail.  There has been some discussion on list of generating this, at least
>>> nightly which would be an improvement.  There would be some benefit in
>>> maintaining it dynamically, ie "triggered" during those relatively rare
>>> updates and insertions of new orgunits.  One possible consequence of this
>>> approach would be that the parentid on the orgunit becomes effectively
>>> redundant. It could be argued that its *only* current use is to generate
>>> the _orgunitstructure table.
>>>
>>> A consequence of the parentid being redundant is that hierarchy is
>>> maintained in a separate table to the orgunits themselves.  And there is no
>>> reason why there should be only one _orgunitstructure table.  There could
>>> be any number, limited only by the number of hierarchies we needed to
>>> maintain.  There's a thought ...
>>>
>>> Of course the other use for the parentid is when you are exporting a
>>> bundle of orgunits in a way which reflects the (or a) hierarchy.  In which
>>> case it is really kind to clients to serialize this tree as a breadth-first
>>> traversal ie. they come out in order which is quick and easy to rebuild on
>>> the client.  That is what I am looking at doing now to try and help out our
>>> mydatamart metadata export.  And I will of course use the _orgunitstructure
>>> table to make this trivial on the server side.
>>>
>>> But some principles derived from the above discussion:
>>> 1.  try to avoid using parentid directly in code ... always link to the
>>> orgunitstructuretable. This might ease its eventual demise
>>> 2.  all places where we export collections of orgunits should be in a
>>>  proper traversal - breadth first or depth firt shouldn't matter really but
>>> I'm going to do the latter.
>>>
>>> Bob
>>>
>>
>> Merry Christmas
>> :-)
>>
>> Regards,
>> Brajesh
>> --
>> Regards,
>> Brajesh
>>
>>
>>
>

Follow ups

References