← Back to team overview

dhis2-devs team mailing list archive

Re: scaling to many orgunits ...

 

Hi Morten,

I am using java version with info below

brajesh.murari@mohlapbrajesh /cygdrive/d/src/dhis2/dhis-2
$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode)

After your fix, its now working, with same above java version. If you do
any
java version up-gradation in scm-dhis2, don't forget to update its info
here as well on

http://www.dhis2.org/development

I will upgrade my java version on 24th December in morning at 7:30 AM IST
after
taking fresh shower with cold/child water. I thought, may be there would be
any real time idea implementation with your commit 9242 without blueprint
or any bug fixing after Bob's universal declaration ie. "Been looking at
the case of a certain West African country with *40000 orgunits*. *Which is
actually not a huge number of objects and we could very likely be looking
at orgunit tables bigger than this as villages and even households start
finding their way into the tree*."

On Sat, Dec 15, 2012 at 10:58 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx>wrote:

> If you have an old version of Java, the commit I just did should probably
> fix it. But I would suggest that you upgrade.
>
> --
> Morten
>
>
>
> On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx>wrote:
>
>> There should not be any issues with that code, and as bob said our ci
>> server seems to compile it fine (also, no other dev has complained) .
>> Update, clean, install and if it fails update java.
>>
> *>>>*Developers usually don't take fresh checkout nor they like to do
some crazy white box testing all the time, that's why perhaps they may not
complain usually, but there are some developers who don't even relay on CI
nightly build status update, that's why they complain about build
failure...which was perhaps true.
*:-)*


> On Dec 15, 2012 2:50 AM, "Bob Jolliffe" <bobjolliffe@xxxxxxxxx> wrote:
>>
>>> Brajesh I am not sure I understand the tone of your email nor the
>>> impressive cc list.  But anyway ...
>>>
>>> If you have a build failure then please describe that simply on the list
>>> and perhaps people can help you.  It is worth noting that the continuous
>>> integration server http://apps.dhis2.org/ci/ is building fine so at
>>> least the current trunk is fine.
>>>
>>> In general if there was a compilation failure on trunk (and I am not
>>> sure I can verify because I don't think I built 9242 but you can look back
>>> through the emails and see) an alert is sent to the devs list and it is
>>> resolved very quickly.  Never more than a few hours.  So I think you are
>>> safe regarding the issue being resolved before December 24.
>>>
>>> Please verify that you have updated to latest trunk and see if the
>>> problem persists.  If so then give a detailed report to the list.  But
>>> please do so in a separate thread.
>>>
>>> Regards
>>> Bob
>>>
>>>
>>> On 14 December 2012 21:31, Brajesh Murari <brajesh2murari@xxxxxxxxx>wrote:
>>>
>>>> Hello Bob,
>>>>
>>>> Wishing you and everyone on dhis2-dev list, a joyous Christmas season.
>>>>
>>>> I am getting build failure in main trunk build with info as given below,
>>>>
>>>> [INFO] -------------------------------------------------------------
>>>> [ERROR] COMPILATION ERROR :
>>>> [INFO] -------------------------------------------------------------
>>>> [ERROR]
>>>> \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22]
>>>> cannot find symbol
>>>> symbol  : method compare(int,int)
>>>> location: class java.lang.Integer
>>>> [INFO] 1 error
>>>>
>>>> On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>wrote:
>>>>
>>>>> Just sharing some thoughts before I lose them ...
>>>>>
>>>>> Been looking at the case of a certain West African country with 40000
>>>>> orgunits.   Which is actually not a huge number of objects and we could
>>>>> very likely be looking at orgunit tables bigger than this as villages and
>>>>> even households start finding their way into the tree.
>>>>>
>>>>>
>>>> One day later after your this email, Morten Olav Hansen did
>>>> a miner change
>>>> in one custom comparator ie. OrganisationUnitByLevelComparator.java<http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/trunk/revision/9242/dhis-2/dhis-api/src/main/java/org/hisp/dhis/organisationunit/comparator/OrganisationUnitByLevelComparator.java>
>>>>  .
>>>> in lp:dhis2 branch with revision number 9242. Before 9242, this
>>>> comparator
>>>> was looks like this given below
>>>>
>>>> @Override
>>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>>          return o1.getLevel() - o2.getLevel();
>>>>     }
>>>> }
>>>>
>>>> Now, after 9242, it has been like
>>>>
>>>> @Override
>>>> public int compare( OrganisationUnit o1, OrganisationUnit o2 ){
>>>>          return Integer.compare( o1.getOrganisationUnitLevel(),
>>>> o2.getOrganisationUnitLevel() );
>>>>     }
>>>> }
>>>>
>>>> After doing this change, committer left a log as "Updated
>>>> OrgUnitByLevelComparator to not rely on pre-populated level field, Added
>>>> 'level Sorted' parameter to OrgUnitController, set to true if you want
>>>> orgUnits sored by level". Could you let us know why this pretty change
>>>> results with DHIS2 build failure !!! I am not sure but is there any link in
>>>> between pretty cool commit 9242 and your this email !!!  Pls elaborate and
>>>> provide us your expert view on this issue, i am expecting, we should do
>>>> pretty cool hand around this issue and you let us know how we should comes
>>>> out from this
>>>> "Build Failure" !!! I wish this issue should be closed before 23rd
>>>> December.
>>>>
>>>>  Our orgunit tree (the one-true-tree) is maintained using parentid
>>>>> pointers.  This is simple enough to maintain and updates and insertions are
>>>>> efficient.
>>>>>
>>>>> Of course updates and insertions are relatively rare.  What we need to
>>>>> do much more frequently is selecting various subtrees.  All the facilities
>>>>> in a district, all the districts in a province etc.
>>>>>
>>>>> Also the depth of our trees are relatively shallow.  Most places seem
>>>>> to have around 5 levels.  There is some trend that this starts to increase,
>>>>> but not exponentially. We might conceive one day of 8 or even 10 levels but
>>>>> not 100s or 1000s of levels.  Its this shallowness which makes the
>>>>> _orgunitstructure table viable as the number of columns in that table will
>>>>> always be within a practical limit..
>>>>>
>>>>> Selecting for tree structures (traversal) which are built using
>>>>> 'parentid' is not very efficient.  Postgres offers 'WITH RECURSIVE' which
>>>>> is pretty cool if you can get your head around it, but not supported in
>>>>> mysql (and not necessarily fast either).
>>>>>
>>>>> So when selecting subtrees the _orgunitstructure table is our best
>>>>> friend.  I use it for the mydatamart aggregatedXXvalue queries and it looks
>>>>> like Lars uses it in the scheduled datamart job as well as well.   And it
>>>>> is the obvious table to use when implementing filtering as I am now looking
>>>>> at re mydatamart export.
>>>>>
>>>>> The problem is (with this and the other resource tables) how to
>>>>> maintain integrity.  Currently we generate this table on demand (from the
>>>>> user) from the parentid pointers.  If users forget to do it then all sorts
>>>>> of things fail.  There has been some discussion on list of generating this,
>>>>> at least nightly which would be an improvement.  There would be some
>>>>> benefit in maintaining it dynamically, ie "triggered" during those
>>>>> relatively rare updates and insertions of new orgunits.  One possible
>>>>> consequence of this approach would be that the parentid on the orgunit
>>>>> becomes effectively redundant. It could be argued that its *only* current
>>>>> use is to generate the _orgunitstructure table.
>>>>>
>>>>> A consequence of the parentid being redundant is that hierarchy is
>>>>> maintained in a separate table to the orgunits themselves.  And there is no
>>>>> reason why there should be only one _orgunitstructure table.  There could
>>>>> be any number, limited only by the number of hierarchies we needed to
>>>>> maintain.  There's a thought ...
>>>>>
>>>>> Of course the other use for the parentid is when you are exporting a
>>>>> bundle of orgunits in a way which reflects the (or a) hierarchy.  In which
>>>>> case it is really kind to clients to serialize this tree as a breadth-first
>>>>> traversal ie. they come out in order which is quick and easy to rebuild on
>>>>> the client.  That is what I am looking at doing now to try and help out our
>>>>> mydatamart metadata export.  And I will of course use the _orgunitstructure
>>>>> table to make this trivial on the server side.
>>>>>
>>>>> But some principles derived from the above discussion:
>>>>> 1.  try to avoid using parentid directly in code ... always link to
>>>>> the orgunitstructuretable. This might ease its eventual demise
>>>>> 2.  all places where we export collections of orgunits should be in a
>>>>>  proper traversal - breadth first or depth firt shouldn't matter really but
>>>>> I'm going to do the latter.
>>>>>
>>>>> Bob
>>>>>
>>>>
>>>> Merry Christmas
>>>> :-)
>>>>
>>>> Regards,
>>>> Brajesh
>>>> --
>>>>
>>>
Any way, build procedure is working fine now in my system.
thanks for informing us.
:-)

-- 
Regards,
Brajesh

Follow ups

References