dhis2-devs team mailing list archive

[Knut Staring <knutst@xxxxxxxxx>]

On Sat, Jul 24, 2010 at 5:15 PM, Jason Pickering
<jason.p.pickering@xxxxxxxxx> wrote:
> Not sure really where this is headed, but I am not 100% sure I am
> comfortable with the direction of the conversation. Let me intervene.
>
> We need to think about a few things here.
>
> We are currently only supporting EPSG 4326. This is likely only
> because we have really not needed any other projection (yet) but I
> suspect this is going to bite us on the ass sooner or later. If we
> think about other projections (UTM for instance) there are not always
> decimals. If we think about the use of DHIS as as data collection tool
> for humanitarian situations, or even in countries with an established
> master facility list that does not use 4326 as a native projection,
> then we should think about how we can extend the use of DHIS for these
> use cases.  Force them to transform the data prior to importation or
> to use another data collection tool?

I understand your concern for the wider use of DHIS with us currently
pretty much ignoring the excellent real GIS tools and libraries out
there. But for now, I think external transformations with ogr2ogr will
work ok, since reprojection is basically just one more parameter in
the one line conversion to GML. We have already come across UTM36S
being used as a standard in Malawi. As you say below, we could easily
add some of this with GeoTools - but we have to prioritize. Plenty of
student projects around this, though, especially if you have some time
to guide them.

> If there is not a strong enough use case (which right now it seems
> there is not) perhaps it is not really an issue. It just sort of makes
> me a bit uncomfortable, when we have to potentially do some gymnastics
> to get data in a format that is fast enough for mapping purposes.
> Generalization and reprojection of GIS data is pretty much part and
> parcel when it comes to mapping of any GIS data. If DHIS requires data
> in a certain format, it should be able to transform more precise data
> in arbitrary projections into its own requirements.

> If you look at
> FAOs KIDS application, they used a custom binary format. Not because
> it was standards compliant, just because it was really damn fast to
> read and render. I am not saying we need to go that route, but just to
> highlight that if we are focused on using DHIS as a presentation
> mechanism instead of a repository, then we should focus on that.

It may be that we want to use DHIS as both a repository with full
precision (though not ridiculously artifical ones like 15 decimal
lat/lon) and have a faster way of renderin. But for a repo, I think
something like PostGIS is in order. Or we could just store things as
GML...

> Ideally, DHIS should be able to consume any GML stream and convert it
> into a format that is required. If a user attempts to import a stream
> that is not EPSG 4326, it should be able to be converted and "trimmed"
> into a format that is appropriate for the application. I would also
> say that DHIS should eventually should be able to "push" data to a
> repository, such as Geoserver which is much more suited to handling
> transformation and presentation of many different types of data.

We should be very conscient of not pushing the new, very simple
solution too far, for more complex functionality we should rather
employ Geoserver and PostGIS - and I still think this is the best
solution for a national repository. Our new way of storing orgunit
boundaries is a very small subset of such a full blown GIS solution,
but has the advantage of being simple, lightweight and portable.

> Geoserver however is built upon GeoTools, which should be more than
> capable of handling reprojection, generalization and transformation of
> polygons to centroids, regardless of which back-end database system is
> being used.

Agree that some incorporation of GeoTools might be in order. In fact,
a GIS application using GeoTools was developed back in 2005 by the
master students Trond Andresen and Lars Gunnar Vik:

http://www.duo.uio.no/sok/work.html?WORKID=27723
http://folk.uio.no/trondand/hispgis/

However, for more complex functionality such as handling raster
layers, an optional integration with Geoserver is probably the way to
go.

Knut

> Regards,
> Jason
>
>
> On Fri, Jul 23, 2010 at 10:31 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>> In fact the xpath expression will be a bit tricky.  xpath is not
>> really well suited to this kind of data where the coordinates are not
>> "proper" xml elements but are encoded in a string.  In fact right the
>> way through the process from gml to database we don't ever see the
>> coordinates as numbers.  (Pet gripe about gml - that's really bad from
>> an xml processing perspective).  So they have to be tokenized first,
>> then rounded with a pretty primitive xpath 1.0 function then
>> re-concatenated.  I would do that armed with a stiff whiskey but not
>> otherwise.
>>
>> I think the better place to round these values will in fact (as Jan
>> suggested) be in the orgunit  converter.  That is the one place we do
>> tokenize the coordinates anyway to put '['s around them.  Can't
>> promise to look at this right now but put it in a blueprint lest we
>> lose it (I guess its not really a bug).
>>
>> Cheers
>> Bob
>>
>> On 23 July 2010 21:13, Knut Staring <knutst@xxxxxxxxx> wrote:
>>> It obviously also depends a bit on the scale - if we are talking about
>>> world maps and just national borders, in my experience 2 decimals is
>>> more than enough. But for a subdistrict map I think you would at least
>>> want 3.
>>>
>>> k
>>>
>>> On Fri, Jul 23, 2010 at 10:11 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
>>>> There is probably some difference between polygons and points. For the location of health facilities (points), I think it makes sense to retain 4-5 decimals. Polygons (which obviously have a lot more coordinates) can probably do fine with 3-4. Let us settle for 4 overall for now, and then ppl like Jan and I should do some testing.
>>>>
>>>> k
>>>>
>>>> On Fri, Jul 23, 2010 at 9:54 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>>>>> 2010/7/23 Knut Staring <knutst@xxxxxxxxx>:
>>>>>> 2010/7/14 Lars Helge Øverland <larshelge@xxxxxxxxx>:
>>>>>>>
>>>>>>> We are in the process of changing the GIS module in terms of how the
>>>>>>> geographical information is persisted and presented.
>>>>>>> In the snapshot version we now store the coordinates in JSON format directly
>>>>>>> in the database on the OrganisationUnit.coordinates property. This gives us
>>>>>>> a lot more flexibility in the way maps are presented.
>>>>>>> Previously maps had to be registered explicitly either in the form of
>>>>>>> GeoJson files or Shapefiles. Then the user had to select a map together with
>>>>>>> indicator and period. Now the user can select an orgunit from a tree and the
>>>>>>> children of that orgunit at the level below will be displayed in the map.
>>>>>>> In large countries in India it is impossible to display a single map at the
>>>>>>> lower levels (eg. for thousands of districts) as the map will be too heavy
>>>>>>> and slow to load. Registering and managing maps for every e.g. provinces
>>>>>>> will also be too cumbersome. With the current solution there is no more work
>>>>>>> of registering and selecting maps - only the one time job of importing
>>>>>>> geographical data/coordinates into the database.
>>>>>>> Importing is a 4 step process:
>>>>>>>
>>>>>>> 1. Convert your shapefiles (or whatever format you have) into GML.
>>>>>>> The recommended tool is FWTools, http://fwtools.maptools.org/ ;. The command
>>>>>>> for converting shapefiles into GML is
>>>>>>> ogr2ogr -F GML output.gml input.shp
>>>>>>
>>>>>> One problem is that the conversion to GML seems to generate very large
>>>>>> representations, because the GML coordinates are output with 15
>>>>>> decimals, whereas you would normally be happy with 5-6. Here is a
>>>>>> comparison of what I get in GeoJSON vs GML (converting from the same
>>>>>> shapefile):
>>>>>> GeoJSON: 38.415412, 1.750212
>>>>>> GML:        38.415411724082148,1.750212388592194
>>>>>
>>>>> I think this is just a question of understanding how many decimal
>>>>> points are required/optimal/acceptable or what have you.  If you think
>>>>> 4 or 5 or 6 is enough then I guess this can be reduced in the xslt
>>>>> transform.
>>>>>
>>>>> Just saw Jan's mail.  3 is fine as well.  It could be done at gml2dxf
>>>>> stage or at db insertion.  Swings 'n roundabouts.  The earlier in the
>>>>> process the better from an efficiency perspective.  But java rounding
>>>>> is maybe more efficient than xpath function.  Perhaps the best of both
>>>>> worlds might be a simple java rounder function available to the xslt.
>>>>> Ideally would be a parameter to ogr2ogr but we probably don't want our
>>>>> own custom binary.
>>>>>
>>>>> Bob
>>>>>
>>>>>>
>>>>>> It also seems that there the import process puts a bit too many
>>>>>> brackets on points:
>>>>>> Example point representation (which also has a ridiculous tail of decimals):
>>>>>> [[[[37.270000000000003,-0.69]]]]
>>>>>>
>>>>>> Example full polygon representation (gets very big when adding
>>>>>> hundreds of polygons into a layer):
>>>>>> [[[[35.241617396557501,-1.042755167363498],[35.082178302163747,-0.910721897610392],[35.016946729972211,-0.895020643910023],[35.011665399182093,-0.885742630359804],[35.024369140812389,-0.87746378749961],[35.019944242042286,-0.853483690939046],[35.045208986632879,-0.856766680349123],[35.060339285653235,-0.839352562608713],[35.077610664723643,-0.860192408429204],[35.085033075563814,-0.846061280098871],[35.139987463515105,-0.881032254249694],[35.136418996765023,-0.843206506698804],[35.168249720175773,-0.842064597338777],[35.172674618945877,-0.792391540177609],[35.202078784966567,-0.793390710867632],[35.231625689657264,-0.819083671468237],[35.262599981047991,-0.804667065797898],[35.297285477858807,-0.830645503738509],[35.334540270729683,-0.814087818018119],[35.343961022949905,-0.790393198797562],[35.27601741602831,-0.724162455916004],[35.242759305917524,-0.747857075136561],[35.241331919217494,-0.694758289895313],[35.262029026367976,-0.680056206884967],[35.291861408398681,-0.655076939634379],[35.387496317300929,-0.65935909973448],[35.397773501541174,-0.647940006134211],[35.420611688741708,-0.729301048036125],[35.481989316843155,-0.77797493450727],[35.539513000854505,-0.798101086977743],[35.434029123722027,-0.895020643910023],[35.408478901791426,-0.957111965361483],[35.360233231330291,-0.98123480059205],[35.346673057679972,-0.973812389751876],[35.241617396557501,-1.042755167363498]]]]
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Knut Staring
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Knut Staring
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+17069260025
>



-- 
Cheers,
Knut Staring