dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Not sure really where this is headed, but I am not 100% sure I am
comfortable with the direction of the conversation. Let me intervene.

We need to think about a few things here.

We are currently only supporting EPSG 4326. This is likely only
because we have really not needed any other projection (yet) but I
suspect this is going to bite us on the ass sooner or later. If we
think about other projections (UTM for instance) there are not always
decimals. If we think about the use of DHIS as as data collection tool
for humanitarian situations, or even in countries with an established
master facility list that does not use 4326 as a native projection,
then we should think about how we can extend the use of DHIS for these
use cases.  Force them to transform the data prior to importation or
to use another data collection tool?

If there is not a strong enough use case (which right now it seems
there is not) perhaps it is not really an issue. It just sort of makes
me a bit uncomfortable, when we have to potentially do some gymnastics
to get data in a format that is fast enough for mapping purposes.
Generalization and reprojection of GIS data is pretty much part and
parcel when it comes to mapping of any GIS data. If DHIS requires data
in a certain format, it should be able to transform more precise data
in arbitrary projections into its own requirements. If you look at
FAOs KIDS application, they used a custom binary format. Not because
it was standards compliant, just because it was really damn fast to
read and render. I am not saying we need to go that route, but just to
highlight that if we are focused on using DHIS as a presentation
mechanism instead of a repository, then we should focus on that.

Ideally, DHIS should be able to consume any GML stream and convert it
into a format that is required. If a user attempts to import a stream
that is not EPSG 4326, it should be able to be converted and "trimmed"
into a format that is appropriate for the application. I would also
say that DHIS should eventually should be able to "push" data to a
repository, such as Geoserver which is much more suited to handling
transformation and presentation of many different types of data.
Geoserver however is built upon GeoTools, which should be more than
capable of handling reprojection, generalization and transformation of
polygons to centroids, regardless of which back-end database system is
being used.

Regards,
Jason


On Fri, Jul 23, 2010 at 10:31 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
> In fact the xpath expression will be a bit tricky.  xpath is not
> really well suited to this kind of data where the coordinates are not
> "proper" xml elements but are encoded in a string.  In fact right the
> way through the process from gml to database we don't ever see the
> coordinates as numbers.  (Pet gripe about gml - that's really bad from
> an xml processing perspective).  So they have to be tokenized first,
> then rounded with a pretty primitive xpath 1.0 function then
> re-concatenated.  I would do that armed with a stiff whiskey but not
> otherwise.
>
> I think the better place to round these values will in fact (as Jan
> suggested) be in the orgunit  converter.  That is the one place we do
> tokenize the coordinates anyway to put '['s around them.  Can't
> promise to look at this right now but put it in a blueprint lest we
> lose it (I guess its not really a bug).
>
> Cheers
> Bob
>
> On 23 July 2010 21:13, Knut Staring <knutst@xxxxxxxxx> wrote:
>> It obviously also depends a bit on the scale - if we are talking about
>> world maps and just national borders, in my experience 2 decimals is
>> more than enough. But for a subdistrict map I think you would at least
>> want 3.
>>
>> k
>>
>> On Fri, Jul 23, 2010 at 10:11 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
>>> There is probably some difference between polygons and points. For the location of health facilities (points), I think it makes sense to retain 4-5 decimals. Polygons (which obviously have a lot more coordinates) can probably do fine with 3-4. Let us settle for 4 overall for now, and then ppl like Jan and I should do some testing.
>>>
>>> k
>>>
>>> On Fri, Jul 23, 2010 at 9:54 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>>>> 2010/7/23 Knut Staring <knutst@xxxxxxxxx>:
>>>>> 2010/7/14 Lars Helge Øverland <larshelge@xxxxxxxxx>:
>>>>>>
>>>>>> We are in the process of changing the GIS module in terms of how the
>>>>>> geographical information is persisted and presented.
>>>>>> In the snapshot version we now store the coordinates in JSON format directly
>>>>>> in the database on the OrganisationUnit.coordinates property. This gives us
>>>>>> a lot more flexibility in the way maps are presented.
>>>>>> Previously maps had to be registered explicitly either in the form of
>>>>>> GeoJson files or Shapefiles. Then the user had to select a map together with
>>>>>> indicator and period. Now the user can select an orgunit from a tree and the
>>>>>> children of that orgunit at the level below will be displayed in the map.
>>>>>> In large countries in India it is impossible to display a single map at the
>>>>>> lower levels (eg. for thousands of districts) as the map will be too heavy
>>>>>> and slow to load. Registering and managing maps for every e.g. provinces
>>>>>> will also be too cumbersome. With the current solution there is no more work
>>>>>> of registering and selecting maps - only the one time job of importing
>>>>>> geographical data/coordinates into the database.
>>>>>> Importing is a 4 step process:
>>>>>>
>>>>>> 1. Convert your shapefiles (or whatever format you have) into GML.
>>>>>> The recommended tool is FWTools, http://fwtools.maptools.org/ ;. The command
>>>>>> for converting shapefiles into GML is
>>>>>> ogr2ogr -F GML output.gml input.shp
>>>>>
>>>>> One problem is that the conversion to GML seems to generate very large
>>>>> representations, because the GML coordinates are output with 15
>>>>> decimals, whereas you would normally be happy with 5-6. Here is a
>>>>> comparison of what I get in GeoJSON vs GML (converting from the same
>>>>> shapefile):
>>>>> GeoJSON: 38.415412, 1.750212
>>>>> GML:        38.415411724082148,1.750212388592194
>>>>
>>>> I think this is just a question of understanding how many decimal
>>>> points are required/optimal/acceptable or what have you.  If you think
>>>> 4 or 5 or 6 is enough then I guess this can be reduced in the xslt
>>>> transform.
>>>>
>>>> Just saw Jan's mail.  3 is fine as well.  It could be done at gml2dxf
>>>> stage or at db insertion.  Swings 'n roundabouts.  The earlier in the
>>>> process the better from an efficiency perspective.  But java rounding
>>>> is maybe more efficient than xpath function.  Perhaps the best of both
>>>> worlds might be a simple java rounder function available to the xslt.
>>>> Ideally would be a parameter to ogr2ogr but we probably don't want our
>>>> own custom binary.
>>>>
>>>> Bob
>>>>
>>>>>
>>>>> It also seems that there the import process puts a bit too many
>>>>> brackets on points:
>>>>> Example point representation (which also has a ridiculous tail of decimals):
>>>>> [[[[37.270000000000003,-0.69]]]]
>>>>>
>>>>> Example full polygon representation (gets very big when adding
>>>>> hundreds of polygons into a layer):
>>>>> [[[[35.241617396557501,-1.042755167363498],[35.082178302163747,-0.910721897610392],[35.016946729972211,-0.895020643910023],[35.011665399182093,-0.885742630359804],[35.024369140812389,-0.87746378749961],[35.019944242042286,-0.853483690939046],[35.045208986632879,-0.856766680349123],[35.060339285653235,-0.839352562608713],[35.077610664723643,-0.860192408429204],[35.085033075563814,-0.846061280098871],[35.139987463515105,-0.881032254249694],[35.136418996765023,-0.843206506698804],[35.168249720175773,-0.842064597338777],[35.172674618945877,-0.792391540177609],[35.202078784966567,-0.793390710867632],[35.231625689657264,-0.819083671468237],[35.262599981047991,-0.804667065797898],[35.297285477858807,-0.830645503738509],[35.334540270729683,-0.814087818018119],[35.343961022949905,-0.790393198797562],[35.27601741602831,-0.724162455916004],[35.242759305917524,-0.747857075136561],[35.241331919217494,-0.694758289895313],[35.262029026367976,-0.680056206884967],[35.291861408398681,-0.655076939634379],[35.387496317300929,-0.65935909973448],[35.397773501541174,-0.647940006134211],[35.420611688741708,-0.729301048036125],[35.481989316843155,-0.77797493450727],[35.539513000854505,-0.798101086977743],[35.434029123722027,-0.895020643910023],[35.408478901791426,-0.957111965361483],[35.360233231330291,-0.98123480059205],[35.346673057679972,-0.973812389751876],[35.241617396557501,-1.042755167363498]]]]
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Knut Staring
>>>
>>>
>>
>>
>>
>> --
>> Cheers,
>> Knut Staring
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+17069260025