dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

Okay. In any case, UIDs cannot have apostrophes as its [0-9a-zA-Z] so I
wouldn't spend too much time on that.

Yes we did a fix at some point for the enum stuff so try again with latest
2.21.

Lars


On Fri, Dec 18, 2015 at 12:42 PM, Uwe Wahser <uwe@xxxxxxxxx> wrote:

> Hi Lars,
>
> I did some more testing on this:
>
> 1) featuretype: I managed to upload the minimized demo from the
> documentation to
> the dhis2-demo server. On my server I got the same error message as
> before, so
> I'll just try to upload a new build of 2.21 and hopefully I'll be fine.
>
> 2) apostrophe in uid: I loaded the attached test-file into the demo
> installation: Test-province and Toast-province with one child respectively.
> Test-province received an uid with apostrophe: ROCKN'ROLL. The effect is,
> that
> you can open Toast in the editor, but not Test although the plus sign
> indicates
> children (see screenshot). Equally the api only returns a few
> organizations -
> alphabetically sorted - up to Baoma MCHP
> (https://play.dhis2.org/demo/api/organisationUnits.json).
>
> I didn't do another test for spaces, but I assume that the uid can only
> digest
> [A..Za..z0..9] without causing hick-ups elsewhere, although other
> characters get
> imported without an error message. I propose to specify this in the manual
> and
> to do some checks on valid characters when importing.
>
> Sorry for not updating the docu myself, but I still feel shy ;-)
>
> Regards,
>
> Uwe
>
> > Lars Helge Øverland <larshelge@xxxxxxxxx> hat am 18. Dezember 2015 um
> 10:37
> > geschrieben:
> >
> >
> > Hi Uwe,
> >
> > unfortunately the CSV metadata import docs lagged behind a bit when we
> > forked the 2.21 docs.
> >
> > If you go to the "latest build documentation" on the docs page you will
> the
> > correct one. Direct link:
> >
> > <goog_851896030>
> >
> http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964
> >
> > The reason for the "java.lang.NullPointerException: Name is null" crash
> is
> > that we changed to using enums
> > <https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html> , so
> that
> > several the "predefined options" changed:
> >
> > Feature typeNoNONE | MULTI_POLYGON | POLYGON | POINT | SYMBOLGeospatial
> > feature type.
> >
> >
> > In general, uids must be 11 alphanumerics, letter first. Please have a
> look
> > at that link, try again and let us know how it goes.
> >
> > regards,
> >
> > Lars
> >
> >
> >
> >
> >
> > On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser <uwe@xxxxxxxxx> wrote:
> >
> > > Thanks for quick answers, Jason, Knut and Calle.
> > >
> > > Unfortunately the original csv is no longer available, as I modified
> it a
> > > lot in
> > > order to get my data in. I can list my steps and observations, though,
> if
> > > that
> > > helps. Some things will be fine with updated documentations (header
> line,
> > > to
> > > some extend the UID-problem).
> > >
> > > Here's my log of today:
> > >
> > > The initial file looked like the demo from the documentation, except
> that
> > > the
> > > uid was filled with the county-name and the description-column was
> filled.
> > > This
> > > file produced the following error in catalina.out (no error in gui,
> just
> > > rest of
> > > import parameters):
> > >
> > > * ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled
> > > task.
> > > (TaskUtils.java [taskScheduler-5])
> > > java.lang.NullPointerException: Name is null
> > >         at java.lang.Enum.valueOf(Enum.java:236)
> > >         at
> > > org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)
> > >         at
> > >
> > >
> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)
> > >         at
> > >
> > >
> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)
> > >         at
> > >
> > >
> org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)
> > > [...]
> > >
> > > I then specified all columns as empty values and inserted values for
> the
> > > feature
> > > type. DHIS2 would do the import, but reject all counties < 11char with
> the
> > > posted error message.
> > >
> > > I then filled up the field uid with spaces to complete to 11char. This
> > > worked -
> > > all counties were imported and visible in the left hand hierarchy in
> the
> > > OrgUnit-Editor and in the pivot table.
> > >
> > > However, when I imported the next hierarchy level in the same way,
> DHIS2
> > > refused
> > > to show the newly imported OrgUnits under their respective parents (not
> > > even
> > > under root) - in pivot tables they showed up correctly. The counties
> would
> > > show
> > > a plus-sign to indicate that there is something underneath, but the
> > > wouldn't
> > > open up when clicking on them.
> > >
> > > I then replaced all spaces with underscore (_) in uid, code, name etc
> > > fields of
> > > the tables organisationunit and _orgunitstructure. As an effect this
> let
> > > me open
> > > all county-nodes to see the underlying orgunits - except for one: this
> one
> > > had a
> > > (') in uid, code, name etc (eg. ROCK'NROLL).
> > >
> > > I replaced this as well. Now the hierarchies seem to show up properly
> > > everywhere
> > > (editor, pivot tables) and I can even export all orgunits via
> > > Metadata-Export
> > > (detailed and simple). The only problem remaining is the api-export:
> here
> > > DHIS2
> > > sorts the OrgUnits nicely by name and then stops the export after 49
> > > entries
> > > (the json-file is properly terminated with all teh brackets) - however,
> > > that's
> > > the most important part for me as I need to match the orgunit-codes
> first,
> > > when
> > > importing data into datasets via api.
> > >
> > > Anyway, I'm giving up for today - tomorrow I might decide to use the
> api's
> > > to
> > > import the orgHierarchy in the ETL process. I'll need that anyway for
> > > nightly
> > > updates, I was just hoping to get some quick gains by doing this
> manually.
> > >
> > > Lot's of words, hope someone can get clues out of this, maybe the
> reason
> > > lies in
> > > the (') although all contents were encapsulated by ("") ...
> > >
> > > Regards, Uwe
> > >
> > > > Jason Pickering <jason.p.pickering@xxxxxxxxx> hat am 17. Dezember
> 2015
> > > um
> > > > 08:01 geschrieben:
> > > >
> > > >
> > > > Hi Uwe,
> > > > Thanks for the feedback. We will take a look at it.
> > > >
> > > > From the docs
> > > >
> > > > "DHIS 2 supports import of metadata in the CSV format. Columns which
> are
> > > > not required can be omitted in the CSV file, but the order will be
> > > > affected. If you would like to specify columns which appear late in
> the
> > > > order but not specify columns which appear early in the order you can
> > > > include empty columns ("") for them. "
> > > >
> > > > The language here is not as clear as it could be. I will edit it a
> bit to
> > > > reflect the fact that as Calle says, the header is optional and
> ignored.
> > > > The order of the columns however is what is important. The required
> > > columns
> > > > are mandatory, other columns are optional as indicated in the docs,
> > > > however, if you are using columns which appear later in the order,
> you
> > > must
> > > > include empty columns, so that the order of the columns is correct.
> > > > FeatureType should not be required, but it should be required if you
> > > choose
> > > > to include coordinates. We will check on this.
> > > >
> > > > It might be helpful if you could share a snippet of your CSV file
> with
> > > us,
> > > > which might make diagnosing things a bit easier.
> > > >
> > > > Regards,
> > > > Jason
> > > >
> > > >
> > > > On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg <
> calle.hedberg@xxxxxxxxx>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > 1.
> > > > > The header row is optional - it will automatically be discarded if
> it's
> > > > > there. CSV import follows a pre-defined list of columns - you can
> drop
> > > > > anything at the end, but must include extra commas if you have
> blank
> > > > > columns in between relevant columns.
> > > > >
> > > > > 2. All UIDs must be 11 characters, and starting with a letter (NOT
> a
> > > > > digit).
> > > > >
> > > > > Regards
> > > > > Calle
> > > > >
> > > > > On 17 December 2015 at 08:06, Knut Staring <knutst@xxxxxxxxx>
> wrote:
> > > > >
> > > > >> I think header is completely optional (so docs should reflect
> that).
> > > > >>
> > > > >> On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser <uwe@xxxxxxxxx>
> wrote:
> > > > >>
> > > > >>> update inline
> > > > >>> > Uwe Wahser <uwe@xxxxxxxxx> hat am 17. Dezember 2015 um 06:23
> > > > >>> geschrieben:
> > > > >>> >
> > > > >>> >
> > > > >>> > Dear all,
> > > > >>> >
> > > > >>> > according to a previous post on the users list there have
> obviously
> > > > >>> been
> > > > >>> > changes
> > > > >>> > in the CSV-format for importing OrgUnits. Some have been
> > > documented,
> > > > >>> others
> > > > >>> > not
> > > > >>> > (or might be bugs):
> > > > >>> >
> > > > >>> > - feature format seems to be mandatory now. I don't know
> whether
> > > that
> > > > >>> makes
> > > > >>> > sense, if it does, please indicate that in the docs, too.
> > > > >>> > - If you decide to change the docs on that, please also add a
> > > column
> > > > >>> with the
> > > > >>> > technical names to be included in the header line of the csv -
> the
> > > > >>> names in
> > > > >>> > the
> > > > >>> > example differ from the names in the format specification. (or
> > > > >>> indicate,
> > > > >>> > whether
> > > > >>> > the header line is optional)
> > > > >>> > - the import summary records a funny error message. Don't know
> > > whether
> > > > >>> it's
> > > > >>> > new,
> > > > >>> > but it seems to be wrong: Validation Violations:
> > > > >>> > [ValidationViolation{property='id', message='Allowed range for
> > > length
> > > > >>> [11,
> > > > >>> > 11],
> > > > >>> > length is 7', value=NAIROBI}]
> > > > >>> UPDATE: only uid's >= 11char were imported - must be a bug
> > > > >>> >
> > > > >>> > I could have posted this as bug-report, but I wasn't sure,
> whether
> > > > >>> those are
> > > > >>> > bugs or features ...
> > > > >>> >
> > > > >>> > Regards,
> > > > >>> >
> > > > >>> > Uwe
> > > > >>> >
> > > > >>> > _______________________________________________
> > > > >>> > Mailing list: https://launchpad.net/~dhis2-users
> > > > >>> > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > > > >>> > Unsubscribe : https://launchpad.net/~dhis2-users
> > > > >>> > More help   : https://help.launchpad.net/ListHelp
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> Mailing list: https://launchpad.net/~dhis2-users
> > > > >>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > > > >>> Unsubscribe : https://launchpad.net/~dhis2-users
> > > > >>> More help   : https://help.launchpad.net/ListHelp
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Knut Staring
> > > > >> Dept. of Informatics, University of Oslo
> > > > >> Norway: +4791880522
> > > > >> Skype: knutstar
> > > > >> http://dhis2.org
> > > > >>
> > > > >> _______________________________________________
> > > > >> Mailing list: https://launchpad.net/~dhis2-devs
> > > > >> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> > > > >> Unsubscribe : https://launchpad.net/~dhis2-devs
> > > > >> More help   : https://help.launchpad.net/ListHelp
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *******************************************
> > > > >
> > > > > Calle Hedberg
> > > > >
> > > > > 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
> > > > >
> > > > > Tel/fax (home): +27-21-685-6472
> > > > >
> > > > > Cell: +27-82-853-5352
> > > > >
> > > > > Iridium SatPhone: +8816-315-19119
> > > > >
> > > > > Email: calle.hedberg@xxxxxxxxx
> > > > >
> > > > > Skype: calle_hedberg
> > > > >
> > > > > *******************************************
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Mailing list: https://launchpad.net/~dhis2-devs
> > > > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> > > > > Unsubscribe : https://launchpad.net/~dhis2-devs
> > > > > More help   : https://help.launchpad.net/ListHelp
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Jason P. Pickering
> > > > email: jason.p.pickering@xxxxxxxxx
> > > > tel:+46764147049
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~dhis2-users
> > > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > More help   : https://help.launchpad.net/ListHelp
> > >
> >
> >
> >
> > --
> > Lars Helge Øverland
> > Lead developer, DHIS 2
> > University of Oslo
> > Skype: larshelgeoverland
> > http://www.dhis2.org <https://www.dhis2.org>




-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org <https://www.dhis2.org>