dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

These changes have been backported to 2.21 docs now.

https://www.dhis2.org/documentation

On Fri, Dec 18, 2015 at 10:37 AM, Lars Helge Øverland <larshelge@xxxxxxxxx>
wrote:

> Hi Uwe,
>
> unfortunately the CSV metadata import docs lagged behind a bit when we
> forked the 2.21 docs.
>
> If you go to the "latest build documentation" on the docs page you will
> the correct one. Direct link:
>
> <http://goog_851896030>
>
> http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964
>
> The reason for the "java.lang.NullPointerException: Name is null" crash
> is that we changed to using enums
> <https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html> , so that
> several the "predefined options" changed:
>
> Feature typeNoNONE | MULTI_POLYGON | POLYGON | POINT | SYMBOLGeospatial
> feature type.
>
>
> In general, uids must be 11 alphanumerics, letter first. Please have a
> look at that link, try again and let us know how it goes.
>
> regards,
>
> Lars
>
>
>
>
>
> On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser <uwe@xxxxxxxxx> wrote:
>
>> Thanks for quick answers, Jason, Knut and Calle.
>>
>> Unfortunately the original csv is no longer available, as I modified it a
>> lot in
>> order to get my data in. I can list my steps and observations, though, if
>> that
>> helps. Some things will be fine with updated documentations (header line,
>> to
>> some extend the UID-problem).
>>
>> Here's my log of today:
>>
>> The initial file looked like the demo from the documentation, except that
>> the
>> uid was filled with the county-name and the description-column was
>> filled. This
>> file produced the following error in catalina.out (no error in gui, just
>> rest of
>> import parameters):
>>
>> * ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled
>> task.
>> (TaskUtils.java [taskScheduler-5])
>> java.lang.NullPointerException: Name is null
>>         at java.lang.Enum.valueOf(Enum.java:236)
>>         at
>> org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)
>>         at
>>
>> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)
>>         at
>>
>> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)
>>         at
>>
>> org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)
>> [...]
>>
>> I then specified all columns as empty values and inserted values for the
>> feature
>> type. DHIS2 would do the import, but reject all counties < 11char with the
>> posted error message.
>>
>> I then filled up the field uid with spaces to complete to 11char. This
>> worked -
>> all counties were imported and visible in the left hand hierarchy in the
>> OrgUnit-Editor and in the pivot table.
>>
>> However, when I imported the next hierarchy level in the same way, DHIS2
>> refused
>> to show the newly imported OrgUnits under their respective parents (not
>> even
>> under root) - in pivot tables they showed up correctly. The counties
>> would show
>> a plus-sign to indicate that there is something underneath, but the
>> wouldn't
>> open up when clicking on them.
>>
>> I then replaced all spaces with underscore (_) in uid, code, name etc
>> fields of
>> the tables organisationunit and _orgunitstructure. As an effect this let
>> me open
>> all county-nodes to see the underlying orgunits - except for one: this
>> one had a
>> (') in uid, code, name etc (eg. ROCK'NROLL).
>>
>> I replaced this as well. Now the hierarchies seem to show up properly
>> everywhere
>> (editor, pivot tables) and I can even export all orgunits via
>> Metadata-Export
>> (detailed and simple). The only problem remaining is the api-export: here
>> DHIS2
>> sorts the OrgUnits nicely by name and then stops the export after 49
>> entries
>> (the json-file is properly terminated with all teh brackets) - however,
>> that's
>> the most important part for me as I need to match the orgunit-codes
>> first, when
>> importing data into datasets via api.
>>
>> Anyway, I'm giving up for today - tomorrow I might decide to use the
>> api's to
>> import the orgHierarchy in the ETL process. I'll need that anyway for
>> nightly
>> updates, I was just hoping to get some quick gains by doing this manually.
>>
>> Lot's of words, hope someone can get clues out of this, maybe the reason
>> lies in
>> the (') although all contents were encapsulated by ("") ...
>>
>> Regards, Uwe
>>
>> > Jason Pickering <jason.p.pickering@xxxxxxxxx> hat am 17. Dezember 2015
>> um
>> > 08:01 geschrieben:
>> >
>> >
>> > Hi Uwe,
>> > Thanks for the feedback. We will take a look at it.
>> >
>> > From the docs
>> >
>> > "DHIS 2 supports import of metadata in the CSV format. Columns which are
>> > not required can be omitted in the CSV file, but the order will be
>> > affected. If you would like to specify columns which appear late in the
>> > order but not specify columns which appear early in the order you can
>> > include empty columns ("") for them. "
>> >
>> > The language here is not as clear as it could be. I will edit it a bit
>> to
>> > reflect the fact that as Calle says, the header is optional and ignored.
>> > The order of the columns however is what is important. The required
>> columns
>> > are mandatory, other columns are optional as indicated in the docs,
>> > however, if you are using columns which appear later in the order, you
>> must
>> > include empty columns, so that the order of the columns is correct.
>> > FeatureType should not be required, but it should be required if you
>> choose
>> > to include coordinates. We will check on this.
>> >
>> > It might be helpful if you could share a snippet of your CSV file with
>> us,
>> > which might make diagnosing things a bit easier.
>> >
>> > Regards,
>> > Jason
>> >
>> >
>> > On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg <calle.hedberg@xxxxxxxxx
>> >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > 1.
>> > > The header row is optional - it will automatically be discarded if
>> it's
>> > > there. CSV import follows a pre-defined list of columns - you can drop
>> > > anything at the end, but must include extra commas if you have blank
>> > > columns in between relevant columns.
>> > >
>> > > 2. All UIDs must be 11 characters, and starting with a letter (NOT a
>> > > digit).
>> > >
>> > > Regards
>> > > Calle
>> > >
>> > > On 17 December 2015 at 08:06, Knut Staring <knutst@xxxxxxxxx> wrote:
>> > >
>> > >> I think header is completely optional (so docs should reflect that).
>> > >>
>> > >> On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser <uwe@xxxxxxxxx> wrote:
>> > >>
>> > >>> update inline
>> > >>> > Uwe Wahser <uwe@xxxxxxxxx> hat am 17. Dezember 2015 um 06:23
>> > >>> geschrieben:
>> > >>> >
>> > >>> >
>> > >>> > Dear all,
>> > >>> >
>> > >>> > according to a previous post on the users list there have
>> obviously
>> > >>> been
>> > >>> > changes
>> > >>> > in the CSV-format for importing OrgUnits. Some have been
>> documented,
>> > >>> others
>> > >>> > not
>> > >>> > (or might be bugs):
>> > >>> >
>> > >>> > - feature format seems to be mandatory now. I don't know whether
>> that
>> > >>> makes
>> > >>> > sense, if it does, please indicate that in the docs, too.
>> > >>> > - If you decide to change the docs on that, please also add a
>> column
>> > >>> with the
>> > >>> > technical names to be included in the header line of the csv - the
>> > >>> names in
>> > >>> > the
>> > >>> > example differ from the names in the format specification. (or
>> > >>> indicate,
>> > >>> > whether
>> > >>> > the header line is optional)
>> > >>> > - the import summary records a funny error message. Don't know
>> whether
>> > >>> it's
>> > >>> > new,
>> > >>> > but it seems to be wrong: Validation Violations:
>> > >>> > [ValidationViolation{property='id', message='Allowed range for
>> length
>> > >>> [11,
>> > >>> > 11],
>> > >>> > length is 7', value=NAIROBI}]
>> > >>> UPDATE: only uid's >= 11char were imported - must be a bug
>> > >>> >
>> > >>> > I could have posted this as bug-report, but I wasn't sure, whether
>> > >>> those are
>> > >>> > bugs or features ...
>> > >>> >
>> > >>> > Regards,
>> > >>> >
>> > >>> > Uwe
>> > >>> >
>> > >>> > _______________________________________________
>> > >>> > Mailing list: https://launchpad.net/~dhis2-users
>> > >>> > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> > >>> > Unsubscribe : https://launchpad.net/~dhis2-users
>> > >>> > More help   : https://help.launchpad.net/ListHelp
>> > >>>
>> > >>> _______________________________________________
>> > >>> Mailing list: https://launchpad.net/~dhis2-users
>> > >>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> > >>> Unsubscribe : https://launchpad.net/~dhis2-users
>> > >>> More help   : https://help.launchpad.net/ListHelp
>> > >>>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Knut Staring
>> > >> Dept. of Informatics, University of Oslo
>> > >> Norway: +4791880522
>> > >> Skype: knutstar
>> > >> http://dhis2.org
>> > >>
>> > >> _______________________________________________
>> > >> Mailing list: https://launchpad.net/~dhis2-devs
>> > >> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> > >> Unsubscribe : https://launchpad.net/~dhis2-devs
>> > >> More help   : https://help.launchpad.net/ListHelp
>> > >>
>> > >>
>> > >
>> > >
>> > > --
>> > >
>> > > *******************************************
>> > >
>> > > Calle Hedberg
>> > >
>> > > 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>> > >
>> > > Tel/fax (home): +27-21-685-6472
>> > >
>> > > Cell: +27-82-853-5352
>> > >
>> > > Iridium SatPhone: +8816-315-19119
>> > >
>> > > Email: calle.hedberg@xxxxxxxxx
>> > >
>> > > Skype: calle_hedberg
>> > >
>> > > *******************************************
>> > >
>> > >
>> > > _______________________________________________
>> > > Mailing list: https://launchpad.net/~dhis2-devs
>> > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
>> > > More help   : https://help.launchpad.net/ListHelp
>> > >
>> > >
>> >
>> >
>> > --
>> > Jason P. Pickering
>> > email: jason.p.pickering@xxxxxxxxx
>> > tel:+46764147049
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
> Lars Helge Øverland
> Lead developer, DHIS 2
> University of Oslo
> Skype: larshelgeoverland
> http://www.dhis2.org <https://www.dhis2.org>
>
>


-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org <https://www.dhis2.org>