dhis2-devs team mailing list archive

[Uwe Wahser <uwe@xxxxxxxxx>]

Thanks for quick answers, Jason, Knut and Calle.

Unfortunately the original csv is no longer available, as I modified it a lot in
order to get my data in. I can list my steps and observations, though, if that
helps. Some things will be fine with updated documentations (header line, to
some extend the UID-problem).

Here's my log of today:

The initial file looked like the demo from the documentation, except that the
uid was filled with the county-name and the description-column was filled. This
file produced the following error in catalina.out (no error in gui, just rest of
import parameters):

* ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled task.
(TaskUtils.java [taskScheduler-5])
java.lang.NullPointerException: Name is null
        at java.lang.Enum.valueOf(Enum.java:236)
        at
org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)
        at
org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)
        at
org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)
        at
org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)
[...]

I then specified all columns as empty values and inserted values for the feature
type. DHIS2 would do the import, but reject all counties < 11char with the
posted error message.

I then filled up the field uid with spaces to complete to 11char. This worked -
all counties were imported and visible in the left hand hierarchy in the
OrgUnit-Editor and in the pivot table. 

However, when I imported the next hierarchy level in the same way, DHIS2 refused
to show the newly imported OrgUnits under their respective parents (not even
under root) - in pivot tables they showed up correctly. The counties would show
a plus-sign to indicate that there is something underneath, but the wouldn't
open up when clicking on them.

I then replaced all spaces with underscore (_) in uid, code, name etc fields of
the tables organisationunit and _orgunitstructure. As an effect this let me open
all county-nodes to see the underlying orgunits - except for one: this one had a
(') in uid, code, name etc (eg. ROCK'NROLL).

I replaced this as well. Now the hierarchies seem to show up properly everywhere
(editor, pivot tables) and I can even export all orgunits via Metadata-Export
(detailed and simple). The only problem remaining is the api-export: here DHIS2
sorts the OrgUnits nicely by name and then stops the export after 49 entries
(the json-file is properly terminated with all teh brackets) - however, that's
the most important part for me as I need to match the orgunit-codes first, when
importing data into datasets via api.

Anyway, I'm giving up for today - tomorrow I might decide to use the api's to
import the orgHierarchy in the ETL process. I'll need that anyway for nightly
updates, I was just hoping to get some quick gains by doing this manually.

Lot's of words, hope someone can get clues out of this, maybe the reason lies in
the (') although all contents were encapsulated by ("") ...

Regards, Uwe

> Jason Pickering <jason.p.pickering@xxxxxxxxx> hat am 17. Dezember 2015 um
> 08:01 geschrieben:
> 
> 
> Hi Uwe,
> Thanks for the feedback. We will take a look at it.
> 
> From the docs
> 
> "DHIS 2 supports import of metadata in the CSV format. Columns which are
> not required can be omitted in the CSV file, but the order will be
> affected. If you would like to specify columns which appear late in the
> order but not specify columns which appear early in the order you can
> include empty columns ("") for them. "
> 
> The language here is not as clear as it could be. I will edit it a bit to
> reflect the fact that as Calle says, the header is optional and ignored.
> The order of the columns however is what is important. The required columns
> are mandatory, other columns are optional as indicated in the docs,
> however, if you are using columns which appear later in the order, you must
> include empty columns, so that the order of the columns is correct.
> FeatureType should not be required, but it should be required if you choose
> to include coordinates. We will check on this.
> 
> It might be helpful if you could share a snippet of your CSV file with us,
> which might make diagnosing things a bit easier.
> 
> Regards,
> Jason
> 
> 
> On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
> wrote:
> 
> > Hi,
> >
> > 1.
> > The header row is optional - it will automatically be discarded if it's
> > there. CSV import follows a pre-defined list of columns - you can drop
> > anything at the end, but must include extra commas if you have blank
> > columns in between relevant columns.
> >
> > 2. All UIDs must be 11 characters, and starting with a letter (NOT a
> > digit).
> >
> > Regards
> > Calle
> >
> > On 17 December 2015 at 08:06, Knut Staring <knutst@xxxxxxxxx> wrote:
> >
> >> I think header is completely optional (so docs should reflect that).
> >>
> >> On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser <uwe@xxxxxxxxx> wrote:
> >>
> >>> update inline
> >>> > Uwe Wahser <uwe@xxxxxxxxx> hat am 17. Dezember 2015 um 06:23
> >>> geschrieben:
> >>> >
> >>> >
> >>> > Dear all,
> >>> >
> >>> > according to a previous post on the users list there have obviously
> >>> been
> >>> > changes
> >>> > in the CSV-format for importing OrgUnits. Some have been documented,
> >>> others
> >>> > not
> >>> > (or might be bugs):
> >>> >
> >>> > - feature format seems to be mandatory now. I don't know whether that
> >>> makes
> >>> > sense, if it does, please indicate that in the docs, too.
> >>> > - If you decide to change the docs on that, please also add a column
> >>> with the
> >>> > technical names to be included in the header line of the csv - the
> >>> names in
> >>> > the
> >>> > example differ from the names in the format specification. (or
> >>> indicate,
> >>> > whether
> >>> > the header line is optional)
> >>> > - the import summary records a funny error message. Don't know whether
> >>> it's
> >>> > new,
> >>> > but it seems to be wrong: Validation Violations:
> >>> > [ValidationViolation{property='id', message='Allowed range for length
> >>> [11,
> >>> > 11],
> >>> > length is 7', value=NAIROBI}]
> >>> UPDATE: only uid's >= 11char were imported - must be a bug
> >>> >
> >>> > I could have posted this as bug-report, but I wasn't sure, whether
> >>> those are
> >>> > bugs or features ...
> >>> >
> >>> > Regards,
> >>> >
> >>> > Uwe
> >>> >
> >>> > _______________________________________________
> >>> > Mailing list: https://launchpad.net/~dhis2-users
> >>> > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> >>> > Unsubscribe : https://launchpad.net/~dhis2-users
> >>> > More help   : https://help.launchpad.net/ListHelp
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~dhis2-users
> >>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~dhis2-users
> >>> More help   : https://help.launchpad.net/ListHelp
> >>>
> >>
> >>
> >>
> >> --
> >> Knut Staring
> >> Dept. of Informatics, University of Oslo
> >> Norway: +4791880522
> >> Skype: knutstar
> >> http://dhis2.org
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~dhis2-devs
> >> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~dhis2-devs
> >> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >
> >
> > --
> >
> > *******************************************
> >
> > Calle Hedberg
> >
> > 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
> >
> > Tel/fax (home): +27-21-685-6472
> >
> > Cell: +27-82-853-5352
> >
> > Iridium SatPhone: +8816-315-19119
> >
> > Email: calle.hedberg@xxxxxxxxx
> >
> > Skype: calle_hedberg
> >
> > *******************************************
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~dhis2-devs
> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~dhis2-devs
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
> 
> 
> -- 
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+46764147049