dhis2-users team mailing list archive

[Aaron C White <aaroncwhite@xxxxxxxxx>]

Isaiah,

I've found the best way is to convert rows of a CSV into JSON structure and
then post to the appropriate endpoint using httr.

I've made a slew of helper functions that make it easy for me to do along
with some markdown files starting to explain things.  When I started
working with dhis2, I was an analyst and most comfortable using dataframes,
so this kind of fits that perspective.  I've learned a lot along the way
and plan to refactor things at some point.  Eventually, I'd like to make it
a real R package- just haven't had the time!
https://github.com/aaroncwhite/dhis2R

Feel free to PM me for more detail/explanation.

Aaron


--
Aaron C. White
919.522.5585

On Mon, Jan 29, 2018 at 12:24 PM, Jason Pickering <
jason.p.pickering@xxxxxxxxx> wrote:

> Hi Isaiah,
>
> Yes, R is definitely capable of creating a file which you can import into
> DHIS2.
>
> If you want to use CSV, be sure to have your file with  the "row.names=FALSE
> " parameter and be sure to convert all "NA" values to with " na = '' "
>
> You can also post the file directly to DHIS2. Have a look at
> https://github.com/jason-p-pickering/dhis2-data-munging/
> blob/master/bootstrap/bootstrap.Rmd or more details of how to do this in
> the section "Loading the data elements"
>
> Regards,
> Jason
>
>
> On Mon, Jan 29, 2018 at 6:04 PM, Timothy Harding <tharding@xxxxxxxxxxxxxx>
> wrote:
>
>> Isaiah:
>>
>> org_kqmh_list_test.csv:
>>
>> Has an extra column
>>
>> [image: Inline image 1]
>>
>> 1, 2, 3, etc.. should not be in there unless the OUs are named 1, 2, 3.
>>
>> Probably should be LF instead of CRLF:
>> [image: Inline image 2]
>> but I'm not sure it makes a difference here.
>>
>> Also you've got a lot of NA's in there, not sure those are needed unless
>> you are setting something to be NA, best just have ,, rather than ,NA,
>>
>> [image: Inline image 3]
>>
>>
>> Good luck!
>>
>>
>> *Timothy Harding*
>> Sr. Systems Analyst, BAO Systems
>> +1 202-536-1541 | tharding@xxxxxxxxxxxxxx | http://www.baosystems.com | Skype:
>> hardingt@xxxxxxxxx | 2900 K Street, Suite 406, Washington D.C. 20007
>> <https://maps.google.com/?q=2900+K+Street,+Suite+406,+Washington+D.C.+20007&entry=gmail&source=g>
>>
>> On Mon, Jan 29, 2018 at 11:30 AM, Isaiah Nyabuto <isaiahnyabuto@xxxxxxxxx
>> > wrote:
>>
>>> Hi all,
>>>
>>> Thanks Timothy for sharing this. I am trying to import organisation
>>> units metadata in csv file with UTF-8 encoding. When I import the file with
>>> Metadata import in dhis2, the file is not imported at all and I do get the
>>> summary report.
>>>
>>> I wrote the csv file with R function write.table.
>>>
>>> Its funny enough that when I write the same file using Excel and save as
>>> csv file with UTF-8, it works well.
>>>
>>> I do not understand what could be the problem here, has anyone tried to
>>> write a csv file with an R function that can successfully be imported in
>>> the dhis2?
>>>
>>> Looking for your help. I have attached my csv files in this email.
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>> ==========================
>>> Isaiah Nyabuto
>>> +254729572616 <+254%20729%20572616>
>>> Bsc. Health Records Management and Informatics
>>>
>>> On Jan 17, 2018 17:51, "Thomas Lavreys" <thomaslavreys@xxxxxxxxx> wrote:
>>>
>>>> Thanks for the in depth info! It will surely help along the way. I’ve
>>>> only started with some testers, but I will probably run into some more
>>>> issues when I load my real data.
>>>>
>>>>
>>>>
>>>> Thanks again!!
>>>>
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>>
>>>>
>>>> Thomas
>>>>
>>>>
>>>>
>>>> *From:* Timothy Harding [mailto:tharding@xxxxxxxxxxxxxx]
>>>> *Sent:* Wednesday, January 17, 2018 3:35 PM
>>>> *To:* Thomas Lavreys <thomaslavreys@xxxxxxxxx>
>>>> *Cc:* Adebusoye Anifalaje <busoye@xxxxxxxxxxxxxx>; dhis2-users <
>>>> dhis2-users@xxxxxxxxxxxxxxxxxxx>
>>>> *Subject:* Re: [Dhis2-users] Data element not found or not accessible
>>>>
>>>>
>>>>
>>>> Hello Thomas!
>>>>
>>>>
>>>>
>>>> I am glad you have solved your issue! I noticed a few gotchas so here
>>>> are a couple of things to note when importing csv files in DHIS 2:
>>>>
>>>>
>>>>
>>>> 1. The first row
>>>>
>>>>
>>>>
>>>> dataelement Data element not found or not accessible
>>>>
>>>>
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>>
>>>> is a* funny error*, because it is actually telling you something about
>>>> a single row, found here in the documentation
>>>> <https://docs.dhis2.org/master/en/developer/html/webapi_data_values.html#webapi_data_values_csv>
>>>> :
>>>>
>>>> "dataelement","period","orgunit","catoptcombo","attroptcombo","value","storedby","lastupd","comment"
>>>>
>>>> DHIS 2 is literally complaining about the fact that it *cannot find a
>>>> data element with the uid "dataelement"*, which is what is assumes you
>>>> are trying to load in that top row. You can try yourself by loading a file
>>>> with only this top line, the error will be the same, 1 ignored row, data
>>>> element not found or not accessible. It is safe to ignore this error, I'd
>>>> be worried if there was a UID called dataelement, which is actually a valid
>>>> UID! Just extremely unlikely to pop up from the random UID generator.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2. Ignored
>>>>
>>>>
>>>>
>>>> Unlike the metadata importer, the data importer will report that it
>>>> ignored *every row on a dry run*. It can make testing difficult
>>>> because you don't know if it will actually import correctly or not as it
>>>> reports back that every row was ignored. Two rules of thumb here:
>>>>
>>>>    1. The actual errored out row will be listed in the "Conflicts"
>>>>    section, even in a dry run
>>>>    2. Sadly, some errors you can only find on a real run, so always
>>>>    upload your files with a comment, so you can find them later via the DB to
>>>>    delete them if something goes wrong and all else fails
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 3. File Type
>>>>
>>>>
>>>>
>>>> UTF-8 works well and I use it for all my CSV uploads. Be wary of
>>>> EXCEL's CSV saving habits though, especially on a Mac. Your file should be *UTF-8
>>>> without a BOM* (a byte order marker, DHIS 2 doesn't _really_ care, as
>>>> it will error out on row one anyway (see above)) AND it should be standard *unix
>>>> line feeds (LF)*. Excel for the Mac will default to saving it with
>>>> circa 1990's CRs only AND a BOM (unless you are editing a CSV, in which
>>>> case it will keep the settings it found when it opened it). Easiest way to
>>>> check is to grab a copy of Notepad++ (windows) or BBedit (mac).
>>>>
>>>>
>>>>
>>>> BBedit:
>>>>
>>>> [image: Inline image 2]
>>>>
>>>>
>>>>
>>>> and
>>>>
>>>>
>>>>
>>>> [image: Inline image 3]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Notepad++
>>>>
>>>> [image: Inline image 4]
>>>>
>>>>
>>>>
>>>> and
>>>>
>>>>
>>>>
>>>> [image: Inline image 5]
>>>>
>>>>
>>>>
>>>> Hope this helps!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Timothy Harding*
>>>> Sr. Systems Analyst, BAO Systems
>>>>
>>>> +1 202-536-1541 | tharding@xxxxxxxxxxxxxx | http://www.baosystems.com |
>>>>  Skype: hardingt@xxxxxxxxx | 2900 K Street, Suite 406, Washington D.C.
>>>> 20007
>>>> <https://maps.google.com/?q=2900+K+Street,+Suite+406,+Washington+D.C.+20007&entry=gmail&source=g>
>>>>
>>>>
>>>>
>>>> On Wed, Jan 17, 2018 at 9:11 AM, Thomas Lavreys <
>>>> thomaslavreys@xxxxxxxxx> wrote:
>>>>
>>>> I found the issue. The file was saved as a csv utf-8, which is
>>>> different to a normal csv (aparrently).
>>>>
>>>>
>>>>
>>>> Thanks for the advice!
>>>>
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>>
>>>>
>>>> Thomas
>>>>
>>>>
>>>>
>>>> *From:* Adebusoye Anifalaje [mailto:busoye@xxxxxxxxxxxxxx]
>>>> *Sent:* Wednesday, January 17, 2018 12:31 PM
>>>>
>>>>
>>>> *To:* Thomas Lavreys <thomaslavreys@xxxxxxxxx>
>>>> *Cc:* dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> *Subject:* Re: [Dhis2-users] Data element not found or not accessible
>>>>
>>>>
>>>>
>>>> Hi Thomas,
>>>>
>>>>
>>>>
>>>> I aslo see that you are using version 2.23, it is possible that the csv
>>>> column structure is different for this version.  There might be someone on
>>>> this forum who is currently working on this version and can provide input
>>>> on that.
>>>>
>>>> - -
>>>>
>>>>
>>>>
>>>>
>>>> *Busoye Anifalaje (PhD)*
>>>> Director of Services (Principal), BAO Systems
>>>>
>>>> UK: +44 7901-740-757 <+44%207901%20740757> | US: +1 682-307-0986
>>>> <(682)%20307-0986>|
>>>>
>>>>  busoye@xxxxxxxxxxxxxx | http://www.baosystems.com |
>>>>
>>>> Skype: busoye | 2900 K Street, Suite 406, Washington D.C. 20007
>>>> <https://maps.google.com/?q=2900+K+Street,+Suite+406,+Washington+D.C.+20007&entry=gmail&source=g>
>>>>
>>>>
>>>>
>>>> On 17 Jan 2018, at 12:23, Thomas Lavreys <thomaslavreys@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> I tried it anyway, and it doesn’t recognize it:
>>>>
>>>>
>>>>
>>>> Import summary
>>>>
>>>> Import count
>>>>
>>>> *Type*
>>>>
>>>> *Count*
>>>>
>>>> Imported
>>>>
>>>> 0
>>>>
>>>> Updated
>>>>
>>>> 0
>>>>
>>>> Deleted
>>>>
>>>> 0
>>>>
>>>> Ignored
>>>>
>>>> 13
>>>>
>>>> Conflicts
>>>>
>>>> *Element*
>>>>
>>>> *Description*
>>>>
>>>> dataelement
>>>>
>>>> Data element not found or not accessible
>>>>
>>>> bjDvmb4bfuf
>>>>
>>>> Category option combo not found or not accessible
>>>>
>>>>
>>>>
>>>> So I’m guessing I need to find the catoptioncombo ID somewhere.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Adebusoye Anifalaje [mailto:busoye@xxxxxxxxxxxxxx
>>>> <busoye@xxxxxxxxxxxxxx>]
>>>> *Sent:* Wednesday, January 17, 2018 12:11 PM
>>>> *To:* Thomas Lavreys <thomaslavreys@xxxxxxxxx>
>>>> *Cc:* dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> *Subject:* Re: [Dhis2-users] Data element not found or not accessible
>>>>
>>>>
>>>>
>>>> Hi Thomas,
>>>>
>>>>
>>>>
>>>> Can you try specifying the catoptioncombo even if it is default? This
>>>> was an issue with one of the version builds a while ago.
>>>>
>>>>
>>>>
>>>> Let’s know if that helps.
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>>
>>>> - -
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *<image001.png>*
>>>>
>>>> *Busoye Anifalaje (PhD)*
>>>> Director of Services (Principal), BAO Systems
>>>>
>>>> UK: +44 7901-740-757 <+44%207901%20740757> | US: +1 682-307-0986
>>>> <(682)%20307-0986>|
>>>>
>>>>  busoye@xxxxxxxxxxxxxx | http://www.baosystems.com |
>>>>
>>>> Skype: busoye | 2900 K Street, Suite 406, Washington D.C. 20007
>>>> <https://maps.google.com/?q=2900+K+Street,+Suite+406,+Washington+D.C.+20007&entry=gmail&source=g>
>>>>
>>>>
>>>>
>>>> On 17 Jan 2018, at 12:02, Thomas Lavreys <thomaslavreys@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Dear,
>>>>
>>>>
>>>>
>>>> I get the message “Data element not found or not accessible” when
>>>> trying to import a csv file. I looked at some old threads, but failed to
>>>> find an answer.
>>>>
>>>> The csv file is attached (with headers, but it gave the same answer
>>>> without headers).
>>>>
>>>> Attached also the API page of the data element, the sharing of the data
>>>> element (I have superuser access), and the orgunit page to confirm the ID.
>>>> Also a screenshot of the import summary and my DHIS2 instance info.
>>>>
>>>>
>>>>
>>>> I guess I might be missing something but I can’t figure out what
>>>> exactly.
>>>>
>>>>
>>>>
>>>> Thank you in advance for your help.
>>>>
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>>
>>>>
>>>> Thomas Lavreys
>>>>
>>>>
>>>>
>>>> *Thomas Lavreys, MSc, PgC Trop Med*
>>>>
>>>> *Sustainability Department SOCFINCO*
>>>>
>>>> *Maisha Consulting bvba*
>>>>
>>>> *Bisschoppenhof 9*
>>>>
>>>> *3150 Tildonk*
>>>>
>>>> *Belgium*
>>>>
>>>>
>>>>
>>>> *Tel Socfin: +32-25-080050*
>>>>
>>>> *Tel Maisha: +32-16-850242 <+32%2016%2085%2002%2042>*
>>>>
>>>> *Cell phone: +32-474-717641 <+32%20474%2071%2076%2041>*
>>>>
>>>> *Fax: +32-16-850242 <+32%2016%2085%2002%2042>*
>>>>
>>>>
>>>>
>>>> <TEST.csv><data element id.JPG><sharing data element.JPG><orgunit
>>>> id.JPG><import csv.JPG><instance info.JPG>_____________________
>>>> __________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+46764147049 <+46%2076%20414%2070%2049>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>