dhis2-devs team mailing list archive

[Halvdan Hoem Grelland <halvdan@xxxxxxxxx>]

To update everyone: this is definitely a bug with this stuff when using
FileResource with Tracker (2.23 -> ). I'm currently working on a fix, but
it will take some time.

It hopefully won't take long, but as it's a pretty critical issue to have
in production you will need to work around it in the mean time. The least
intrusive way is to run a cronjob at just before the cleanup task triggers
(2:00 server time) and update all fileresource rows in the DB with
isassigned=TRUE. Alternately a patch could be applied (comments out a
single line to disable the deletion/cleanup task). Let me know if you need
any more details.


On Mon, Jul 25, 2016 at 2:06 PM, Halvdan Hoem Grelland <halvdan@xxxxxxxxx>
wrote:

> Ok.
>
> That task *should* detect *unreferenced* fileResource objects and delete
> them. If they are referenced, however (i.e. there is a data value
> referencing their UID) they should not be deleted. If the uid in your
> example is in fact stored in a data value with a FR data element there is a
> bug, if not it works as intended (to prevent orphaned files).
>
> Could you check if the 'assigned' field of the FR is true after creating a
> new event with a FR data elements? If not, there is a problem in the server
> side event creation logic.
>
> There is no way to disable the task (also, you really shouldn't).
>
> On Mon, Jul 25, 2016 at 1:37 PM, Mike Nelushi <mikeevolution@xxxxxxxxx>
> wrote:
>
>> Hi Halvdan,
>>
>> We want the user to upload the file and be able to retrieve it but the
>> files are deleted(purged) at 2AM everyday.
>> * INFO  2016-07-25 02:00:00,072 'system-process' delete
>> org.hisp.dhis.fileresource.FileResource, name:
>> PERSONAL_CONSIDERATION_SUBMISSION. Samuel Mur 7910025121081.pdf, uid:
>> ko0TqodGcPh (AuditLogUtil.java [taskScheduler-4])
>> * INFO  2016-07-25 02:00:00,082 'system-process' delete
>> org.hisp.dhis.fileresource.FileResource, name: Craig Anders Personal
>> Consideration.pdf, uid: ShiqhAsXc78 (AuditLogUtil.java [taskScheduler-4])
>> * INFO  2016-07-25 02:00:00,096 'system-process' delete
>> org.hisp.dhis.fileresource.FileResource, name: Motivation for Comm -
>> Copy.pdf, uid: L1fLHOSqwkM (AuditLogUtil.java [taskScheduler-4])
>> * INFO  2016-07-25 02:00:00,102 'system-process' delete
>> org.hisp.dhis.fileresource.FileResource, name: SATA - TOUCH WORLD CUP 2015
>> - SA MIXED TEAM CONFIRMATION.pdf, uid: EWQXKQsCunv (AuditLogUtil.java
>> [taskScheduler-4])
>> * INFO  2016-07-25 02:00:00,108 'system-process' delete
>> org.hisp.dhis.fileresource.FileResource, name: N_Isaacs_WPTA_July2016.pdf,
>> uid: AmoMiCsoSvD (AuditLogUtil.java [taskScheduler-4])
>> *.....*
>>
>> How do we disable this task from running?
>>
>> Regards,
>> Mike
>>
>> On Mon, Jul 25, 2016 at 1:27 PM, Halvdan Hoem Grelland <halvdan@xxxxxxxxx
>> > wrote:
>>
>>> This should be fine. Are you experiencing that it doesn't work as
>>> expected? (I didn't consider the tracker use case when I wrote 'dataset'.
>>> My bad...)
>>>
>>> Calle: your assumptions are correct. DV to FR is one-to-one. You would
>>> need a DE per specific file for your event/dataset, or alternately package
>>> multiple files in some archive format and store that instead (you could
>>> handle this on the client side).
>>>
>>> What if you want multiple data value records to reference the same
>>>> document [...]
>>>>
>>>
>>> A FR data value is just a data value with a weak reference to a file.
>>> Grouping data elements is no different than for straight numeric or text
>>> values. That is, there is no 'link' between the fields, but you could use a
>>> construct such as a data set (aggr) or programstageinstance/event (tracker)
>>> to group DEs.
>>>
>>> On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi <mikeevolution@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Halvdan,
>>>>
>>>> The following are the steps I'm using:
>>>> 1. Save the file on the api/fileResource endpoint, get the UID back,
>>>> then save the uid as of a dataValue on the event. Example:
>>>>
>>>>     POST to api/fileResources, get id: “a11111bbbb” back.
>>>> 2. PUT the event:
>>>>
>>>> {
>>>>   "event" : "<some-event-uid>",
>>>>   "dataValues" : [
>>>>     {
>>>>       "value" : "a11111bbbb",
>>>>       "dataElement" : "<some-fileResource-dataelement-uid>"
>>>>     }
>>>>   ]
>>>> }
>>>>
>>>> Are these steps correct?
>>>> Are the datasets required in these case? If yes, how should I go about
>>>> doing this in the tracker module.
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland <
>>>> halvdan@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> As is explained in the docs, the FR must be referenced by a data
>>>>> value. If the FR is referenced it will not be deleted.
>>>>>
>>>>> The FR object is not usable by itself, so the upload to the
>>>>> api/*/fileResources is step 1 of 2 in the process of uploading a file (it
>>>>> needs to be tied to a DV afterwards).
>>>>>
>>>>> In practice this means that in order to use FR you need to create data
>>>>> elements (and datasets) to contain your files.
>>>>>
>>>>> Hope that answers your question. If not, please let me know.
>>>>>
>>>>> On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi <mikeevolution@xxxxxxxxx
>>>>> > wrote:
>>>>>
>>>>>> Hi Halvdan/Morten,
>>>>>>
>>>>>> There seem to be a cleanup task which is ran at 2AM every 24H which
>>>>>> necessary maintenance which removes orphaned files (FileResources) which
>>>>>> have been posted to the instance but are marked for deletion as Halvdan
>>>>>> explained here
>>>>>> <http://dhis-developers-list.1563109.n2.nabble.com/Re-Dhis2-users-Nightly-Batch-Jobs-td7604891.html>.
>>>>>> And from File resource constraints explained *here
>>>>>> <http://dhis2.github.io/dhis2-docs/2.23/en/developer/html/ch01s12.html#d0e2396> "Any
>>>>>> file resources which are in this state and are older than two hours will be
>>>>>> marked for deletion and will eventually be purged from the system." *
>>>>>>
>>>>>> Our new application allows users to upload files and would like to
>>>>>> prevent this from happening.
>>>>>> How do we stop the system from running this file deletion task or tag
>>>>>> the files to not be deleted?
>>>>>>
>>>>>> Please advice.
>>>>>>
>>>>>> Regards,
>>>>>> Mike
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Halvdan Hoem Grelland
>>>>> Software developer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org <https://www.dhis2.org/>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Halvdan Hoem Grelland
>>> Software developer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org <https://www.dhis2.org/>
>>>
>>>
>>
>
>
> --
> Halvdan Hoem Grelland
> Software developer, DHIS 2
> University of Oslo
> http://www.dhis2.org <https://www.dhis2.org/>
>
>


-- 
Halvdan Hoem Grelland
Software developer, DHIS 2
University of Oslo
http://www.dhis2.org <https://www.dhis2.org/>