dhis2-devs team mailing list archive

[Orvalho Augusto <orvaquim@xxxxxxxxx>]

Thank you every one!

I was granted permission to remove zeros and negative "energy" from the
dataset. And applied the Lars's medicine and Mr Datamart got back to our
lives again:

* INFO  2013-05-14 20:46:08,118 Completeness export task done
(AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-14])
* INFO  2013-05-14 20:46:08,708 Completeness export task done
(AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-15])
* INFO  2013-05-14 20:46:08,952 Completeness export task done
(AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-13])
* INFO  2013-05-14 20:46:11,194 Completeness export task done
(AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-16])
* INFO  2013-05-14 20:46:11,391 Created index: 0:00:11.515 (Clock.java
[taskScheduler-1])
* INFO  2013-05-14 20:46:11,392 Completeness export process completed:
0:00:11.516 (Clock.java [taskScheduler-1])
* INFO  2013-05-14 20:46:11,392 [Level: INFO, category: DATAMART, time: Tue
May 14 20:46:11 UTC 2013, message: Completeness process completed]
(InMemoryNotifier.java [taskScheduler-1])

Thank you!
Orvalho



On Tue, May 14, 2013 at 4:34 PM, Lars Helge Øverland <larshelge@xxxxxxxxx>wrote:

> Hi Orvalho,
>
> I am working on a fix, in the meantime you can provide the following
> medicine as SQL:
>
> delete from systemsetting where name='orgUnitGroupSetAggregationLevel';
>
> Then go to data admin -> cache and clear cache.
>
> regards,
>
> Lars
>
>
>
>
> On Tue, May 14, 2013 at 2:49 PM, Orvalho Augusto <orvaquim@xxxxxxxxx>wrote:
>
>> Lord forgive me for my sins.
>>
>> These criminal negatives come from a legacy system. They have been
>> migrated... I am in the process of migrating them actually:
>>   308527 records < 1
>> 2275573 records are zeros!
>> 1096806 records > 0
>>
>> I will report this to the owners of the data.
>>
>> Thank you
>> Orvalho
>>
>>
>>
>> On Tue, May 14, 2013 at 2:35 PM, <deemoyes@xxxxxxxxx> wrote:
>>
>>> **
>>> There is not suppose to be negative counting though. That is why it is
>>> very necessary to have validation rules in place.
>>> Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed
>>> mobile broadband on our easyblaze and plans for BlackBerry. Visit
>>> www.etisalat.com.ng for details.
>>> ------------------------------
>>> *From: * Orvalho Augusto <orvaquim@xxxxxxxxx>
>>> *Date: *Tue, 14 May 2013 12:49:39 +0200
>>> *To: *Ocaya Stephen<stephocay@xxxxxxxxx>
>>> *Cc: *Dayo Adeyomoye<deemoyes@xxxxxxxxx>;
>>> Dhis2-users<dhis2-users-bounces+deemoyes=yahoo.com@xxxxxxxxxxxxxxxxxxx>;
>>> dhis1-users<dhis2-users@xxxxxxxxxxxxxxxxxxx>; DHIS 2 developers<
>>> dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>> *Subject: *Re: [Dhis2-users] Troubles running datamart
>>>
>>> One thing:
>>>
>>> Couldn't negative numbers trouble datamart?
>>>
>>> I do not know how people here count negatively!
>>>
>>> Orvalho
>>>
>>>
>>>
>>> On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto <orvaquim@xxxxxxxxx>wrote:
>>>
>>>> Thank you!
>>>>
>>>> Yes those logs seemed to be clean.
>>>>
>>>> I can not run datamart now because it is a modest machine  and there is
>>>> a lot of users.
>>>>
>>>> I will try to set logs to DEBUG level.
>>>>
>>>> By the way tail, is a nice tool but when one expects huge logs like
>>>> that is better to have a new log file.. and use less, grep etc;  And my
>>>> connection to the server keeps failing a lot of times.
>>>>
>>>> Orvalho
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen <stephocay@xxxxxxxxx>wrote:
>>>>
>>>>> Did the manual generation produce any data/reports ??
>>>>>
>>>>> Looking at the Log file, I looks rather crowded and Setting Tomcat LOG
>>>>> LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that
>>>>> it skips out all the logs like Hibernate:...
>>>>>
>>>>> Could also try to rectify the indicators(I am not sure whether it also
>>>>> affects DATAMART generation). I did not see the cause of the error in the
>>>>> log.
>>>>>
>>>>> To View Log on the Fly, you could
>>>>>
>>>>> tail -f /var/logs/tomcat7/catalina.out (if the log file is in
>>>>> /var/logs/tomcat7 path ... and the log file is catalina.out
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto <orvaquim@xxxxxxxxx>wrote:
>>>>>
>>>>>> OK!
>>>>>>
>>>>>> The log I gave is after I start datamart the same way you describe. I
>>>>>> was not aware that it is manual way :-)
>>>>>>
>>>>>> The catalinaout.txt is attached... since the begining of the
>>>>>> datamart. How I wish I have zipped that file!
>>>>>>
>>>>>> I saw some invalid indicators there but those have been there before
>>>>>> these symptoms.
>>>>>>
>>>>>> Orvalho
>>>>>>
>>>>>>
>>>>>> On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen <stephocay@xxxxxxxxx>wrote:
>>>>>>
>>>>>>> To run DataMart Manually (U need to have the Right i guess),
>>>>>>> Analytics and Data Mart
>>>>>>>
>>>>>>>    - Analytics and Data Mart  (In version 2.11 Hopefully 2.10 also)
>>>>>>>
>>>>>>> But idealy it should be under Reports then on the left pane, select
>>>>>>> *Analytics and Data Mart*
>>>>>>>
>>>>>>> *There, You can select the periods, and other parameters ...*
>>>>>>>
>>>>>>> *The hit Start Export*
>>>>>>> On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto <
>>>>>>> orvaquim@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Thank you!
>>>>>>>>
>>>>>>>> The dataset report does not bring any data that I see on dataentry.
>>>>>>>> For other reports the fields are numeric.
>>>>>>>>
>>>>>>>> For the datamart. Pardon my ignorance. How do I run it mannualy?
>>>>>>>>
>>>>>>>> Orvalho
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen <stephocay@xxxxxxxxx
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> What are the properties of the Fields for the Data Elements used
>>>>>>>>> for reporting?
>>>>>>>>>
>>>>>>>>> Could you try manually generating the datamart and send us the
>>>>>>>>> complete LOG from where the error start upto where you can see Caused by:
>>>>>>>>> ....
>>>>>>>>>
>>>>>>>>> That kind of gives a clear picture of what is causing the problem
>>>>>>>>> Waiting ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto <
>>>>>>>>> orvaquim@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Dears!
>>>>>>>>>>
>>>>>>>>>> Thank you for the answers.
>>>>>>>>>>
>>>>>>>>>> I have no text on value field according to this
>>>>>>>>>>
>>>>>>>>>> dhis2=# select * from datavalue where value is NULL;
>>>>>>>>>>  dataelementid | periodid | sourceid | categoryoptioncomboid |
>>>>>>>>>> value | storedby | lastupdated | comment | followup
>>>>>>>>>>
>>>>>>>>>> ---------------+----------+----------+-----------------------+-------+----------+-------------+---------+----------
>>>>>>>>>> (0 rows)
>>>>>>>>>>
>>>>>>>>>> dhis2=# select * from datavalue where trim(value) = '';
>>>>>>>>>>  dataelementid | periodid | sourceid | categoryoptioncomboid |
>>>>>>>>>> value | storedby | lastupdated | comment | followup
>>>>>>>>>>
>>>>>>>>>> ---------------+----------+----------+-----------------------+-------+----------+-------------+---------+----------
>>>>>>>>>> (0 rows)
>>>>>>>>>>
>>>>>>>>>> dhis2=# select * from datavalue where trim(value) ~ '[A-Za-z]';
>>>>>>>>>>  dataelementid | periodid | sourceid | categoryoptioncomboid |
>>>>>>>>>> value | storedby | lastupdated | comment | followup
>>>>>>>>>>
>>>>>>>>>> ---------------+----------+----------+-----------------------+-------+----------+-------------+---------+----------
>>>>>>>>>> (0 rows)
>>>>>>>>>>
>>>>>>>>>> dhis2=# select * from datavalue where trim(value) !~ '[0-9]';
>>>>>>>>>>  dataelementid | periodid | sourceid | categoryoptioncomboid |
>>>>>>>>>> value | storedby | lastupdated | comment | followup
>>>>>>>>>>
>>>>>>>>>> ---------------+----------+----------+-----------------------+-------+----------+-------------+---------+----------
>>>>>>>>>> (0 rows)
>>>>>>>>>>
>>>>>>>>>> Orvalho
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 14, 2013 at 9:30 AM, <deemoyes@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> I think the value type for the data elements are not properly
>>>>>>>>>>> set because if you set the value to be α number and an integer and also
>>>>>>>>>>> specify to or not to allow zeros plus your validation rules then I don't
>>>>>>>>>>> think you would these issues.
>>>>>>>>>>>
>>>>>>>>>>> The values are probably set to be text or so.
>>>>>>>>>>> Sent from my BlackBerry® smartphone from Etisalat. Enjoy high
>>>>>>>>>>> speed mobile broadband on our easyblaze and plans for BlackBerry. Visit
>>>>>>>>>>> www.etisalat.com.ng for details.
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Ocaya Stephen <stephocay@xxxxxxxxx>
>>>>>>>>>>> Sender: "Dhis2-users"
>>>>>>>>>>>         <dhis2-users-bounces+deemoyes=
>>>>>>>>>>> yahoo.com@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>>>> Date: Tue, 14 May 2013 10:22:45
>>>>>>>>>>> To: Orvalho Augusto<orvaquim@xxxxxxxxx>
>>>>>>>>>>> Cc: dhis1-users<dhis2-users@xxxxxxxxxxxxxxxxxxx>; DHIS 2
>>>>>>>>>>> developers<dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>>>> Subject: Re: [Dhis2-users] Troubles running datamart
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>  *You can do anything, but not everything.*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *You can do anything, but not everything.*
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *You can do anything, but not everything.*
>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>