dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Lars,

Yes, I think this makes sense. Having a mechanism to sort through the fatal
versus non-fatal errors would be good, but in this case, it seems to be
pretty difficult, as you are only getting an SQL exception. There are lots
of these and errors, but only this one proved to be fatal.

Although again a little off-topic to the original post, the reason why this
happened is still not 100% clear to me, but seems to be have been
introduced with a script which I wrote to convert datasets to data element
groups (so that they could be used in MyDatamart). I have previously posted
some R code which will generate DHIS2 style UIDs, and I suspect that this
code introduced the duplicate UIDs, which proved to be OK in 2.10 and 2.11,
but not in 2.12. Of course, if I had checked this before creating the data
element groups (which I didn't) this would have never happened. But I find
it strange that there was not a unique constraint on this key in 2.10.
Again, maybe a result of  hack perhaps on my part, or just an oversight in
the schema. So another lesson if you are gonna hack DHIS2, be careful about
relying too heavily on the constraints in the schema!

Regards,
Jason



On Tue, Sep 24, 2013 at 4:52 PM, Lars Helge Øverland <larshelge@xxxxxxxxx>wrote:

> Hi Jason,
>
> sorry for late reply. Like you figured out, changing the log level is the
> best approach here. We emit a debug-level log message when this fails -
> reason it simply that it is hard to distinguish between failures caused by
> the constraint to already be in place (which is normal after the first run)
> and failures caused by other things like non-uniqueness in a db-standard
> way (not impossible, but requires some work). So we kept the debug level
> since we don't want to give the impression that something is wrong if the
> constraint fails when it is already present.
>
> Lars
>
>
>
>
>
>
> On Tue, Sep 24, 2013 at 4:41 PM, Jason Pickering <
> jason.p.pickering@xxxxxxxxx> wrote:
>
>> As it turns out, there were somehow (which I have no clue how this
>> happened) duplicated UIDs in the dataelementgroup table.
>>
>> This caused at error only visible when changing the logging level to
>> debug, which gave a clue.
>>
>>
>> * DEBUG 2013-09-24 14:54:18,868 Testing a Connection in response to an
>> Exception: (DefaultConnectionTester.java [main])
>> org.postgresql.util.PSQLException: ERROR: could not create unique index
>> "dataelementgroup_uid_key"
>>   Detail: Key (uid)=(KSbQUJxvNPc) is duplicated.
>>
>> Turns out there were actually three sets of duplicates, which seems
>> exceedingly unlikely to be caused by a key clash happening. One I might be
>> able to believe, but not three in the same table.
>>
>> Anyway, not exactly sure what might have caused this, but throwing that
>> error would have been useful. However lesson learned here is that if you
>> are getting these errors when upgrading, up the logging level to DEBUG and
>> wade through about 20,000 lines of crytpic messages, and you might get
>> lucky.
>>
>> Regards,
>> Jason
>>
>>
>>
>> On Tue, Sep 24, 2013 at 3:13 PM, Jason Pickering <
>> jason.p.pickering@xxxxxxxxx> wrote:
>>
>>> Hi Gerald,
>>>
>>> Although this is off the topic of this post, you can change the password
>>> of any user in a Postgresql database  by executing
>>>
>>> sudo -u postgres psql  -c "ALTER USER <user> WITH password '<password>';"
>>>
>>> in case you have forgotten it. Just replace <user> and <password> with
>>> their appropriate values. You  will obviously need sudo privileges with
>>> your current user.
>>>
>>> However, if DHIS2 starts with the password in the hibernate.properties
>>> file, it sounds like it may be something else. Maybe you can start a new
>>> thread on the list and we can help you with this.
>>>
>>>  Regards,
>>> Jason
>>>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 2:49 PM, Gerald Thomas <gerald.81@xxxxxxxxxxx>wrote:
>>>
>>>> Gomez,
>>>> This had been a question I posted before but I didn't get response. Now
>>>> when I am trying to backup my database using that command; it is asking me
>>>> for a password for the dhis account. The dhis password which I am expecting
>>>> to be on the hibernate file is not working. Any help with that???
>>>> Sent from my BlackBerry® smartphone from Airtel Sierra Leone
>>>>
>>>> -----Original Message-----
>>>> From: Gomez Phiri <gomezphiri@xxxxxxxxxxx>
>>>> Date: Tue, 24 Sep 2013 11:51:57
>>>> To: <jason.p.pickering@xxxxxxxxx>; <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>>> Subject: Re: [Dhis2-devs] Upgrade from 2.10 to 2.12 failing
>>>>
>>>>
>>>> Hi Jason,
>>>>
>>>>
>>>> I just upgraded our demo server from 2.10 to 2.12 today. This is what I
>>>> did to avoid all the troubles.
>>>>
>>>>
>>>> Steps
>>>>
>>>>
>>>>
>>>> 1. service tomcat stop
>>>> 2. back up database "pg_dump dhis2 -U dhis | gzip > db.gz" Our database
>>>> is large so I had to compress it with gzip.
>>>> 3. restart postgres
>>>> 4. su postgres
>>>> 5. psql
>>>> 6. DROP DATABASE db;
>>>> 7. CREATE DATABASE "db" WITH OWNER "owner" ENCODING 'UTF8';
>>>> 8. \q
>>>> 9. exit
>>>> 10. Restore database "gunzip -c db.gz | psql db"
>>>> 11. service tomcat start
>>>>
>>>>
>>>>
>>>> Its working superb  Emoji  <https://a.gfx.ms/Emoji_1F60A.png>
>>>>
>>>>
>>>> Gomez.
>>>>
>>>>
>>>> ----------------
>>>> Date: Tue, 24 Sep 2013 12:26:56 +0200
>>>> From: jason.p.pickering@xxxxxxxxx
>>>> To: dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Subject: [Dhis2-devs] Upgrade from 2.10 to 2.12 failing
>>>>
>>>>
>>>>
>>>> Using the latest 2.12 build from Jenkins.
>>>>
>>>>
>>>> Looks like something is failing on the startup, but not sure what?
>>>>
>>>>
>>>> This DB runs fine on 2.10. The first time around, a bunch of timestamps
>>>> were updated. I killed the server, restarted, but still getting stuck at
>>>> step 4 /10.
>>>>
>>>>
>>>> Any tips?
>>>>
>>>> Regards,
>>>> Jason
>>>>
>>>>
>>>>
>>>>
>>>> * INFO  2013-09-24 11:21:23,080 Found the following message senders:
>>>> [org.hisp.dhis.message.EmailMessageSender@17a220d7,
>>>> org.hisp.dhis.sms.outbound.SmsSender@151a0f60]
>>>> (DefaultMessageService.java [main])
>>>> * INFO  2013-09-24 11:21:39,093 Camel context started
>>>> (DHIS2LifecycleStrategy.java [main])
>>>> * INFO  2013-09-24 11:21:39,094 Searching for routes in directory:
>>>> /home/testinstance/dhis_home/routes (DHIS2LifecycleStrategy.java [main])
>>>> * INFO  2013-09-24 11:21:40,242 Executing startup routine [1 of 10,
>>>> runlevel 1]: TableAlteror (DefaultStartupRoutineExecutor.java [main])
>>>> * INFO  2013-09-24 11:21:46,749 Tables updated (TableAlteror.java
>>>> [main])
>>>> * INFO  2013-09-24 11:21:46,750 Executing startup routine [2 of 10,
>>>> runlevel 1]: MapViewUpgrader (DefaultStartupRoutineExecutor.java [main])
>>>> * INFO  2013-09-24 11:21:47,012 Executing startup routine [3 of 10,
>>>> runlevel 2]: IdentityPopulator (DefaultStartupRoutineExecutor.java [main])
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.core.StandardContext start
>>>> SEVERE: Error listenerStart
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.core.StandardContext start
>>>> SEVERE: Context [/testinstance] startup failed due to previous errors
>>>> * INFO  2013-09-24 11:21:48,017 De-registering jdbc driver:
>>>> org.h2.Driver@65b8b5cd (StartupListener.java [main])
>>>> * INFO  2013-09-24 11:21:48,018 De-registering jdbc driver:
>>>> org.hsqldb.jdbc.JDBCDriver@2830803a (StartupListener.java [main])
>>>> * INFO  2013-09-24 11:21:48,018 De-registering jdbc driver:
>>>> com.mysql.jdbc.Driver@1ccdf3c2 (StartupListener.java [main])
>>>> * INFO  2013-09-24 11:21:48,018 De-registering jdbc driver:
>>>> org.postgresql.Driver@2f1261b1 (StartupListener.java [main])
>>>> * INFO  2013-09-24 11:21:48,186 Camel context stopped
>>>> (DHIS2LifecycleStrategy.java [main])
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearReferencesThreads
>>>> SEVERE: A web application appears to have started a thread named
>>>> [Resource Destroyer in BasicResourcePool.close()] but has failed to stop
>>>> it. This is very likely to create a memory leak.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Transactional
>>>> resources]) and a value of type [null] (value [null]) but failed to remove
>>>> it when the web application was stopped. To prevent a memory leak, the
>>>> ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Transaction
>>>> synchronizations]) and a value of type [null] (value [null]) but failed to
>>>> remove it when the web application was stopped. To prevent a memory leak,
>>>> the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Prototype beans
>>>> currently in creation]) and a value of type [null] (value [null]) but
>>>> failed to remove it when the web application was stopped. To prevent a
>>>> memory leak, the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@9db900f]) and a
>>>> value of type
>>>> [org.springframework.security.core.context.SecurityContextImpl] (value
>>>> [org.springframework.security.core.context.SecurityContextImpl@ffffffff:
>>>> Null authentication]) but failed to remove it when the web application was
>>>> stopped. To prevent a memory leak, the ThreadLocal has been forcibly
>>>> removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Current transaction
>>>> name]) and a value of type [null] (value [null]) but failed to remove it
>>>> when the web application was stopped. To prevent a memory leak, the
>>>> ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Current aspect-driven
>>>> transaction]) and a value of type [null] (value [null]) but failed to
>>>> remove it when the web application was stopped. To prevent a memory leak,
>>>> the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Prototype beans
>>>> currently in creation]) and a value of type [null] (value [null]) but
>>>> failed to remove it when the web application was stopped. To prevent a
>>>> memory leak, the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a
>>>> value of type [null] (value [null]) but failed to remove it when the web
>>>> application was stopped. To prevent a memory leak, the ThreadLocal has been
>>>> forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Current transaction
>>>> read-only status]) and a value of type [null] (value [null]) but failed to
>>>> remove it when the web application was stopped. To prevent a memory leak,
>>>> the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Current transaction
>>>> isolation level]) and a value of type [null] (value [null]) but failed to
>>>> remove it when the web application was stopped. To prevent a memory leak,
>>>> the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedThreadLocal] (value [Actual transaction
>>>> active]) and a value of type [null] (value [null]) but failed to remove it
>>>> when the web application was stopped. To prevent a memory leak, the
>>>> ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type [null]
>>>> (value [com.sun.xml.bind.v2.ClassFactory$1@642bf5bf]) and a value of
>>>> type [java.util.WeakHashMap] (value [{class
>>>> org.apache.camel.spring.CamelProducerTemplateFactoryBean=java.lang.ref.WeakReference@32b7e05f,
>>>> class
>>>> org.apache.camel.model.PackageScanDefinition=java.lang.ref.WeakReference@45eb96fc,
>>>> class
>>>> javax.xml.bind.annotation.adapters.CollapsedStringAdapter=java.lang.ref.WeakReference@6e059e40,
>>>> class java.util.ArrayList=java.lang.ref.WeakReference@32d8b42e, class
>>>> org.apache.camel.spring.CamelContextFactoryBean=java.lang.ref.WeakReference@311d5216}])
>>>> but failed to remove it when the web application was stopped. To prevent a
>>>> memory leak, the ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedInheritableThreadLocal] (value [Locale
>>>> context]) and a value of type [null] (value [null]) but failed to remove it
>>>> when the web application was stopped. To prevent a memory leak, the
>>>> ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:48 AM org.apache.catalina.loader.WebappClassLoader
>>>> clearThreadLocalMap
>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>> [org.springframework.core.NamedInheritableThreadLocal] (value [Locale
>>>> context]) and a value of type [null] (value [null]) but failed to remove it
>>>> when the web application was stopped. To prevent a memory leak, the
>>>> ThreadLocal has been forcibly removed.
>>>> Sep 24, 2013 11:21:49 AM org.apache.coyote.http11.Http11AprProtocol
>>>> start
>>>> INFO: Starting Coyote HTTP/1.1 on http-8082
>>>> Sep 24, 2013 11:21:49 AM org.apache.catalina.startup.Catalina start
>>>>  INFO: Server startup in 77181 ms
>>>>
>>>>
>>>> _______________________________________________ Mailing list:
>>>> https://launchpad.net/~dhis2-devs Post to :
>>>> dhis2-devs@xxxxxxxxxxxxxxxxxxx Unsubscribe :
>>>> https://launchpad.net/~dhis2-devs More help :
>>>> https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>