← Back to team overview

dhis2-devs team mailing list archive

Re: Server processor use 100%

 

Yes, Hannan that is similar to what I have seen a number of times this
year.  The attacker makes use of atd and/or crontab to execute malicious
code.  The good thing is that your tomcat was not running as root which
would be potentially more damaging.

Obviously with access to the tomcat user then access to the database itself
has been exposed.  There is no indication that the database was the target
of previous exploits so probably (hopefully) that is your case too.  It is
a really good illustration though of why, when you have multiple instances
attaching to a database server, you should always use a separate database
role/user for each.  So when one database is exposed (through access to
dhis.conf), at least they are not all exposed.

Enjoy your holiday.  I am hoping to get off as well soon :-)

Regards
Bob

On 13 July 2017 at 16:01, Hannan Khan <hannank@xxxxxxxxx> wrote:

> Dear Bob
>
> Sorry for replaying late. I quite busy to complete few incomplete tasks
> before I am going on holiday tomorrow for a week.
>
> I have checked for few day with various options and my conclusion is that
> the security hole might be created by our old war file (version 16) with
> Stuart vulnerability which Lars warn all of us earlier. We upgraded all our
> servers and applications except this server. No suspicious files in the tmp
> folders.
>
> It took control of Tomcat8 user and run SSHD and occupies 100% of 2
> processors. When we kill the process and remove all war files and stop
> tomcat8 service it stared ATD command and it also occupy 100% of 2
> processors. The data seems intact (through query and size). As our all DB
> servers have similar IP structure we immediately remove tomcat8 service,
> package and user. The VM server will also be decommissioned and will setup
> a new server with new cardinals. I will start upgrade work after I return.
>
> Thank you for your valuable advice and kind concern.
>
> Best regards
>
> Hannan
>
> On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>
> wrote:
>
>> Sorry that should have been 'ls -la /tmp'
>>
>> On 10 July 2017 at 10:50, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>>
>>> Hi Hannan
>>>
>>> There is no circumstance that tomcat user should be running the sshd
>>> command.  It could be this machine has been compromised.  Unless you have
>>> some strange setup that you are logging in as tomcat user.
>>>
>>> Please contact me directly if you want me to check.
>>>
>>> Meanwhile you might want to have a look in /tmp directory and tomcat8
>>> home directory to see if there are any strange files there:
>>>
>>> ls -ls /tmp
>>>
>>> You might find that there is a rogue sshd program that has been
>>> installed there.  Note that if you are running a very old war file your
>>> risk of compromise is very high.
>>>
>>> Bob
>>>
>>> On 10 July 2017 at 05:09, Hannan Khan <hannank@xxxxxxxxx> wrote:
>>>
>>>> Dear Experts
>>>>
>>>> I have an wired situation. one of our DHIS2 server running older war
>>>> files (version 16), the OS was outdated and we have to upgrade the OS.
>>>> After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and
>>>> Tomcat 7 was installed by after running war file (version 16) after few
>>>> minutes the tomcat7 is not operational as the processor use is 100%. there
>>>> is only 1 user logged in and the application server using 2 processor and
>>>> DB server is separate.
>>>>
>>>> After trying several times I remove tomcat7 and install tomcat 8 with
>>>> same war file, but situation is same. I called it wired as the db size is
>>>> quite small, user is only few and the listing showing SSHD command by
>>>> tomcat8 user is using 100% processor.
>>>>
>>>> Any idea about the under line reason? need urgent help. Thank you all
>>>> in advance.
>>>>
>>>> Regards
>>>>
>>>> Muhammad Abdul Hannan Khan
>>>> Team Leader
>>>> Support to the National HMIS
>>>> MIS, Director General of Health Service
>>>> Ministry of Health and Family Welfare
>>>>
>>>> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
>>>> F +88 02 58813 875
>>>> M+88 01819 239 241
>>>> M+88 01534 312 066
>>>> E hannank@xxxxxxxxx
>>>> S hannan.khan.dhaka
>>>> B hannan-tech.blogspot.com
>>>> L https://bd.linkedin.com/in/hannankhan
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Muhammad Abdul Hannan Khan
> Team Leader
> Support to the National HMIS
> MIS, Director General of Health Service
> Ministry of Health and Family Welfare
>
> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
> F +88 02 58813 875
> M+88 01819 239 241
> M+88 01534 312 066
> E hannank@xxxxxxxxx
> S hannan.khan.dhaka
> B hannan-tech.blogspot.com
> L https://bd.linkedin.com/in/hannankhan
>
>
>
>

Follow ups

References