dhis2-users team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Hi Jason

Halvdan is right - these are more likely file descriptors associated with
sockets.

The limit you have dug up is the kernel limit (ie total no of "files" that
the kernel can have open in total) which is probably not the limit you are
reaching.

If you have a very busy server you can have a lot of sockets lingering in
what is called tcp's TIME_WAIT state (on tomcat the problem becomes visible
quicker than nginx because you also have postgres connections using tcp
sockets).

netstat -t | grep -c TIME_WAIT

will give you an idea.  Sockets in this state hang around for a minute or
so and add to the total number of connections.

>From which there are only two real solutions .. either reduce the system's
TIME_WAIT time or increase the number of sockets the process is allowed to
open. Reducing TIME_WAIT is not really recommended as you might end up with
lost packets when connections are closing.  To increase the max number of
open fds you need to increase the ulimit for the user running the tomcat
process.  There are different ways depending a bit on how your tomcat is
installed ... google tomcat ulimit will give some ideas.

On 7 September 2016 at 07:21, Halvdan Hoem Grelland <halvdan@xxxxxxxxx>
wrote:

> Hi Jason,
>
> ‘Files’ in this context is really a file descriptor, which is any
> connectable resource in the system, including both files and sockets.
>
> I do agree that it’s unlikely that Tomcat hogs 6.5 million FDs, but I’m
> not convinced that is the real limit either.
>
> You could try:
>
> ulimit -n
>
> To get the max for open file descriptors in the system.
>
> You can check max FDs for the Tomcat process by running:
>
> cat /proc/<pid-of-tomcat>/limits
>
> Which is usually around 1024.
>
> Also this
> <https://www.jayway.com/2012/02/11/how-to-really-fix-the-too-many-open-files-problem-for-tomcat-in-ubuntu/>
> might be useful.
>
> Lastly, if the limits are sane (which they probably are) and you are able
> to reproduce the issue you could try using this tool
> <http://file-leak-detector.kohsuke.org/> to pinpoint the problem.
> (Disclaimer: I have no experience with it).
> ​
>
> On Mon, Sep 5, 2016 at 11:22 PM, Jason Phillips <jason@xxxxxxxx> wrote:
>
>> Hi all,
>>
>>
>>
>> We have a system that tanked last night, reporting:
>>
>>
>>
>> 14-Aug-2016 19:13:57.440 SEVERE [http-nio-8080-Acceptor-0]
>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
>>
>> java.io.IOException: Too many open files
>>
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>
>> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChanne
>> lImpl.java:422)
>>
>> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChanne
>> lImpl.java:250)
>>
>> at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpo
>> int.java:687)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> Any ideas what the cause might be?  The environment is Ubuntu 14.04,
>> Tomcat 8, Java 8.
>>
>> A cat /proc/sys/fs/file-max returns 6556412 – and that system was
>> physically restarted within the last week.  I find it hard to believe that
>> the web server would have 6,5 million files open in such a short period of
>> time.
>>
>> War is 2.23
>>
>> OS is Ubuntu 14.04 LTS
>>
>> Tomcat 8
>>
>> Java 8
>>
>>
>>
>> Kind Regards,
>>
>> *Jason Phillips*
>>
>> [image: hisp]
>> *Information Systems / Infrastructure*
>>
>> *Health Information Systems Program____________________________________*
>>
>> eMail:               jason@xxxxxxxx
>> Tel/Fax:            +27 21 712 0170
>> Cell:                 +27 72 973 7250
>> Skype:             jason.n.phillips
>>
>> This message and any attachments are subject to a disclaimer published at
>> http://www.hisp.org/policies.html#comms_disclaimer.  Please read the
>> disclaimer before opening any attachment or taking any other action in
>> terms of this electronic transmission.  If you cannot access the
>> disclaimer, kindly send an email to disclaimer@xxxxxxxx and a copy will
>> be provided to you. By replying to this e-mail or opening any attachment
>> you agree to be bound by the provisions of the disclaimer.
>>
>>
>>
>> *This message and any attachments are subject to a disclaimer published
>> at http://www.hisp.org/policies.html#comms_disclaimer
>> <http://www.hisp.org/policies.html#comms_disclaimer>.  Please read the
>> disclaimer before opening any attachment or taking any other action in
>> terms of this electronic transmission.  If you cannot access the
>> disclaimer, kindly send an email to disclaimer@xxxxxxxx
>> <disclaimer@xxxxxxxx> and a copy will be provided to you. By replying to
>> this e-mail or opening any attachment you agree to be bound by the
>> provisions of the disclaimer.*
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Halvdan Hoem Grelland
> Software developer, DHIS 2
> University of Oslo
> http://www.dhis2.org <https://www.dhis2.org/>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>