monasca team mailing list archive

[<Yuan.Pen@xxxxxxxxxxxxx>]

Hi Jakub,
Can you show the heap memory you assigned to the supervisors and workers in storm.yam?  like:

worker.childopts: "-Xmx2048m -XX:PermSize=64M -XX:MaxPermSize=512M -Djava.net.preferIPv4Stack=false"

nimbus.childopts: "-Xmx512m -Djava.net.preferIPv4Stack=false"

supervisor.childopts: "-Xmx512m -Djava.net.preferIPv4Stack=false"

Yuan
571-594-6155

From: Monasca [mailto:monasca-bounces+yuan.pen=t-systems.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Tomasz.Trebski@xxxxxxxxxxxxxx
Sent: Thursday, April 6, 2017 4:42 AM
To: monasca@xxxxxxxxxxxxxxxxxxx
Cc: Hochmuth, Roland M
Subject: [Monasca] Fw: [monasca] Apache Storm / Monasca Thresh unable to create new native thread


Hi there,



sending on behalf of Jakub.



@Roland, could you distribute ''java.lang.OutOfMemoryError: unable to create new native thread'' down to HPE, maybe you've observed something like this.​



regards,

Tomasz & Jakub

________________________________
From: Wachowski, Jakub
Sent: 06 April 2017 10:24
To: Trębski, Tomasz
Subject: FW: [monasca] Apache Storm / Monasca Thresh unable to create new native thread

Tomasz,
could you please send this email on my behalf?

Thanks,
Jakub

From: Wachowski, Jakub
Sent: Thursday, April 6, 2017 10:12 AM
To: monasca@xxxxxxxxxxxxxxxxxxx
Subject: [monasca] Apache Storm / Monasca Thresh unable to create new native thread


Hello everyone,



Unfortunately I've run out of ideas, so I'm putting this here - maybe someone will have any idea...

In our SUSE Cloud 7 production-like environment (i.e. running on bare metal, pretty strong machines), we've observed, that Apache Storm/Monasca Thresh fails with the following error:
java.lang.OutOfMemoryError: unable to create new native thread


Funny fact is, that this problem does not seem to occur neither in our CentOS (virtualized) environment nor in SUSE virtualized (mkcloud) environment....

I've checked all the limits I was aware of (like max processes/file descriptors per user), I've also run a simple test, which showed that it is possible to create over 12,000 threads in Java, while Storm uses only around 150.

Restarting all the services and the machine itself didn't help.



Some details:

Java version:

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (suse-20.1-x86_64)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)



Apache-Storm v1.0.2

Monasca Thresh v2.1.0



Please also see the attached log file.

Any help is welcome



BR,

Jakub