openjdk team mailing list archive

Thread
Date
[Bug 1951117] [NEW] jvm-lts crashing during tomcat session replication

To: openjdk@xxxxxxxxxxxxxxxxxxx
From: Marco Ferretti <1951117@xxxxxxxxxxxxxxxxxx>
Date: Tue, 16 Nov 2021 14:16:39 -0000
Reply-to: Bug 1951117 <1951117@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I have setup a test environment with two tomcat9 (tomee-plume-8.0.6) nodes running behind an apache 2.4 with mod_jk.
At the startup everything's fine, the application deploys and apache is happy about the workers.
I open a session to the application by visiting localhost/app; mod_jk directs to node1, the java session is opened and the application is accessed. As soon as the session replication fires from node1 to node2 the latter crashes.
Here's the log: 

Nov 16, 2021 2:43:24 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [localhost#/mui], requesting session state from [org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 1, 1}:4000,{127, 0, 1, 1},4000, alive=47530, securePort=-1, UDP Port=-1, id={-95 99 47 -24 97 65 64 -102 -74 110 -80 109 79 57 66 0 }, payload={}, command={}, domain={}]]. This operation will timeout if no session state has been received within [60] seconds.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2c7c1bac14, pid=49141, tid=49237
#
# JRE version: OpenJDK Runtime Environment (11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x564c14]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/marco/servers/tomee-cluster/node2/core.49141)
#
# An error report file with more information is saved as:
# /home/marco/servers/tomee-cluster/node2/hs_err_pid49141.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#

I have tried different versions of OpenJDK but the result is always the
same

Here are the details of the system

➜ lsb_release -rd
Description:	Ubuntu 20.04.3 LTS
Release:	20.04

2) The version of the package you are using, via 'apt-cache policy
pkgname' or by checking in Software Center

java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

apachectl -v
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2021-10-14T16:24:43

➜ ./catalina.sh version
Using CATALINA_BASE:   /home/marco/servers/tomee
Using CATALINA_HOME:   /home/marco/servers/tomee
Using CATALINA_TMPDIR: /home/marco/servers/tomee/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /home/marco/servers/tomee/bin/bootstrap.jar:/home/marco/servers/tomee/bin/tomcat-juli.jar
Using CATALINA_OPTS:   
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Server version: Apache Tomcat/9.0.41
Server built:   Dec 3 2020 11:43:00 UTC
Server number:  9.0.41.0
OS Name:        Linux
OS Version:     5.11.0-40-generic
Architecture:   amd64
JVM Version:    11.0.11+9-Ubuntu-0ubuntu2.20.04
JVM Vendor:     Ubuntu

node1 and node2 inherit the binaries from above. The configuration is
changed by prepending node number to the ports (eg. port 8080->18080 and
28080) and by adding the JVM route to the Engine entry.

workers.properties:
---
# workers.tomcat_home should point to the location where you
# installed tomcat. This is where you have your conf, webapps and lib
# directories.
#
workers.tomcat_home=/home/marco/servers/apache-tomee-plume-8.0.6
#
# workers.java_home should point to your Java installation. Normally
# you should have a bin and lib directories beneath it.
#
workers.java_home=/usr/lib/jvm/default-java

# Define 1 real worker using ajp13
worker.list=jkstatus, lb

worker.jkstatus.type=status

worker.lb.type=lb

# Set properties for worker (ajp13)
worker.node1.type=ajp13
worker.node1.host=localhost
worker.node1.port=18009
worker.node1.ping_mode=A
worker.node1.lbfactor=1

worker.node2.type=ajp13
worker.node2.host=localhost
worker.node2.port=28009
worker.node2.ping_mode=A
worker.node2.lbfactor=1

## associate workers with load balancer
worker.lb.balance_workers=node1,node2
worker.lb.sticky_session=1
---

3) What you expected to happen
the session being replicated on both nodes
4) What happened instead
the JVM crashed

** Affects: openjdk-lts (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of OpenJDK,
which is subscribed to openjdk-lts in Ubuntu.
https://bugs.launchpad.net/bugs/1951117

Title:
  jvm-lts crashing during tomcat session replication

Status in openjdk-lts package in Ubuntu:
  New

Bug description:
  I have setup a test environment with two tomcat9 (tomee-plume-8.0.6) nodes running behind an apache 2.4 with mod_jk.
  At the startup everything's fine, the application deploys and apache is happy about the workers.
  I open a session to the application by visiting localhost/app; mod_jk directs to node1, the java session is opened and the application is accessed. As soon as the session replication fires from node1 to node2 the latter crashes.
  Here's the log: 

  Nov 16, 2021 2:43:24 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
  INFO: Manager [localhost#/mui], requesting session state from [org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 1, 1}:4000,{127, 0, 1, 1},4000, alive=47530, securePort=-1, UDP Port=-1, id={-95 99 47 -24 97 65 64 -102 -74 110 -80 109 79 57 66 0 }, payload={}, command={}, domain={}]]. This operation will timeout if no session state has been received within [60] seconds.
  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGSEGV (0xb) at pc=0x00007f2c7c1bac14, pid=49141, tid=49237
  #
  # JRE version: OpenJDK Runtime Environment (11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
  # Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
  # Problematic frame:
  # V  [libjvm.so+0x564c14]
  #
  # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/marco/servers/tomee-cluster/node2/core.49141)
  #
  # An error report file with more information is saved as:
  # /home/marco/servers/tomee-cluster/node2/hs_err_pid49141.log
  #
  # If you would like to submit a bug report, please visit:
  #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
  #

  I have tried different versions of OpenJDK but the result is always
  the same

  Here are the details of the system

  ➜ lsb_release -rd
  Description:	Ubuntu 20.04.3 LTS
  Release:	20.04

  2) The version of the package you are using, via 'apt-cache policy
  pkgname' or by checking in Software Center

  java -version
  openjdk version "11.0.11" 2021-04-20
  OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
  OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

  apachectl -v
  Server version: Apache/2.4.41 (Ubuntu)
  Server built:   2021-10-14T16:24:43

  ➜ ./catalina.sh version
  Using CATALINA_BASE:   /home/marco/servers/tomee
  Using CATALINA_HOME:   /home/marco/servers/tomee
  Using CATALINA_TMPDIR: /home/marco/servers/tomee/temp
  Using JRE_HOME:        /usr
  Using CLASSPATH:       /home/marco/servers/tomee/bin/bootstrap.jar:/home/marco/servers/tomee/bin/tomcat-juli.jar
  Using CATALINA_OPTS:   
  NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
  Server version: Apache Tomcat/9.0.41
  Server built:   Dec 3 2020 11:43:00 UTC
  Server number:  9.0.41.0
  OS Name:        Linux
  OS Version:     5.11.0-40-generic
  Architecture:   amd64
  JVM Version:    11.0.11+9-Ubuntu-0ubuntu2.20.04
  JVM Vendor:     Ubuntu

  node1 and node2 inherit the binaries from above. The configuration is
  changed by prepending node number to the ports (eg. port 8080->18080
  and 28080) and by adding the JVM route to the Engine entry.

  workers.properties:
  ---
  # workers.tomcat_home should point to the location where you
  # installed tomcat. This is where you have your conf, webapps and lib
  # directories.
  #
  workers.tomcat_home=/home/marco/servers/apache-tomee-plume-8.0.6
  #
  # workers.java_home should point to your Java installation. Normally
  # you should have a bin and lib directories beneath it.
  #
  workers.java_home=/usr/lib/jvm/default-java

  # Define 1 real worker using ajp13
  worker.list=jkstatus, lb

  worker.jkstatus.type=status

  worker.lb.type=lb

  # Set properties for worker (ajp13)
  worker.node1.type=ajp13
  worker.node1.host=localhost
  worker.node1.port=18009
  worker.node1.ping_mode=A
  worker.node1.lbfactor=1

  worker.node2.type=ajp13
  worker.node2.host=localhost
  worker.node2.port=28009
  worker.node2.ping_mode=A
  worker.node2.lbfactor=1

  ## associate workers with load balancer
  worker.lb.balance_workers=node1,node2
  worker.lb.sticky_session=1
  ---

  3) What you expected to happen
  the session being replicated on both nodes
  4) What happened instead
  the JVM crashed

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openjdk-lts/+bug/1951117/+subscriptions
Follow ups

[Bug 1951117] Re: jvm-lts crashing during tomcat session replication
From: Launchpad Bug Tracker, 2023-10-06