openjdk team mailing list archive

[tony mancill <tmancill@xxxxxxxxxx>]

Hello Erich,

On Sat, Aug 17, 2019 at 08:41:23PM +0200, Erich Schubert wrote:
> Subject: openjdk-11-jre: Performance of JDK11/12 is 10%-15% worse than JDK8
> Package: openjdk-11-jre
> Version: 11.0.4+11-1
> Severity: normal
> Tags: upstream
> 
> In my benchmarks, JDK 11 and 12 scores around 10%-20% worse runtime
> than JDK 8.
> 
> I also tried Oracle Graal CE and Graal EE, and they perform even
> worse. I also tried AdoptOpenJDK packages, and these perform similar
> to the Debian versions.
> 
> The difference is substantial; this is not a toy microbenchmark that
> only tests a single tiny code fragment. Instead, I run ELKI to
> benchmark DBSCAN clustering on a dataset with over 100k points; a
> typical workload that I am interested in.  This code is well
> optimized, and requires very little garbage collection (5 young gen
> runs each, about 80 ms total GC time). Memory limits are high enough
> to let this run smoothly.  I have measured both in-Java time of two
> steps, and wall-clock time of the entire process until completion.
> 
> Loading 110250 vectors from a .csv.gz file with
> 
> Java   8: 1837 ms
> Java  11: 1955 ms
> Java  12: 1946 ms
> Graal EE: 4834 ms (ouch)
> 
> The difference between 8 and 11/12 is significant (this is within
> Java, so it is not just a startup difference). The difference between
> 11 and 12 is random.  Graal is really bad here, but loading the data
> is just 2-5 seconds total here anyway.
> 
> Running DBSCAN clustering on this data, using linear scans (I also
> have results with index acceleration):
> 
> Java   8: 231677 ms
> Java  11: 255017 ms
> Java  12: 255266 ms
> Graal EE: 261761 ms
> 
> Again, Java 11 and 12 are similar, within the usual measurement differences.
> But they are both 10% worse than Java 8, which is significant. Graal is even
> 13% worse. In other experiments, I have seen differences of 15% (J11/12) and
> 20% (Graal).
> 
> For larger data sets, many iterations, etc. this runtime difference of 30
> seconds quickly adds up to a lot of time...
> 
> Because of this, I currently stick to Java 8.

Thank you for the bug report and the detailed results.  Would you be
able to share a gist/pointer with more details about how you conducted
the benchmark?  I'm not familiar with ELKI but am interested in doing
this sort of benchmarking on an ongoing basis.

Thank you,
tony

Attachment: signature.asc
Description: PGP signature