openjdk team mailing list archive

[Erich Schubert <erich@xxxxxxxxxx>]

Hi,

To reproduce the experiments, get the (older than what I used) ELKI 
standalone jar here:

https://elki-project.github.io/releases/release0.7.5/elki-bundle-0.7.5.jar

The data set can be downloaded here:

http://elki.dbs.ifi.lmu.de/datasets/multiview/aloi/aloi-hsb-2x2x2.csv.gz

A suitable command line is:

taskset -c 0 $JAVA_HOME/bin/java \
-jar elki-bundle-0.7.5.jar \
cli -time -dbc.in aloi-hsb-2x2x2.csv.gz \
-algorithm clustering.DBSCAN -dbscan.minpts 20 -dbscan.epsilon 0.01 \
-evaluator NoAutomaticEvaluation -resulthandler DiscardResultHandler

the "cli" command runs this headless; the "-time" parameter enables a 
minimal output of statistics. The last two parameter disable the 
(unnecessary for this purpose) evaluation and result output. tasksets 
forces this to use only the first CPU (except for the garbage 
collection, everything is single-threaded in this call).

The output is just three lines of statistics:

de.lmu.ifi.dbs.elki.datasource.FileBasedDatabaseConnection.load: 1967 ms
de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.average-neighbors: 
84.39775963718822
de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.runtime: 258124 ms

You can ignore the second line. The first is the time needed for parsing 
the csv.gz, the last line is the runtime of DBSCAN without index (adding 
an index with additional command line options speeds this up a lot, but 
for regression testing it does make sense to also look at the 
linear-scanning runtime). Alternatively, you can use the wall-clock-time 
measurement with /usr/bin/time, if you prefer that.

Regards,
Erich


On 8/17/19 9:29 PM, tony mancill wrote:

> Hello Erich,
>
> On Sat, Aug 17, 2019 at 08:41:23PM +0200, Erich Schubert wrote:
> > Subject: openjdk-11-jre: Performance of JDK11/12 is 10%-15% worse than JDK8
> > Package: openjdk-11-jre
> > Version: 11.0.4+11-1
> > Severity: normal
> > Tags: upstream
> >
> > In my benchmarks, JDK 11 and 12 scores around 10%-20% worse runtime
> > than JDK 8.
> >
> > I also tried Oracle Graal CE and Graal EE, and they perform even
> > worse. I also tried AdoptOpenJDK packages, and these perform similar
> > to the Debian versions.
> >
> > The difference is substantial; this is not a toy microbenchmark that
> > only tests a single tiny code fragment. Instead, I run ELKI to
> > benchmark DBSCAN clustering on a dataset with over 100k points; a
> > typical workload that I am interested in.  This code is well
> > optimized, and requires very little garbage collection (5 young gen
> > runs each, about 80 ms total GC time). Memory limits are high enough
> > to let this run smoothly.  I have measured both in-Java time of two
> > steps, and wall-clock time of the entire process until completion.
> >
> > Loading 110250 vectors from a .csv.gz file with
> >
> > Java   8: 1837 ms
> > Java  11: 1955 ms
> > Java  12: 1946 ms
> > Graal EE: 4834 ms (ouch)
> >
> > The difference between 8 and 11/12 is significant (this is within
> > Java, so it is not just a startup difference). The difference between
> > 11 and 12 is random.  Graal is really bad here, but loading the data
> > is just 2-5 seconds total here anyway.
> >
> > Running DBSCAN clustering on this data, using linear scans (I also
> > have results with index acceleration):
> >
> > Java   8: 231677 ms
> > Java  11: 255017 ms
> > Java  12: 255266 ms
> > Graal EE: 261761 ms
> >
> > Again, Java 11 and 12 are similar, within the usual measurement differences.
> > But they are both 10% worse than Java 8, which is significant. Graal is even
> > 13% worse. In other experiments, I have seen differences of 15% (J11/12) and
> > 20% (Graal).
> >
> > For larger data sets, many iterations, etc. this runtime difference of 30
> > seconds quickly adds up to a lot of time...
> >
> > Because of this, I currently stick to Java 8.
> Thank you for the bug report and the detailed results.  Would you be
> able to share a gist/pointer with more details about how you conducted
> the benchmark?  I'm not familiar with ELKI but am interested in doing
> this sort of benchmarking on an ongoing basis.
>
> Thank you,
> tony