Benchmark Calibration
Description
This is the harness tracker for all benchmark calibrations.
The updated benchmarks need to be calibrated to modern machines.
See the corresponding issues on the official GitHub repo. Clearly document corresponding issues when one is finished.
Progress & TODOs
Mostly done. There are still a few benchmarks that need discussion
-
batik
: there is a choice between a few sizes (test-{a,b,c}
); decide what to do with these, and which one is fitting fordefault
size. -
fop
: there has been some difficulty finding large FOP-XLS files; check if the updated workload is okay. -
PMD
: discuss about thejunit
size. -
lusearch
: difficulty in making the right decision on increasing the workload (queries) size; the generation script consumes too much memory, and for large queries the design doesn't make sense (not enough amount of words). -
DayTrader: mysterious bug in patching Geronimo SUID bug.
Performance Figures
before calibration
default
size
default (s) |
fisher.moma |
lynx.moma |
weasel.moma |
weasel.moma (-t 8 ) |
---|---|---|---|---|
avrora | 2.436 | 6.167 | 10.157 | 10.442 |
batik | 1.664 | 2.851 | 2.284 | 2.160 |
eclipse | 17.934 | 28.422 | 21.938 | 22.398 |
fop | 1.108 | 1.835 | 1.154 | 1.173 |
h2 | 4.034 | 8.007 | 17.906 | 10.031 |
jython | 3.621 | 7.522 | 4.858 | 4.862 |
luindex | 1.350 | 2.111 | 1.534 | 1.543 |
lusearch | 1.018 | 1.782 | 0.952 | 0.972 |
pmd | 4.378 | 7.388 | 5.655 | 5.501 |
sunflow | 1.588 | 2.972 | 1.317 | 2.517 |
tomcat | 2.240 | 3.521 | 3.319 | 3.385 |
tradebeans | 5.348 | 10.908 | 18.113 | 10.639 |
tradesoap | 9.392 | 13.913 | 10.041 | 8.630 |
xalan | 1.257 | 2.398 | 1.692 | 1.096 |
after calibration
default
size
default (s) |
fisher.moma |
lynx.moma |
---|---|---|
avrora | 11.554 | 25.899 |
batik | TBC | TBC |
eclipse | 10.017 | 16.389 |
fop | 2.662 | 4.899 |
h2 | 10.032 | 19.184 |
jython | 10.619 | 26.868 |
luindex | 10.115 | 17.808 |
lusearch | N/A | N/A |
pmd | 9.637 | 18.076 |
sunflow | 10.419 | 22.419 |
tomcat | 9.322 | 11.604 |
tradebeans | 11.663 | 23.996 |
tradesoap | 8.820 | 12.777 |
xalan | 10.549 | 19.662 |
large
size
large (s) |
fisher.moma |
lynx.moma |
---|---|---|
avrora | 37.356 | 95.976 |
batik | TBC | TBC |
eclipse | 35.972 | 62.602 |
fop | N/A | N/A |
h2 | 37.895 | 72.162 |
jython | N/A | N/A |
luindex | 469.390 | 845.633 |
lusearch | N/A | N/A |
pmd | 32.609 | 67.738 |
sunflow | 104.177 | 224.306 |
tomcat | 118.791 | 128.725 |
tradebeans | 37.185 | 85.180 |
tradesoap | 15.698 | 24.931 |
xalan | N/A | N/A |
LuSearch | ox.moma |
---|---|
default | 1.534 |
large | 39.463 |