GitLab will be upgraded on 30 Jan 2023 from 2.00 pm (AEDT) to 3.00 pm (AEDT). During the update, GitLab and Mattermost services will not be available. If you have any concerns with this, please talk to us at N110 (b) CSIT building.

RELEASE_NOTES.txt 8.83 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
dacapo-9.10-beta0 

RELEASE NOTES  2009-09-10

************************************************************************
* IMPORTANT: This beta release is NOT SUITABLE AS A RESEARCH TOOL.     *
*                                                                      *
* Please wait for the full release before using the suite for research *
* purposes. The benchmarks in this beta release are:                   *
*                                                                      *
*    1) not fully tuned,                                               *
*    2) not fully evaluated,                                           *
*    3) known to have various limitations and bugs, and                *
*    4) subject to change without notice.                              *
*                                                                      *
************************************************************************


This is the first beta release of the anticipated 9.10 release of the
DaCapo benchmark suite.  These notes are structured as follows:

    1. Purpose
    2. Overview
steveb-oss's avatar
steveb-oss committed
24
25
26
    3. Usage
    4. Changes
    5. Known problems and limitations
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44



1. Purpose
----------

This beta release exists specifically (and only) to allow community
feedback and contribution to the upcoming major release, and to allow
JVM vendors to test against the suite prior to the final release. We
strongly encourage the community to explore and evaluate these
benchmarks and to send feedback to the dacapo research group via the
mailing list and/or bug tracker. Please check the list of known issues
(below) before sending feedback.


   mailing list: dacapobench-researchers@lists.sourceforge.net
     (subscribe via: http://sourceforge.net/mail/?group_id=172498)

45
46
47
48
   bug tracker:
       http://sourceforge.net/tracker/?group_id=172498&atid=861957
     or navigate to it from
       http://sourceforge.net/projects/dacapobench/develop
49
50
51
52


We are particularly interested in feedback on the following:

steveb-oss's avatar
steveb-oss committed
53
54
    o The decision to add benchmarks "avrora", "batik", "derby",
      "sunflow", "tomcat", "tradebeans", and "tradesoap" (see 4.1
55
56
      below).

steveb-oss's avatar
steveb-oss committed
57
58
    o The decision to "retire" benchmarks "antlr", "bloat", "chart",
      and "hsqldb" (see 4.2 below).
59

steveb-oss's avatar
steveb-oss committed
60
61
    o The tuning / running time of each of the benchmarks in their
      default settings.
62

steveb-oss's avatar
steveb-oss committed
63
64
    o The structure and behavior of the tradebeans, tradesoap and
      tomcat client/server benchmarks.
65

66
67
68
    o Any issues associated with building, running and usability of
      the suite.

69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

2. Overview
-----------

The DaCapo benchmark suite is slated to be updated every few years.
The 9.10 release will be the first major update of the suite, and is
strictly incompatible with previous releases: new benchmarks have been
added, old benchmarks have been removed, all other benchmarks have
been substantially updated. It is for this reason that in any
published use of the suite, the version of the suite must be
explicitly stated.

The release sees the retirement of a number of single-threaded
benchmarks (antlr, bloat and chart), the replacement of hsqldb by
derby, the addition of six completely new benchmarks, and the upgrade
of all other benchmarks to reflect the current release state of the
applications from which the benchmarks were derived. These changes are
consistent with the original goals of the DaCapo project, which
include the desire for the suite to remain relevant and reflect the
current state of deployed Java applications.

Each of these benchmarks is tested for both performance* and
correctness nightly. Results are available here:

    o performance:  http://dacapo.anu.edu.au/regression/perf/head.html
    o sanity:  http://dacapo.anu.edu.au/regression/sanity/latest/

      * tradebeans and tradesoap are not yet running performance tests


steveb-oss's avatar
steveb-oss committed
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
3. Usage
--------

3.1 Downloading

  o Download the binary jar and/or source zip from:
      https://sourceforge.net/projects/dacapobench/files/
  
  o Access the source from subversion via
      svn co https://dacapobench.svn.sourceforge.net/svnroot/dacapobench dacapobench

3.2 Running

  o Run a benchmark:
      java -jar <dacapo-jar-name>.jar <benchmark>

  o For usage information, run with no arguments.


3.3 Building

  o You must have a working, recent version of ant installed. Change
    to the benchmarks directory and then run:

       ant -p

    for instructions on how to build.


4. Changes
129
130
131
----------


steveb-oss's avatar
steveb-oss committed
132
4.1. Additions
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168

      avrora: AVRORA is a set of simulation and analysis tools in a
              framework for AVR microcontrollers. The benchmark
              exhibits a great deal of fine-grained concurrency. The
              benchmark is courtesy of Ben Titzer (Sun Microsystems)
              and was developed at UCLA.

       batik: Batik is an SVG toolkit produced by the Apache foundation.
              The benchmark renders a number of svg files.

       derby: Derby is an in-memory database benchmark, using the
              Derby database produced by the Apache foundation, and
              executing the pseudojdbc workload previously used by
              hsqldb.  Derby replaces hsqldb.

     sunflow: Sunflow is a rendering system for photo-realistic images
              based on raytracing engine.

      tomcat: Tomcat uses the Apache Tomcat servelet container to run
              some sample web applications.

  tradebeans: Tradebeans runs the Apache daytrader workload "directly"
              (via EJB) within a Geronimo application server.  Daytrader
              is derived from the IBM Trade6 benchmark.

   tradesoap: Tradesoap is identical to the tradebeans workload, except
              that client/server communications is via soap protocols
              (and the workloads are reduced in size to compensate the
              substantially higher overhead).

Tradebeans and tradesoap were added as a pair specifically to allow
researchers to analyze overheads associated with the widely used soap
protocol.
             


steveb-oss's avatar
steveb-oss committed
169
4.2. Deletions
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
 
       antlr: Antlr is single threaded and highly repetitive. The
              most recent version of jython uses antlr; so antlr
              remains represented within the DaCapo suite.

       bloat: Bloat is not as widely used as our other workloads
              and the code exhibited some pathologies that were
              arguably not representive or desirable in a suite that
              was to be representive of modern Java applications.

       chart: Chart was repetitive and used a framework that appears
              not to be as widely used as most of the other DaCapo
              benchmarks.  The Batik workload has some similarities
              with chart (both are render vector graphics), but is
              part of a larger heavyly used framework from Apache.

      hsqldb: Hsqldb has been replaced by derby, which runs the same
              workload but uses a much more widely used database
              engine.


steveb-oss's avatar
steveb-oss committed
191
4.3. Updates
192
193
194
195
196
197
198


All other benchmarks have been updated to reflect the latest release
of the underlying application (with the exception of eclipse, where
the update to 3.5 is underway).


steveb-oss's avatar
steveb-oss committed
199
4.4. Other Notable Changes
200
201
202
203
204

The packaging of the DaCapo suite has been completely re-worked and
the source code is entirely re-organized.


steveb-oss's avatar
steveb-oss committed
205
5. Known Issues
206
207
208
209
210
211
212
213
214
215
216
217
---------------

Please consult the bug tracker for a complete and up to date list of
known issues (http://sourceforge.net/tracker/?group_id=172498&atid=861957).

DaCapo is an open source community project. We welcome all assistance
in addressing bugs and shortcomings in the suite.


A few notable high priority issues are listed here:


steveb-oss's avatar
steveb-oss committed
218
5.1 Concurrency
219
220
221
222
223

We intend to conduct a concurrency audit, analyzing and documenting
the level of concurrency in each of the workloads. The per-benchmark
documentation available at the commandline (with the -i switch) should
report the level of threading for each benchmark. Although most
224
benchmarks exhibit significant concurrency, we have made the conscious
225
226
227
decision to continue to include a few single-threaded benchmarks. We
do this because as long as there exist a significant number of single
threaded applications in popular use, the single-threaded performance
steveb-oss's avatar
steveb-oss committed
228
229
230
231
232
233
234
235
236
of a JVM is important. 


5.2 Trade

The trade benchmarks currently do not reliably run beyond one or two
iterations (and consequently don't appear in our peformance regression
results since those regressions perform 10 iterations). The benchmarks
are known to suffer from memory leaks and database deadlocks.
237
238


steveb-oss's avatar
steveb-oss committed
239
5.3 Eclipse
240
241
242
243
244
245

We are in the process of updating eclipse, but have not yet completed
this, so at present eclipse runs the same version as it did in the
2006 release.


steveb-oss's avatar
steveb-oss committed
246
5.4 Derby
247
248

We intend to move to using a TCPC workload rather than PseudoJDBC for
249
derby.  We also wish to remove derby's build-time dependency on a 1.4 JVM.
250

steveb-oss's avatar
steveb-oss committed
251
252
253
254
255
256

5.5 Tomcat

We intend to make the tomcat workload more interesting. Performacne
results show that tomcat currently has a remarkably flat warm-up curve
when compared to other benchmarks.