Benchmarking framework for Mu projects
We'd want a benchmarking framework to run various tests across different Mu projects (implementations, clients).
This will help us discover bugs or performance problems introduced in commits. It can also facilitate development by knowing, for example, what the impact of decisions are.
Currently, we have a benchmark suite developed by @u5157779 under https://gitlab.anu.edu.au/mu/mu-impl-fast/tree/master/tests/test_jit
It compares the performance of
- RPython with Mu backend (which can be run on different Mu implementations)
- RPython with C backend
- Hand-written C
- Hand-written Mu (which can be run on different Mu implementations)
Some of these are specific to a certain Mu client (in this case, Mu backend of RPython).
For a more general framework, following client-neutral aspects can be abstracted out
- Collecting, storing and processing metrics
- Visualizing Then, each project can have its own agent for the framework to invoke for executing tests and collecting results.