Performance is a feature, your users are needing it and so is your product. The code is
changing, always, keeping track of the effect of those changes on the performance is not easy but it is as important
as fixing bugs.
You don't control what you don't measure. The Eclipse Platform built a performance measure framework for their own needs. The API is pretty simple but the implementation, on the other hand is quite complex, querying for instance system specific commands to retrieve the memory state depending on the platform. Launching a performance test built with this is not easy, neither is getting the result in a human readable way.
Our needs were not as complex as the platform ones, we're not trying to measure the number of microseconds a Shell needs to be instantiated on every platform, we're not trying to launch thousands of performance tests for each of our component.
We decided to write a small framework to measure performance, Pierre-Charles did the first iteration and I'm applying it on EMF Compare
now. This framework is using JUnit4 annotations so that one can easily transform an end user scenario into a
performance test. First warning : this framework is not suited
for micro-benchmarking at all !
Second warning : this framework is not without overhead either, it's filling an EMF model with the
measures (which might take a bit of memory if you write thousands of tests) .
On the other hand, the output of the test suite execution is this EMF model, allowing us to quite easily browse
the results, generate reports, and even more interestingly to compare two snapshots ! It's completely
standalone and is not requiring more than Java, Junit, and the EMF Core Runtime.
In a nutshell, this framework is convenient if you want to measure tasks in the order of seconds, and memory
usage if you're looking at megabytes.
So far it's giving you memory usage (through JMX) and system time for each tests, it's running
the tests several times with a "warmup" phase launching the test but not measuring.
A test looks like this :
The test class
:
And the Test Suite
:
It's here if you want to give it a try, it's EPL, it's on github, feel free to fork it :