Jumi is my latest side project and it will be a test runner for the JVM. Its short-term goals are to run tests faster (via parallelism, class loader caching, heuristics and stuff), provide a better user experience than existing test runners, and to overcome limitations of JUnit's test runner which cause an impedance mismatch with for example most of Scala's testing frameworks. Jumi's long-term goal is to be supported by every build tool and IDE, and to surpass JUnit as the de facto test runner on the JVM.

I'll be documenting the development of this app as screencasts also for the purpose of being some research material (I know some researchers at the University of Helsinki who are interested in it). The resolution is higher (1920x1200 vs. 1440x1080) and font smaller (14 vs. 16) than in my previous videos, to make it more practical for me to do programming. Also I will try to optimize my release process so that I can release episodes faster with less effort. For example I will minimize post-production, and write a new name and description only for each theme instead of each episode.

Here is the first episode. Enjoy!

Creating a multi-module Maven project structure and using end-to-end tests to test-drive the build configuration, especially some unusual build requirements (i.e. putting a JAR in a JAR and hiding external dependencies).

Creating a walking skeleton to get the project started. We will start with an end-to-end test of running an empty test suite, which will drive the design for communicating with the daemon process.

We continue with an end-to-end test for executing one passing test. This requires writing a minimal testing framework for executing the dummy tests, which in turn drives us to create most of Jumi's public API.

Using the previously written public API to write the minimal testing framework, until it can report tests which it finds. Then making the end-to-end test of running one test pass with the simplest possible implementation.

Some simple refactorings before we start implementing the actor model. Using immutable collections from Functional Java simplifies some code and requires TestId to implement Comparable.

Though it was possible to make the previous end-to-end test pass without an actor model, it's in a desperate need of one, so now would be a good time to create it.

A bug in the Shade Plugin's configuration had slipped unnoticed for quite some time, so let's improve the build tests to be stricter about the contents of the build artifacts. Also a bug fix and some improvements to the TestId.compareTo() method.

I had a short up-front design session to figure out what actors the system will need. Many new interfaces will be needed for actor-to-actor communication and the current way of writing the event classes by hand will not scale. I will at first write a reflection-based solution for producing those events, because I can do that faster than it would take to figure out how to do compile-time code generation.

After starting to write the new actors, there comes the need for actors with multiple interfaces - or multiple actors in the same thread, however you want to think about it.

Converting the application to be actor-based. The TestRunCoordinator class which remains from the walking skeleton (written in episode 33) needs to be replaced with actor-based suite and test class runners.

Implementing the actor-based runners got too complicated, but I have a hunch that it would be easier to implement them if I would have a simple way to start worker threads. They would run unattended after started and have a callback of when they finish. So let's improve the actor infrastructure.

Creating a second implementation of the Actors class which will be single-threaded and useful for unit testing. I will use abstract contract tests for the common behaviour which must work with both threading models. Also some tests need to be refactored to decoupled them from the threading model.

Putting the single-threaded actors into use and cleaning up the actor tests by extracting recurring abstractions. Also trying out an idea about extracting details into a super class which I read about recently.

Finishing the work on the new actor-based runners, now that the actor framework has been improved to better match their needs.

I registered the domain jumi.fi so I will need to change all packages and groupIds of the project. The build tests prove their worth once again by reminding about all the places which need to be updated.

After a quick test for the SuiteListener interface, the next big refactoring is to create a source code generator for the event stubs of the actors framework. It should be best implemented as a Maven plugin, which I have never done before, so it will be interesting to find out how to do it.

Creating classes which generate the Java source code of the event stubs based on an event listener interface. There will be one factory, one frontend and one backend class for the interface, plus one event class for each method of the interface.

Generating source code for the factory class which will instantiate all the other generated classes.

Generating source code for the frontend class which will convert method calls into event instances.

Generating source code for the backend class which will convert event instances into method calls.

Generating source code for the event classes whose instances will be passed through event queues.

To get started on writing the Maven plugin, we first need some way to write tests for it. I found from Stack Overflow a list of options, of which I will evaluate the alternatives for integration testing: shitty-maven-plugin and maven-invoker-plugin. I choose the latter one and write an integration test using it.

Our Maven plugin must use the Java compiler to compile the listener interface's source file, because the code generator must be able to use reflection on it. This means fiddling with class loaders and more integration with Maven to find out the project's classpath. Taking a look at the Maven Compiler Plugin is helpful. We will also see how to write post-build verification scripts for the Maven Invoker Plugin.

We can now take our code generator Maven plugin into use, which should weed out the remaining missing features. I also figured out how to configure Maven Invoker Plugin so that we can use JUnit and other libraries in the integration test scripts.

We need to create a new module for a Java agent, so now that we touch the build tests, it's a good idea to refactor them into parameterized tests. That should make them more maintainable. That will also allow me to write tests for the test data, to make sure that all of the artifacts follow some project principles.

I will create a Java agent which does bytecode manipulation to check that non-thread-safe classes (i.e. classes annotated with JSR-305's @NotThreadSafe) are used only from one thread. Unit tests for bytecode manipulation can be written using a custom class loader, which transforms only the bytecode of a selected class, and a bunch of "guinea pig" classes [1] to be used as test data.

[1] See GOOS page 284.

The tests are in place, so it's possible to write the actual bytecode transformations. ASM's user guide is a good reference for learning Java bytecode.

Putting the thread-safety checker's Java agent into real use, which helps to weed out some bugs.

The stack traces from the thread-safety checker are missing line numbers for the method which was called from a wrong thread, so let's add those to the generated bytecode.

Some refactoring to improve thread-safety checker's exception messages before starting to use it.

Adding JSR-305 annotations (@NotThreadSafe etc.) and actually finding one concurrency bug, so our doubts were not unwarranted.

Because of the way how Java shortens chained strack traces, the stack traces in last episode were a bit confusing to read, so let's improve that before continuing with more concurrency testing.

Adding the rest of the JSR-305 annotations and doing some performance testing for the thread safety checker.

Begin adding support for reporting test failures. Let's first focus on getting just the number of failed tests reported, and leave reporting the stack traces for later.

In the last episode there was a test failure with a misleading error message. I will evaluate JMock and EasyMock to see whether they would produce better error messages than Mockito.

None of Java's three popular mocking frameworks produced as readable error messages as I would like, so let's try creating a hand-written mock.

Let's see where the end-to-end tests fail next and continue our implementation of failure reporting from there.

Improving the SpyListener and SingleThreadedActors classes to fail early, to help catching more problems.

Let's see where the end-to-end tests fail next and continue our implementation of failure reporting from there.

The end-to-end tests were failing because our minimal SimpleUnit testing framework is incomplete - it does not yet execute tests and report test execution events.

Let's see where the end-to-end tests fail next and continue our implementation of failure reporting from there.

With the previous feature done, it's a good time to refactor the code generator some. Let's first generate toString() methods for the event classes and improve the code formatting.

Improving the code generator to produce correct generic type parameters. Even after Java's type erasure it's possible to dig out some type parameter information using reflection.

Now I can see clearer what the code generator's structure should be like, so let's replace the previous procedural approach with a builder.

The generated imports can be made a bit more specific, avoiding superfluous imports.

PIT is the first user friendly mutation testing framework for Java that I know of. Line coverage is a lousy measure compared to mutation coverage, which actually makes sure that your tests are testing something.

Trying a more systematic way of creating a new implementation to pass contract tests than what I did a year ago when originally implementing single-threaded actors. See my blog post for more details on this experiment.

Creating a text user interface and changing the end-to-end tests to run against it. Showing exception stack traces in the UI.

Creating a more production-ready text user interface which will print results as they arrive, instead of waiting for all tests to finish executing.

Improving the new text UI to print the names of tests and their beginnings and endings.

With the new event-based text UI working, we can replace the old text UI with it.

I forgot to add the @NotThreadSafe annotation to TextUI, so let's write a test to make sure I will never again forget that. Those annotations have RetentionPolicy.CLASS so using reflection is not an option and bytecode analyzing is required.

It's time to make the system run tests in parallel. We'll write a test class which passes only if its test methods are called in parallel, and then fix one part of the system which accidentally assumed sequential execution.

The tests are now being executed in parallel, though the events from parallel test runs are still mixed in the output. But the new code must be cleaned up before fixing that.

To improve the end-to-end tests to detect the incompleteness of the system, I will need to parse the text UI.

Our code generator tries to import primitive types, so that needs to be fixed before we can add Run IDs to the SuiteListener interface. This is also a chance to leave the code a little bit better than we found it.

Introducing Run IDs step by step, to make it possible to identify concurrent test runs.

Removing primitive obsession by replacing the use of ints with RunId. Tens of places need to be updated, so it's important to proceed in small steps, passing all tests every couple of minutes.

Adding support for multiple test runs into TextUIParser. Lots of regular expressions are needed, including less often used regex features such as positive lookahead and embedded flag expressions.

Adding support for multiple test runs into TextUIParser. Lots of regular expressions are needed, including less often used regex features such as positive lookahead and embedded flag expressions.

With TextUIParser implemented, we can improve the end-to-end tests to check individual test runs, which will expose what has not yet been implemented in the support for multiple test runs.

The next feature to add is generating the RunIds. But before implementing that, let's take a moment to refactor the classes which will soon be modified.

Writing unit tests in SuiteRunnerTest revealed that the DriverFinder interface makes things hard to test, so let's improve the design to make it testable.

An earlier refactoring exposed a problem with the code generator which caused the generated event stubs of one interface to overwrite those of another interface.

The user interface supports multiple test runs, but the backend does not yet generate unique RunIds for the runs.

The RunIds and the onRunStarted/onRunFinished events must still be propagated through the rest of the system and put into full use.

When I rewrote TestClassRunnerTest and SuiteRunnerTest in specification style, I left behind tests which were not isolated - they were checking more things than just the one thing they were interested in. I figured out a way to write them better.

With the onRunStarted event and RunId parameters in place, some of SuiteListener's other parameters can be deduced from the RunId, so they can be removed as redundant.

Some small refactorings. A nice feature in IntelliJ IDEA is "Analyze | Run Inspection by Name" which lets you apply a quickfix with one command for all found problems in the whole project - in this case adding missing @Override annotations.

Extracting from TextUI a bunch of reusable classes for processing suite results.

Still some more small refactorings.

The Jumi Actors library can be made easier and safer to use by introducing a reference class, a bit like in Akka Actors. This is the first step for a bunch of design improvements to the actors library.

By unifying the processing of "primary" and "secondary" actors, the internal design can be improved considerably, and later we can get rid of those two different actor categories completely.

Creating an actor used to be coupled with creating its thread (or reusing the current thread). By separating those operations we can unify "primary" and "secondary" actors, making the actors much simpler to use.

If we create an Executor which notifies when it has finished executing its commands, that should help in getting rid of Actors.startUnattendedWorker() and the numerous callbacks of the old WorkerCounter.

Migrating away from using unattended workers into using the new MonitoredExecutor class.

The MonitoredExecutor class had a bug which was probably the reason why we failed to put it into use.

Migrating away from using unattended workers into using the new MonitoredExecutor class.

After introducing actor threads and removing the unattended workers, the actor tests require some updating.

The usability of the actors library can be improved by making it a single configuration option to support actors of all interface types.

Measuring with Caliper whether it makes sense to cache the DynamicEventizer instances.

Moving thread management and naming from the actor library into an external thread pool and a custom thread factory.

Making debugging actors easier by logging all messages sent to and from actors.

Some actors delegate work to background threads, so those should also be logged to see the full picture.

Jumi has not yet needed support for stopping actor threads (it just exits the whole JVM), but for the Jumi Actors library to be general-purpose, there should be a way to release resources.

Creating hooks for handling uncaught exceptions from actors. Originally only the single-threaded actors had them, but they are also needed in production.

Some polish before the Jumi Actors library is ready for being published, for example improved validation of actor interfaces.

Maven includes in every JAR a pom.properties file which contains the artifact's groupId, artifactId and version. I would like to include there also Git's revision ID. Unfortunately maven-jar-plugin's pomPropertiesFile option is broken and we need a workaround.

A couple of finishing touches before publishing the actor library. For example fixing a cyclic package dependency revealed by IntelliJ IDEA's Dependency Structure Matrix.

Doing the first release of the Jumi Actors library. I will explain my deployment pipeline for implementing Continuous Delivery to deploy Maven artifacts into Maven Central using Go and Sonatype OSSRH.

You may also read about my deployment pipeline, and how to implement it yourself, in the article Continuous Delivery with Maven and Go into Maven Central.

Though it had not yet bitten me, I realized that WorkerCounter has a concurrency bug that could cause two onAllWorkersFinished events being fired. Let's try to find a more reliable design...

I have one more idea to make WorkerCounter's API harder to use wrong and its implementation simpler.

Next I will want to get the shutting down of the daemon process from walking skeleton quality to production quality. To get a feel for the code in that area, let's start with something easy - command line parsing.

With command line parsing working, next up is passing parameters as system properties.

The recent unit tests failed when doing a clean build because JumiLauncher was still coupled with the embedded daemon JAR.

To get started on daemon shutdown commands, we need to expose the java.lang.Process instance to assert whether it's dead or alive.

The code which sends the startup command to the daemon is from the original walking skeleton and not unit tested, which makes it hard to modify. This part must be re-designed before we can add support for a second command - the shutdown command.

The formerly monolithic JumiLauncher has now been split into multiple actors, each of which can be more or less unit tested in isolation.

The current network layer is too tightly coupled to the idea that the launcher connects to the daemon. Let's create an anti-corruption layer over Netty.

The formerly monolithic JumiLauncher has now been split into multiple actors, each of which can be more or less unit tested in isolation.

With the launcher code made clean, it's possible to start implementing the shutting down of the daemon process.

In order to make the daemon exit after the launcher disconnects, we must implement disconnecting network connections and a notification about the disconnection.

We could test individually that the launcher releases each resource (i.e. threads and sockets) when it is closed, but it would be easy to forget some resource. So let's write a generic test that all resources are released.

The easiest resource to release is JumiLauncher's actor threads, which also drives us to promote some bootstrap test code into production code.

I was able to figure out a way to get a list of all Socket and ServerSocket instances, so it will be possible to write a test that the launcher closes all network connections.

Releasing all resources reserved by the network layer.

Improving the network layer to support multiple clients connecting to the server.

Testing anything which deals with the current time is hard, because you need to fake the current time or else suffer from flaky and/or slow tests. Encapsulating time and hiding it behind easily testable abstractions is the way to go.

I think I finally found the cause of the flaky network and end-to-end tests; a concurrency issue in what events Netty fires. Also the tests for the timeout class and resource cleanup had issues. Concurrency is hard.

Connecting the dots to shutdown the daemon after a timeout, after the launcher disconnects or doesn't connect in the first place.

The daemon process should print its version number on startup and another message on shutdown. That would make reading the logs much easier.

To avoid changing JumiLauncher always when new suite parameters are added, the current approach needs to be refactored.

The SuiteConfiguration class is a bit non-cohesive, because it holds also daemon-related parameters which are used differently from suite-related parameters.

I have a bunch of ideas how to still refactor the daemon and suite configuration classes, but a prerequisite for quick refactoring is good unit test coverage.

I invented a design pattern which makes it easy to update immutable objects without language support, by providing a way to make an object immutable and back again mutable.

Refactoring towards a generic way of representing DaemonConfiguration's bean properties, to convert them to and from system properties.

A too high Java class file version can break its consumers, so it's worth writing a test for. At the same time we can make it easier to write assertions for all class files in a JAR file.

It turns out that changing a project's language level is not as simple as it seems. Changing Maven's settings on a per-module basis is easy, but IntelliJ IDEA 11 doesn't support compiling multiple modules with mixed language levels (IIRC, newer IDEA versions do support it).

Java 7 comes with a new file I/O API (NIO.2), which has numerous improvements over java.io.File. Upgrading to the new API is best done in small steps, the same way as when refactoring to remove primitive obsession.