I'm starting a new screencast series Let's Code, where I will be recording myself developing some open source projects. This was inspired by James Shore's Let's Play TDD series and I will try doing something similar. My goal is not to teach the basics of how to do TDD, but to show how one developer does it - in the hope that something can be learned from it. Each episode will be about 25 minutes long ("one pomodoro") and I will try to release a new episode every couple of days, but no promises about that.

This is the first episode of the series. Since I'm starting with an existing code base, it takes some time explain the architecture of the project, so this episode does not yet include coding, but watching it is recommended to understand the following episodes. Implementation will start in the second episode.

If you want to see some code quickly, then episode #5 may be a good place to start. There we start writing some isolated utility classes for asynchronous tests and it doesn't require knowledge of Dimdwarf's architecture (though in my opinion those episodes dealing with Dimdwarf's features are more interesting).

Here is the second episode of the new Let's Code series. We get started with implementing the logging out feature of the system, implementing it step-by-step as driven by our end-to-end tests. The progress is still quite slow, as it takes some time to understand what needs to be done next and to get into the flow. We anyways manage to implement support for one new network message type and get a good start on writing the next test.

Implementing logout continues and we get very close to completing the feature. Our tests are encountering problems with asynchrony, so in a later episode we will create some utility classes to help writing those tests.

Logging out is now working, so it's time to improve the readability of our tests.

In order to easier test our asynchronous system, we need some utility classes which take care of the concurrency issues. In this episode we begin writing them.

This episode can be a good "first episode" for someone evaluating his interest in this show, because here we are writing code which does not require being familiar with the system's architecture. If you like this episode, then you might want to later watch Let's Code Dimdwarf #1, where the current project and its architecture are explained, but where no code is yet written; the coding starts in episode #2.

We try to integrate with the Hamcrest library and make our utility class produce useful exception messages. I haven't written Hamcrest matchers before, so it's not clear that how to do it.

Let's Code is back after a short pause due to the Software Craftsmanship 2010 conference earlier this week. In the concluding talk of the conference there was a fitting quote from The Mythical Man-Month, chapter 1: The Joys of the Craft

In this episode we refactor our test code by extracting a fluent API out of it, to make the test code more readable. There is some unnecessary work in re-inventing wheels, because of not being familiar with what features JUnit and Hamcrest already provide.

We finally figure out what was wrong in the code and manage to clean up the mess which had piled up during the last hour or so. The first draft of the asynchronous testing tool is now ready and in the following episodes we will be making it reusable.

In order to get a reusable utility class for writing asynchronous tests, we re-implement it for another use case. After that it will be possible to extract the common parts into a reusable class.

Now that ByteSink's synchronous tests are passing, it's time to implement the asynchronous behaviour. Unlike with EventSink, this time synchronization needs to be done manually.

Recap of episodes 1 to 10, at 10 times the normal speed, with commentary. Watching this recap should make it easier to follow what is happening in the following episodes, even if you haven't had time to watch the previous episodes.

We move towards a reusable utility class by eliminating the duplication between the existing classes for two different use cases.

Sometimes the rights steps to a refactoring are on a long and narrow path.

Testing error messages requires manual verification plus automatically verifying just the important parts.

When I go through things which I'm about to commit, often I find out that I'm not yet ready to commit.

P.S. After a couple of productive evenings, I've piled up unedited recordings up to episode 30. So at least for now I'll speed up my release pace to reduce the WIP.

Before removing the duplication between two classes or methods, they should be made as similar as possible. There is a small mismatch in what the ByteSink and EventSink matchers match.

If you don't right now know how to do something, embrace the ambiguity and leave it for later.

The proof of a library is in using it.

When you take shortcuts in implementing things, it's important to always remember them (not realistic) or at least make them fail fast, so that you would be reminded about them when they bite you. My IoHandler was not fail-fast; it had a pointless null check.

Coding with backspace is fun. The last half of splitting NetworkSpec requires mostly just removing the excess parts.

A new look at refactoring AsynchronousSink. Earlier we made it work; now we try making it right.

Recap of episodes 11 to 20, at 10 times the normal speed.

The ambiguity left remaining in episode 16 has now been cleared. This ends the current undertaking and in the next episode will begin the work on supporting more than one client.

This episode starts a new arc - implementing support for more than one concurrent client. First we need a reproducable, failing end-to-end test. Also before being able to add support for multiple clients, some preparatory refactoring¹ is needed: wrap the messages from clients into a class, to which we can then easily add a client session handle.

¹ The word prefactoring has already another meaning, so maybe this is "pre-refactoring"? Do you have better ideas for a word?

Between each step of our refactoring, all our unit tests pass, but the end-to-end tests don't. This situtation is possible when the components and their unit tests are so well decoupled (due to using message passing) that changing one component does not affect other components - until they are plugged together. We use the end-to-end tests to drive the refactoring, by fixing components in the order that they fail, until the refactoring has been done to all of them.

When a responsibility of doing something is spread over many classes, it becomes harder to understand and modify. In such a situation the responsibilities should be moved or classes merged, so that all the handling of a responsibility is in one highly cohesive place.

The temptation to make the end-to-end test pass is high, but it's better to slow down and first write some unit tests for the feature. Otherwise the unit test coverage, which is the basis for enabling refactoring, would not be complete. Maybe a rule of thumb would be: "If end-to-end tests fail, but unit tests pass, you need more unit tests."

To be able to write a unit tests for connecting multiple clients, a helper class needs to be extracted from the existing test code.

Earlier we had implemented a feature, but the test had a bug. Now that the test has been fixed, there is a temptation to continue when we see it pass, but that would be dangerous. It's better to revert the implementation and see the fixed test fail, to make sure that the test is working.

The last thing remaining to support multiple clients is to update the authentication layer.

Support for multiple clients has now been implemented. When making this system, initially I made it work on just one client, because I knew that adding features later to a simple codebase is easy and then your understanding of the problem will be better. It took only a bit over 4 hours to add support for more than one client - surely not much longer than if I had built it in already in the beginning.

Recap of episodes 21 to 30, at 10 times the normal speed.

P.S. I just got a decent microphone. The sound is now much clearer than what it used to be. Unfortunately it will take until episode 42 for the regular episodes to be recorded with this mic.

I did some work off-camera to make the actors and message queues type parameterized, so that a similar bug as was in episode 25 would not happen ever again. In the beginning of this episode I will explain what I did and what kinds of Java generics tricks I was able to use to minimize the boilerplate in the Guice modules. Then it's time to write unit tests for things which are covered only by end-to-end tests.

Some areas of the system are still only covered by the end-to-end tests, so they need some unit tests to enable testing them faster and to put more design pressure on them. One of these areas is application loading. I begin by choosing the names of the tests to be written, which is easy because I already know based on the existing implementation that what features it has.

We need a reusable utility class for managing temporary directories, so we will extract one from our existing test utilities.

When creating temporary directories in tests, a timestamp and a global variable is not unique enough. You should always assume that the tests will be run in parallel. Even when the test runner you are using does not have built-in support for running them in parallel (though it could in the future), it's possible to run the same test suite multiple times in different processes (which may happen especially on CI servers - I once had tests conflicting because of listening on a fixed network port number).

I begin writing tests for application class loading. I had planned on test-driving some new code, but because I so seldom write code test-after, instead I decided to refactor the existing code and make it testable as if dealing with legacy code.

I will start by writing one unit test to cover the application loading, after which it will be safe to extract the code into its own class.

The methods were easy to move to another class when they all were static, but extracting the class does not end there - the design needs to be made right.

Still some more cleaning up of the production and test code until they are tidy enough for us to move forward.

P.S. Doesn't (0..*) look like a smiliey to you? :D

Creating your own assert methods is oftentimes useful, to make the test code and the assertion messages read better. In this episode I will also write a test for class loading from JAR files.

Recap of episodes 31 to 40, at 10 times the normal speed.

I had an idea to see how the program would fare without a Dependency Injection (DI) framework such as Guice. It should be possible to use just manual DI. It might even be much simpler that way. To find out, I will do a spike.

Now that a hard coded manual dependency injection configuration is working, it will be possible to begin refactoring it and remove all duplication, until a reusable framework or DSL emerges.

The main principle of dependency injection is to make all dependencies of a module explicit. By refactoring the hard coded DI configuration so that the dependencies between modules become more explicit and the duplication more obvious, the design desired by the code begins to emerge.

Next I refactor the DI configuration towards an easy-to-use API.

I move the generic code into its own classes and the DI configuration becomes quite orderly.

The spike for manual dependency injection configuration is nearing its end and it's a good time to compare it with the original Guice-specific code which was needed to do the same thing. The infrastructure code is 50 lines of Scala for manual DI, vs. over 120 lines of Java code for Guice-based DI (which also includes lots of complex reflection). There is slightly more configuration code with manual DI, but it's also more explicit and it's easier to find out what is happening.

I begin using the args4j library for parsing command line arguments. The tests I write are mostly learning tests.

Here starts the work on implementing the sending and receiving of session messages between the client and server. Implementing that will drive the design of the overall architecture quite far, because it will among other things require executing application code in worker threads and committing transactions. As usual, we start with a failing end-to-end test.

The end-to-end tests proved to be useful in catching a mistake in understanding the requirements which caused the unit tests to specify the wrong kind of encoding for session messages.

Recap of episodes 41 to 50, at 10 times the normal speed.

My strategy for implementing session messages - and the half a dozen still nonexistent components which it depends on - is to start with a walking skeleton. This is the approach recommended in the GOOS book which I wrote about in Design for Integrability. In this episode I will explain my strategy and implement the outline of the walking skeleton.

Actually this is the second time during this project that I've used a walking skeleton. The first walking skeleton included receiving a login request and responding to it with a login failure. The first skeleton let me put in place these components: system startup, application loading, networking, authentication and the controller-actor architecture. This second skeleton will produce: unique timestamps, session IDs, worker thread handling, application APIs to listen for and send session messages, executing tasks in application code, sending session messages and committing transactions.

P.S. I wonder whether these may technically be called the first and second walking skeleton, since they don't include all main architectural components. This system has lots of essential complexity, so I prefer taking as small steps as possible. You could say that at first I implemented the legs of the walking skeleton (the controller-actor achitecture), now I implement the hands of the walking skeleton (task execution), and later I will implement the head of the walking skeleton (persistence).

In this episode I get the session message roundtrip working with the simplest possible implementation - all in one method, without any layering. Of course this cannot work in production, but it lets us get started with something that works. After that it's time to do progressive deepening - introduce more layers into the software and push the functionality to the layer where it belongs to - and also do progressive widening - flesh out those layers with more functionality and production-ready quality (see Craftsmanship and Ethics at 18-21 min).

I will start deepening and widening the design by first fixing the encapsulation issue with the network layer - the NetworkActor's private message queue should only be accessed by the NetworkController.

To be able to implement session IDs, guaranteed unique values will be needed - also in a clustered environment. That can be solved by creating unique timestamps based on the paper Time, Clocks, and the Ordering of Events in a Distributed System (though first I'll implement simpler timestamps). I'll start by implementing the ordering of the timestamps, for which parameterized tests are useful.

It's always good to be extra careful when dealing with the maximum and minimum values of an integer. After solving that bug in the tests, I'll improve the toString method.

There needs to be a factory for generating unique timestamps.

The factory for generating unique timestamps must be thread-safe. So I will write a test for thread-safeness and start using an AtomicReference.

Once we have timestamps, generating session IDs is trivial (but anyways worth writing a test for). Then the real thing is mapping those IDs to sessions, which is far from trivial.

When you would like to expose some internal state of a class just to be able to write tests for it, it's usually better to extract that internal thing into its own class which can then be tested through its public interface. Thus I will start writing a new class for keeping track of the mapping between sessions and session IDs.

As disconnecting sessions comes to play, the code starts to inhibit code smells - conditional logic. It would be possible to get rid of those smells with a state machine and the state pattern, but the smells are still quite faint, so let's implement still one more step before refactoring, so that it will be obvious that what shape the code should take.

The code starts to be ripe enough to be refactored into a state machine.

It's curious how sometimes with TDD the refactoring steps follow each other almost automatically, like dominoes. You make one small change and then just keep on fixing test failures until it all works again and the refactoring is done.

The ClientSessions class will absorb most of the logic from NetworkController, leaving the controller just wiring things together and delegating all work to others. Let's first move the authentication logic.

When a collaborator can work both synchronously and asynchronously, it complicates things as the caller needs to make less assumptions. In our unit tests the authenticator is a synchronous fake (it keeps the tests simpler), but the production authenticator is asynchronous (because it does blocking operations). The state machine needs to be updated to work with both synchronous and asynchronous collaborators.

The state machine has now been implemented and most logic has been moved from the controller to the ClientSessions state machine, leaving the controller focused on just simple mapping. Only some refactoring remains.

The ClientSessions class is good enough for now, so we can resume doing what initially caused the need for it; we need to keep the SessionHandle private to the network layer and use only SessionId elsewhere.

Deepening the design of our walking skeleton by creating the application API for sending and receiving session messages.

Deepening the design of our walking skeleton by introducing asynchronous messages for transaction commit requests.