Create test layers for developers' efficiency

In this post, I explain why neither unit tests nor functional tests are enough, I propose other test layers and I explain how to set them up to enhance the software development process.

When reading articles about development methodologies, I usually realize that the authors either write about unit tests or about functional tests.

Unit tests

Unit tests are considered by some authors as the ultimate testing technic. One is supposed to test everything he writes with unit tests, and everything is supposed to be testable with unit tests. Even if it were theorically possible (is it ?), this point of view would still be nonsense.

As a matter of fact, you don't start your development from nothing. You integrate libraries, you reuse existing code, etc... and this reused code is usually not fully unit tested. The effort to create all the unit tests for any library you use can be unreachable.

Some code writes on the hard disk, some open sockets... many things the 'real' unit tests are not supposed to do.

And even if you don't like it at all, you may have to maintain some old-fashion software, with spaghetti code included. Try to unit test this and you are on your way to a full refactoring.

I don't know of any real software that can be fully unit-tested. Unit-testing most of the code is very efficient, but it's often nor sufficient nor feasible.

Functional tests

At the other end lie the functional tests. What I call functional tests are the ones that prove that the software does what it is supposed to do, on the user's point of view. For instance, when the user clicks on a button, a pop-up appears with the right message.

Most of the functional tests are dedicated to verifying the GUI of the I/O of the system.

Those tests are often under the responsability of QA teams. And QA teams are not well considered. The are at the end of the chain, so they are supposed to be the ones that are late. They discover bugs, and some people don't like to be told their code is not perfect. They often fail to automate their job, because whatever the quality of the tools (Quick Test, for instance), they fail to follow the evolution of a software interface.

So functional tests cost a lot. Have you ever considered that it is one of the first team to be offshored ? No mystery behind this : QA testing is supposed to be expensive, repetitive, and it's not supposed to necessitate to much technical knowledge.

And last but not least, the bugs that are discovered by the QA team are usually very difficult to track. The main reason is that the code has been modified a long time ago, and the developers just don't remember what they did, and why. Code management tools enable the developers to track the differences between versions, but when a new version is delivered to QA, there are usually more than a few modified lines, aren't there ?

Learn from the past...

IMHO, the very first action to take consists in setting up a new process to achieve the goal : no bug in QA functional testing should ever reappear.

The process to set up is quite simple to explain : when you find a functional bug, correct it in 3 steps : First, find the cause. Second, create the unit test that proves the problem. Third, correct it. Don't you ever fix a bug without creating the right unit test ! Except if you want the bug to come back. One day. When you don't need it.

The idea is simple to follow : wherever you find a bug, just create the unit test that proves the bug exists before fixing it. That's the prerequisite for an ever growing number of unit tests that make a stronger software.

And remember, the developers just need feedback. And the faster the feedback, the faster the fixing of the bug. So having powerful but late functional tests is great but not enough. You need more unit tests to give more feedback.

If you need to design a test that is not "unit", you just found out that you will need more than unit tests and functional tests. We'll see that later.

Keeping servers quite, the nightly build myth

I don't understand the nightly build myth.

I know some companies that have a software that builds in 5-10 minutes. And they keep building it from the svn sources every night to run the tests. I just don't understand why the tests are not run for every commit on the code. If someone could tell me, that'd help.

The idea behind this is simple : if you can run all your unit tests for (almost) every commit on the code, just do it. That will help you find out where the errors come from. If you can't, ask yourself if the code could not be refactored, because having the ability to run the tests very often gives you the chance to find bugs sooner.

More layers, more servers

Not everything can be tested with unit tests.

First, it can be of greeat help to test the different modules together. Even if every module is fully unit-tested, gathering several modules together can lead to finding new bugs. So the first new family of tests consists in running tests against several modules together. For instance, you want to test an application that uses a XML parser. You want to test this application against a mock (unit tests...), then with actual XML parsers. Running your unit tests with actual xml parser is an integration test.

One of the main reasons for these integration tests is that you will discover bugs in the libraries you use, and you may have to circonvent them instead of correcting them. That's true if the bug comes from the OS, or from a closed-source library.

Second, you need to test your application in the real world, where you need to open files, sockets, etc. Unit tests are supposed to be independent. But in the reality, your application will actually run on a machine, in an application server, etc. You will need to deploy your app in most architectures where it is supported. For instance, you need to test your app in all versions of a JRE... We can call them deployment tests.

Generally speaking, integration tests and deployment tests are run together, for an obvious reason : it's often impossible to run deployment tests with only mocks. So these layers are often a single one, called integration tests.

Layer Testing Automation
Let's talk about automation :

• The unit tests should be run on the build servers at (almost) every commit
• The integration tests should be run automatically if the unit tests are ok
• The automated functional tests should be run each time the integration tests are ok
  

That's not so easy to set up, but this automation leads to better efficiency in the development process. It's by far more efficient to receive an email each time bug is introduced at integration level than to wait for a team to run the tests manually.

Find bug in any layer, create tests to find it sooner the next time

I do think that the "efficiency obsession" that every team leader should have must lead to this process : whatever the bug you find, try to create a test that will be run automatically. And the sooner the test is run, the better.

If you can create a unit test, please do. If it's not possible, then create an integration test.

But remember : the sooner you find a bug, the faster it is corrected.

The performance tests layer

Some applications have strong performance issues.

Unfortunaltely, testing the performances of an application is quite difficult. A performance is not (usually) "green" or "red". It's better, or worst.

Testing the performance of an application can be done at the end of the whole testing process... but that's dangerous to find issues at the very end.

You can also create performance tests that you will run for every small module of your app. But it will cost a lot to do so.

It's more or less like unit tests : they don't test everything, they are necessary, and you really need to put all the parts together to have a consolidated view of your app.

Testing your performance at the very end is nevertheless necessary. And when it can be done automatically... that's perfect. But you will still need a human being to interpret the results.

updates :
- fixed typos
- Found this link : Agile Testing: Should acceptance tests be included in the continuous build process?