Test Design Manifesto

Why another new Manifesto? The answer is simple: we needed to rethink test design. Everybody believes they know how to do it. This is what we thought before writing our book Practical Test Design. But we failed. As we write this book, we discover many new aspects to test design. We spent a lot of time rethinking and improving approaches. We do not think that everything is perfect now. But we are on the right track. A summary of our book-writing journey led to this manifesto.

We strongly prefer justified methodologies over ad-hoc methodologies. Some test design techniques are ad-hoc, yet they are widely used. For example, during boundary value analysis there may be two solutions: two boundary values per boundary and three boundary values per boundary. Unfortunately, neither of these solutions is optimal and leads either missing or superfluous test cases.  We carefully investigated the optimum, you can find it in Chapter 5, Domain-based testing. 

We strongly prefer combined techniques over single techniques. In the related books, articles, blogs the specifications and the requirements are simple. Hence, a single technique is enough to show its effectiveness. In practice however, verifying complex specification requires using more techniques together. We show several examples, why and how to do it.

We strongly prefer linear techniques over non-linear techniques. Even the simplest combinatorial techniques lead to too many test cases and increased SDLC (software development life-cycle) costs. Using more linear techniques in parallel or our new combinative method result in much fewer tests with higher reliability. 

We prefer cost & quality optimized test design instead of false trade-off between cost & quality. Testing costs are considered by everyone to be self-contained, which usually does not result in an optimal solution in terms of cost vs. quality. The truth is that cost and quality can be optimized together during the SDLC since too many tests do not improve quality significantly but lead to unnecessary costs. Based on the risk, complexity and historical data analysis, the total SDLC costs can be optimized, resulting in appropriate quality as we demonstrated in Chapter 3, Risk Management and Test Optimisation.

We prefer automated techniques instead of manual techniques. Nowadays test automation is a must, but it’s not enough. Test design automation with continuous integration make the automation process complete and more efficient (see the latest Bloor research). In our book we describe the general methods and some of the supporting tools.

We prefer harmonizing academic and industrial results instead of separate living close to each other. The industry doesn’t know about the new results in software testing research and researchers know little about the progress in the industry. Therefore, the industry cannot use the latest research results while the academy remains in its ivory tower. More channels are needed to narrow the gap.