Why another new Manifesto? The answer is simple: we needed to rethink test design. Everybody believes they know how to do it. This is what we thought before writing our book Practical Test Design. But we failed. As we write this book, we discover many new aspects to test design. We spent a lot of time rethinking and improving approaches. We do not think that everything is perfect now. But we are on the right track. A summary of our book-writing journey led to this manifesto.
We strongly prefer justified methodologies over
ad-hoc methodologies. Some test
design techniques are ad-hoc, yet they are widely used. For example, during boundary value
analysis there may be two solutions: two boundary values per boundary and three
boundary values per boundary. Unfortunately, neither of these solutions is
optimal and leads either missing or superfluous test cases. We carefully investigated the optimum, you can
find it in Chapter 5, Domain-based testing.
We strongly prefer combined techniques over
single techniques. In the related
books, articles, blogs the specifications and the requirements are simple.
Hence, a single technique is enough to show its effectiveness. In practice
however, verifying complex specification requires using more techniques
together. We show several examples, why and how to do it.
We strongly prefer linear techniques over
non-linear techniques. Even the simplest combinatorial techniques lead to too many test cases and increased SDLC (software development life-cycle) costs. Using more linear techniques in parallel or our new combinative method result in much fewer tests with higher reliability.
We prefer cost & quality optimized test design instead of
false trade-off between cost & quality. Testing costs are considered by everyone to be self-contained, which usually does not result in an optimal solution in terms of cost vs. quality. The truth is that cost and quality can be optimized together during the SDLC since too many tests do not improve quality significantly but lead to unnecessary costs. Based on the risk, complexity and historical data analysis, the total SDLC costs can be optimized, resulting in appropriate quality as we demonstrated in Chapter 3, Risk Management and Test Optimisation.
We prefer automated techniques instead of
manual techniques. Nowadays
test automation is a must, but it’s not enough. Test design automation with
continuous integration make the automation process complete and more efficient (see
the latest Bloor
research). In our book we describe the general methods and some of
the supporting tools.
We prefer harmonizing academic and industrial results instead of
living close to each other. The industry doesn’t know about the new results
in software testing research and researchers know little about the progress in
the industry. Therefore, the industry cannot use the latest research results
while the academy remains in its
ivory tower. More channels are needed to narrow the gap.