Combinative testing – make your test design efficient

Combinatorial testing aims to determine data combinations to be tested. In some cases, when the risk is high or the application is very complex, you probably use combinatorial testing.

The most simple and widely used combinatorial method is called “all-pairs” or “pairwise” testing, which is “a black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.”

We demonstrate pairwise testing on a simple example below:


Consider a system under test with 4 parameters, A, B, C, D. These parameters have the following values:

  • Parameter A: a1, a2, a3, a4,
  • Parameter B: b1, b2, b3,
  • Parameter C: c1, c2,
  • Parameter D: d1, d2, d3, d4.

Analyze the possible combinatorial testing techniques.

With the 16 test cases in the table below pairwise test selection criterion is satisfied.


In the table ‘-’ denotes the “do not care” symbol. Please check that all data value pairs can be found in a test case, e.g. a2-d4 pair is in Test 10. Even this simple combinatorial method is not linear, leading usually many test cases. However, by applying more simple techniques, such as equivalent partitioning with boundary value analysis may result in weak quality. What can we do? The solution is combinative testing introduced in Practical Test Design.

The rationale behind the combinative technique is to assure that for a single parameter each computation is calculated in more than one context. The basic variant of combinative testing is called diff-pair testing. Informally, it requires that each value p of any parameter P be tested with at least two different values q and r for any other parameters. For example, if the parameter P has two possible values, let us say x and y, and there are two more parameters Q and R with values: Q: (1, 2) and R: (Y, N), then we have to test the value pairs (x, 1), (x, 2), (x, Y), (x, N). Thus, the test suite required for testing parameter P contains the data combinations {(x, 1, Y), (x, 2, N)} or {(x, 1, N), (x, 2, Y)}. Similarly, for y, we test {(y, 1, Y), (y, 2, N)} or {(y, 1, N), (y, 2, Y)}.

We test the values of the other parameters R and Q similarly.

Considering our configuration testing example, we need only 8 test cases:

Test case Nr.ABCD

You can see that a1 is tested with (b1, b2), (c1, c2) and (d1, d2). Similar is the case for a2, a3 and a4. b1 is tested with (a1, a2), (c1, c2) and (d1, d3). You can easily see that there are pairs with different values for all the parameters.

In our book we proved that this method is linear, i.e. having N parameters with M values each, the number of test cases will be 2×Max(N, M). This is exactly what we need. We test all the parameters with different values of all other parameters. Thus, we combine testing without sacrificing linearity. In our book we showed that combinative testing is very efficient both for computation and control-flow errors.

8 thoughts to “Combinative testing – make your test design efficient”

  1. Thanks for commenting.
    The first step is to introduce the book to the world (new ideas, techniques, approaches).
    The second is to work out the practical exercises in this page.
    We thanks for any support and suggestion.

  2. whoah this weblog is great i like studying your articles. Keep up the great paintings! You already know, a lot of people are searching around for this info, you could help them greatly.

  3. After study a handful of the content on your web site now, and that i really like your way of blogging. I bookmarked it to my bookmark website list and will also be checking back soon. Pls look at my internet site too and told me what you think.

  4. This is really interesting, You are a very skilled blogger.
    I’ve joined your feed and look forward to seeking more of
    your magnificent post. Also, I’ve shared your site in my social networks!

Leave a Reply

Your email address will not be published. Required fields are marked *