Combinative testing – make your test design efficient

Combinatorial testing aims to determine data combinations to be tested. In some cases, when the risk is high or the application is very complex, you probably use combinatorial testing.

The most simple and widely used combinatorial method is called “all-pairs” or “pairwise” testing, which is “a black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.”

We demonstrate pairwise testing on a simple example below:

EXAMPLE: CONFIGURATION TESTING

Consider a system under test with 4 parameters, A, B, C, D. These parameters have the following values:

  • Parameter A: a1, a2, a3, a4,
  • Parameter B: b1, b2, b3,
  • Parameter C: c1, c2,
  • Parameter D: d1, d2, d3, d4.

Analyze the possible combinatorial testing techniques.

With the 16 test cases in the table below pairwise test selection criterion is satisfied.

TestABCD
1a1b1c1d1
2a1b2c2d2
3a2b2c1d1
4a2b1c2d2
5a3b3c2d1
6a3b3c1d2
7a4b1c1d3
8a4b2c2d4
9a1b3c2d3
10a2b3c1d4
11a3b2-d3
12a3b1-d4
13a4b3-d1
14a1--d4
15a2--d3
16a4--d2

In the table ‘-’ denotes the “do not care” symbol. Please check that all data value pairs can be found in a test case, e.g. a2-d4 pair is in Test 10. Even this simple combinatorial method is not linear, leading usually many test cases. However, by applying more simple techniques, such as equivalent partitioning with boundary value analysis may result in weak quality. What can we do? The solution is combinative testing introduced in Practical Test Design.

The rationale behind the combinative technique is to assure that for a single parameter each computation is calculated in more than one context. The basic variant of combinative testing is called diff-pair testing. Informally, it requires that each value p of any parameter P be tested with at least two different values q and r for any other parameters. For example, if the parameter P has two possible values, let us say x and y, and there are two more parameters Q and R with values: Q: (1, 2) and R: (Y, N), then we have to test the value pairs (x, 1), (x, 2), (x, Y), (x, N). Thus, the test suite required for testing parameter P contains the data combinations {(x, 1, Y), (x, 2, N)} or {(x, 1, N), (x, 2, Y)}. Similarly, for y, we test {(y, 1, Y), (y, 2, N)} or {(y, 1, N), (y, 2, Y)}.

We test the values of the other parameters R and Q similarly.

Considering our configuration testing example, we need only 8 test cases:

Test case Nr.ABCD
1a1b1c1d1
2a1b2c2d2
3a2b1c2d3
4a2b2c1d4
5a3b1c1d2
6a3b3c2d1
7a4b3c1d3
8a4b1c2d4

You can see that a1 is tested with (b1, b2), (c1, c2) and (d1, d2). Similar is the case for a2, a3 and a4. b1 is tested with (a1, a2), (c1, c2) and (d1, d3). You can easily see that there are pairs with different values for all the parameters.

In our book we proved that this method is linear, i.e. having N parameters with M values each, the number of test cases will be 2×Max(N, M). This is exactly what we need. We test all the parameters with different values of all other parameters. Thus, we combine testing without sacrificing linearity. In our book we showed that combinative testing is very efficient both for computation and control-flow errors.

2 thoughts to “Combinative testing – make your test design efficient”

  1. Thanks for commenting.
    The first step is to introduce the book to the world (new ideas, techniques, approaches).
    The second is to work out the practical exercises in this page.
    We thanks for any support and suggestion.

Leave a Reply

Your email address will not be published. Required fields are marked *