Combinatorial testing aims to determine data combinations to be tested. In some cases, when the risk is high or the application is very complex, you probably use combinatorial testing.

The most simple and widely used combinatorial method is called “all-pairs” or “pairwise” testing, which is “a *black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.”*

*We demonstrate pairwise testing on a simple example below:*

*EXAMPLE: CONFIGURATION TESTING*

*Consider a system under test with 4 parameters, A, B, C, D. These parameters have the following values:*

*Parameter A: a1, a2, a3, a4,**Parameter B: b1, b2, b3,**Parameter C: c1, c2,**Parameter D: d1, d2, d3, d4.*

*Analyze the possible combinatorial testing techniques. *

*With the 16 test cases in the table below pairwise test selection criterion is satisfied.*

Test | A | B | C | D |
---|---|---|---|---|

1 | a1 | b1 | c1 | d1 |

2 | a1 | b2 | c2 | d2 |

3 | a2 | b2 | c1 | d1 |

4 | a2 | b1 | c2 | d2 |

5 | a3 | b3 | c2 | d1 |

6 | a3 | b3 | c1 | d2 |

7 | a4 | b1 | c1 | d3 |

8 | a4 | b2 | c2 | d4 |

9 | a1 | b3 | c2 | d3 |

10 | a2 | b3 | c1 | d4 |

11 | a3 | b2 | - | d3 |

12 | a3 | b1 | - | d4 |

13 | a4 | b3 | - | d1 |

14 | a1 | - | - | d4 |

15 | a2 | - | - | d3 |

16 | a4 | - | - | d2 |

In the table ‘-’ denotes the “do not care” symbol. Please check that all data value pairs can be found in a test case, e.g. a2-d4 pair is in Test 10. Even this simple combinatorial method is not linear, leading usually many test cases. However, by applying more simple techniques, such as equivalent partitioning with boundary value analysis may result in weak quality. What can we do? The solution is combinative testing introduced in Practical Test Design.

The rationale behind the combinative technique is to assure that for a single parameter each computation is calculated in more than one context. The basic variant of combinative testing is called ** diff-pair testing**. Informally, it requires that each value

*p*of any parameter

*P*be tested with at least two different values

*q*and

*r*for any other parameters. For example, if the parameter

*P*has two possible values, let us say

*x*and

*y*, and there are two more parameters

*Q*and

*R*with values:

*Q: (1, 2)*and

*R: (Y, N)*, then we have to test the value pairs

*(x, 1), (x, 2), (x, Y), (x, N)*. Thus, the test suite required for testing parameter

*P*contains the data combinations

*{(x, 1, Y), (x, 2, N)}*or

*{(x, 1, N), (x, 2, Y)}.*Similarly, for

*y*, we test

*{(y, 1, Y), (y, 2, N)}*or

*{(y, 1, N), (y, 2, Y)}*.

We test the values of the other parameters *R* and *Q* similarly.

Considering our configuration testing example, we need only 8 test cases:

Test case Nr. | A | B | C | D |
---|---|---|---|---|

1 | a1 | b1 | c1 | d1 |

2 | a1 | b2 | c2 | d2 |

3 | a2 | b1 | c2 | d3 |

4 | a2 | b2 | c1 | d4 |

5 | a3 | b1 | c1 | d2 |

6 | a3 | b3 | c2 | d1 |

7 | a4 | b3 | c1 | d3 |

8 | a4 | b1 | c2 | d4 |

You can see that a1 is tested with (b1, b2), (c1, c2) and (d1, d2). Similar is the case for a2, a3 and a4. b1 is tested with (a1, a2), (c1, c2) and (d1, d3). You can easily see that there are pairs with different values for all the parameters.

In our book we proved that this method is linear, i.e. having N parameters with M values each, the number of test cases will be 2×Max(N, M). This is exactly what we need. We test all the parameters with different values of all other parameters. Thus, we combine testing without sacrificing linearity. In our book we showed that combinative testing is very efficient both for computation and control-flow errors.

Ѕtunning quest there. What happened after? Thanks!

Thanks for commenting.

The first step is to introduce the book to the world (new ideas, techniques, approaches).

The second is to work out the practical exercises in this page.

We thanks for any support and suggestion.