# Combinative testing – make your test design efficient

Combinatorial testing aims to determine data combinations to be tested. In some cases, when the risk is high or the application is very complex, you probably use combinatorial testing.

The most simple and widely used combinatorial method is called “all-pairs” or “pairwise” testing, which is “a black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.”

We demonstrate pairwise testing on a simple example below:

EXAMPLE: CONFIGURATION TESTING

Consider a system under test with 4 parameters, A, B, C, D. These parameters have the following values:

• Parameter A: a1, a2, a3, a4,
• Parameter B: b1, b2, b3,
• Parameter C: c1, c2,
• Parameter D: d1, d2, d3, d4.

Analyze the possible combinatorial testing techniques.

With the 16 test cases in the table below pairwise test selection criterion is satisfied.

TestABCD
1a1b1c1d1
2a1b2c2d2
3a2b2c1d1
4a2b1c2d2
5a3b3c2d1
6a3b3c1d2
7a4b1c1d3
8a4b2c2d4
9a1b3c2d3
10a2b3c1d4
11a3b2-d3
12a3b1-d4
13a4b3-d1
14a1--d4
15a2--d3
16a4--d2

In the table ‘-’ denotes the “do not care” symbol. Please check that all data value pairs can be found in a test case, e.g. a2-d4 pair is in Test 10. Even this simple combinatorial method is not linear, leading usually many test cases. However, by applying more simple techniques, such as equivalent partitioning with boundary value analysis may result in weak quality. What can we do? The solution is combinative testing introduced in Practical Test Design.

The rationale behind the combinative technique is to assure that for a single parameter each computation is calculated in more than one context. The basic variant of combinative testing is called diff-pair testing. Informally, it requires that each value p of any parameter P be tested with at least two different values q and r for any other parameters. For example, if the parameter P has two possible values, let us say x and y, and there are two more parameters Q and R with values: Q: (1, 2) and R: (Y, N), then we have to test the value pairs (x, 1), (x, 2), (x, Y), (x, N). Thus, the test suite required for testing parameter P contains the data combinations {(x, 1, Y), (x, 2, N)} or {(x, 1, N), (x, 2, Y)}. Similarly, for y, we test {(y, 1, Y), (y, 2, N)} or {(y, 1, N), (y, 2, Y)}.

We test the values of the other parameters R and Q similarly.

Considering our configuration testing example, we need only 8 test cases:

Test case Nr.ABCD
1a1b1c1d1
2a1b2c2d2
3a2b1c2d3
4a2b2c1d4
5a3b1c1d2
6a3b3c2d1
7a4b3c1d3
8a4b1c2d4

You can see that a1 is tested with (b1, b2), (c1, c2) and (d1, d2). Similar is the case for a2, a3 and a4. b1 is tested with (a1, a2), (c1, c2) and (d1, d3). You can easily see that there are pairs with different values for all the parameters.

In our book we proved that this method is linear, i.e. having N parameters with M values each, the number of test cases will be 2×Max(N, M). This is exactly what we need. We test all the parameters with different values of all other parameters. Thus, we combine testing without sacrificing linearity. In our book we showed that combinative testing is very efficient both for computation and control-flow errors.

## 107 thoughts to “Combinative testing – make your test design efficient”

1. Ѕtunning quest there. What happened after? Thanks!

2. Authors says:

Thanks for commenting.
The first step is to introduce the book to the world (new ideas, techniques, approaches).
The second is to work out the practical exercises in this page.
We thanks for any support and suggestion.

3. Hello, its good piece of writing regarding media
print, we all know media is a impressive source of facts.

4. whoah this weblog is great i like studying your articles. Keep up the great paintings! You already know, a lot of people are searching around for this info, you could help them greatly.

5. Some genuinely wonderful articles on this website, regards for contribution.

6. After study a handful of the content on your web site now, and that i really like your way of blogging. I bookmarked it to my bookmark website list and will also be checking back soon. Pls look at my internet site too and told me what you think.

7. This is really interesting, You are a very skilled blogger.
I’ve joined your feed and look forward to seeking more of
your magnificent post. Also, I’ve shared your site in my social networks!

Thank you.

9. I’m very happy to uncover this great site. I wanted to thank you for your time for this wonderful read!! I definitely liked every little bit of it and I have you saved to fav to see new information in your web site.

10. I love it when individuals get together and share opinions. Great site, stick with it.

11. That is a good tip particularly to those new to the blogosphere. Brief but very accurate information… Thank you for sharing this one. A must read article!

12. Excellent article. I’m facing some of these issues as well..

13. After looking at a handful of the articles on your web page, I truly like your way of writing a blog. I saved it to my bookmark webpage list and will be checking back in the near future. Please check out my website as well and tell me your opinion.

14. You are so cool! I do not believe I have read through anything like this before. So good to find someone with unique thoughts on this subject matter. Seriously.. many thanks for starting this up. This site is something that’s needed on the internet, someone with some originality.

15. This is a topic which is near to my heart… Many thanks! Exactly where can I find the contact details for questions?

16. I blog quite often and I truly appreciate your information. This article has really peaked my interest. I will book mark your website and keep checking for new information about once per week. I opted in for your RSS feed too.

17. An outstanding share! I have just forwarded this onto a co-worker who has been conducting a little research on this. And he in fact ordered me breakfast due to the fact that I discovered it for him… lol. So let me reword this…. Thanks for the meal!! But yeah, thanx for spending time to discuss this matter here on your blog.

18. This blog was… how do you say it? Relevant!! Finally I have found something that helped me. Cheers!

19. I was very happy to find this site. I wanted to thank you for ones time for this particularly wonderful read!! I definitely enjoyed every little bit of it and i also have you book-marked to look at new information on your blog.

20. I was pretty pleased to find this web site. I want to to thank you for ones time just for this wonderful read!! I definitely enjoyed every little bit of it and I have you bookmarked to look at new stuff on your website.

21. Excellent post! We are linking to this particularly great content on our website. Keep up the good writing.

22. I must thank you for the efforts you’ve put in writing this site. I really hope to see the same high-grade blog posts from you in the future as well. In truth, your creative writing abilities has motivated me to get my own, personal website now 😉

23. Spot on with this write-up, I actually believe that this site needs much more attention. I’ll probably be returning to read more, thanks for the information.

24. Your style is unique in comparison to other folks I have read stuff from. Many thanks for posting when you’ve got the opportunity, Guess I’ll just book mark this page.

25. I really like your article. It’s evident that you have a lot knowledge on this topic. Your points are well made and relatable. Thanks for writing engaging and interesting material.

26. Good site! I really love how it is simple on my eyes and the data are well written. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which must do the trick! Have a great day!

27. I absolutely love your site.. Great colors & theme. Did you develop this site yourself? Please reply back as I’m hoping to create my own personal blog and would love to find out where you got this from or just what the theme is named. Kudos.

28. I am lucky that I found this web blog, precisely the right information that I was searching for!

29. Your style is really unique in comparison to other folks I’ve read stuff from. Thanks for posting when you have the opportunity, Guess I will just book mark this blog.

30. Wonderful short article. Would like to know if you are mosting likely to add any added details concerning this in a later message?

31. A motivating discussion is worth comment. I think that you ought to write more about this topic, it might not be a taboo matter but generally people don’t talk about such topics. To the next! Cheers!

32. An outstanding share! I have just forwarded this onto a colleague who had been doing a little research on this. And he actually ordered me dinner due to the fact that I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to talk about this issue here on your blog.

33. This is a topic that is near to my heart… Many thanks! Exactly where can I find the contact details for questions?

34. I blog quite often and I genuinely thank you for your content. This great article has really peaked my interest. I’m going to bookmark your blog and keep checking for new information about once per week. I opted in for your RSS feed too.

35. Good info. Lucky me I discovered your site by accident (stumbleupon). I’ve book marked it for later.

36. You’re so cool! I don’t think I have read something like this before. So wonderful to discover someone with a few original thoughts on this subject. Seriously.. thanks for starting this up. This website is something that is needed on the web, someone with a bit of originality.

37. Good post. I learn something totally new and challenging on blogs I stumbleupon on a daily basis. It will always be interesting to read through content from other writers and use a little something from their sites.

38. After looking into a handful of the blog articles on your website, I truly like your way of writing a blog. I bookmarked it to my bookmark webpage list and will be checking back in the near future. Please check out my website as well and tell me what you think.

39. This is a topic that is near to my heart… Best wishes! Exactly where are your contact details though?

40. Very good post. I’m experiencing some of these issues as well..

41. You need to take part in a contest for one of the finest sites on the net. I most certainly will highly recommend this site!

42. You’re so awesome! I don’t think I’ve read something like this before. So nice to discover someone with a few original thoughts on this issue. Really.. thanks for starting this up. This web site is something that is needed on the internet, someone with a bit of originality.

43. Hi there! I simply wish to give you a huge thumbs up for your great info you have here on this post. I’ll be returning to your blog for more soon.

44. May I simply just say what a comfort to find somebody who really knows what they’re talking about on the internet. You actually understand how to bring an issue to light and make it important. A lot more people must check this out and understand this side of the story. I can’t believe you are not more popular since you most certainly possess the gift.

46. Great article. I’m experiencing a few of these issues as well..

47. Having read this I believed it was rather informative. I appreciate you spending some time and energy to put this short article together. I once again find myself personally spending a significant amount of time both reading and commenting. But so what, it was still worthwhile!

49. This blog was… how do you say it? Relevant!! Finally I’ve found something which helped me. Thanks a lot.

50. Absolute stimulating knowledge that you have remarked, thanks a lot for writing.

51. I just have to inform you that I am new to putting up a blog and certainly adored your post. Quite possibly I am going to remember your blog post . You really have memorable article blog posts. Acknowledge it for swapping with us all of your blog write-up

52. I blog often and I truly appreciate your information. This great article has really peaked my interest. I will take a note of your blog and keep checking for new details about once per week. I opted in for your RSS feed too.

53. This blog was… how do I say it? Relevant!! Finally I’ve found something which helped me. Thank you!

54. An interesting discussion is worth comment. I believe that you should write more about this subject, it may not be a taboo subject but generally people do not speak about such issues. To the next! All the best!

55. It is truly a nice and useful piece of info. I am satisfied that you just shared this useful information with us. Please keep us informed like this. Thanks for sharing.

56. I like what you guys are up too. Such clever work and reporting! Keep up the excellent works guys I’ve incorporated you guys to my blogroll. I think it’ll improve the value of my web site :).

57. I want to to thank you for this wonderful read!! I absolutely loved every little bit of it. I’ve got you saved as a favorite to check out new stuff you post…

58. Very interesting subject, regards for putting up.

59. You should be a part of a contest for one of the finest sites on the web.
I most certainly will highly recommend this blog!

60. Good post. I learn something totally new and challenging on blogs I stumbleupon every day. It’s always interesting to read through articles from other writers and use something from other sites.

61. What i don’t realize is actually how you’re not actually much more smartly-liked than you might be right now. You are so intelligent. You realize therefore significantly on the subject of this topic, produced me for my part imagine it from a lot of numerous angles. Its like men and women don’t seem to be interested unless it is something to do with Girl gaga! Your own stuffs outstanding. Always deal with it up!

62. Hello! I just would like to offer you a big thumbs up for the great info you have got here on this post. I am coming back to your web site for more soon.

63. Greetings! Very useful advice within this post! It’s the little changes that will make the largest changes. Thanks for sharing!

64. Very good article! We will be linking to this great post on our site. Keep up the great writing.

65. I have been browsing online more than 3 hours today, yet I by no means discovered any interesting article like yours. It’s beautiful worth enough for me. In my view, if all web owners and bloggers made excellent content material as you probably did, the internet can be much more useful than ever before. “No nation was ever ruined by trade.” by Benjamin Franklin.

66. I really like reading through an article that can make men and women think. Also, many thanks for permitting me to comment.

67. This site was… how do you say it? Relevant!! Finally I have found something which helped me. Thank you!

68. Everyone loves it when individuals come together and share opinions. Great blog, keep it up!

69. I blog often and I truly thank you for your information. This article has truly peaked my interest. I’m going to bookmark your blog and keep checking for new information about once a week. I opted in for your RSS feed too.

71. Greetings! Very useful advice in this particular post! It is the little changes that produce the most important changes. Thanks for sharing!

72. I’m more than happy to discover this site. I wanted to thank you for your time due to this fantastic read!! I definitely enjoyed every bit of it and i also have you saved as a favorite to check out new stuff in your blog.

73. Way cool! Some very valid points! I appreciate you writing this write-up and also the rest of the site is also very good.

74. It really is almost extremely difficult to see well-advised users on this area, still you appear like you comprehend what you’re revealing! Thank You

75. Aw, this was a very good post. Spending some time and actual effort to generate a great article… but what can I say… I procrastinate a lot and never manage to get anything done.

76. I have realized some important matters through your site post. One other stuff I would like to say is that there are various games available and which are designed especially for toddler age young children. They include pattern identification, colors, pets, and forms. These commonly focus on familiarization as an alternative to memorization. This helps to keep a child engaged without having the experience like they are studying. Thanks

77. When I initially commented I appear to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I receive 4 emails with the exact same comment. Is there a way you are able to remove me from that service? Appreciate it.

78. I simply intend to tell you that I am new to writing a blog and certainly loved your webpage. Most likely I am prone to bookmark your blog post . You indeed have lovely article material. Be Grateful For it for share-out with us all of your web information

79. Hi here, just turned aware about your blog through The Big G, and found that it is genuinely informative. I’ll like should you decide continue on this idea.

80. You’ll find it near unthinkable to see well-advised women and men on this subject, fortunately you look like you know the things you’re preaching about! Bless You

81. Cheap Replica Handbags You guide me by sharing this toolbox i face problem whenever i exploit this promotion reading your blog i am able to easily use.

82. Ultimately, a problem that I’m passionate about. I have looked for data of this caliber for the previous various hours. Your site is greatly appreciated.

83. Heya iâ€™m for the first time here. I found this board and I to find It truly helpful & it helped me out a lot. I hope to provide something back and aid others such as you helped me.

84. Its such as you read my mind! You appear to understand a lot about this, like you wrote the e-book in it or something. I believe that you simply could do with a few to force the message house a bit, however other than that, this is wonderful blog. A great read. I’ll certainly be back.

85. Nice site. On your blogs very interest and i will tell a friends.

86. I am lucky that I found this web blog, precisely the right information that I was searching for!

87. I simply wanted to thank you a lot more for your amazing website you have developed here. It can be full of useful tips for those who are actually interested in this specific subject, primarily this very post. Your all so sweet in addition to thoughtful of others and reading the blog posts is a great delight in my opinion. And thats a generous present! Dan and I usually have enjoyment making use of your recommendations in what we need to do in the near future. Our checklist is a distance long and tips will certainly be put to excellent use.

88. Really nice style and design and excellent content , nothing at all else we need : D.

89. Good site! I really love how it is simple on my eyes and the data are well written. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS which must do the trick! Have a great day!

90. Exceptional entry! I found it very interesting. I’ll check back later to see if more posts are added.

91. I would like to convey my admiration for your generosity in support of men and women that have the need for help with this particular concern. Your special dedication to getting the message all over had been wonderfully productive and have all the time made professionals much like me to attain their dreams. Your own invaluable tutorial means a great deal to me and additionally to my office workers. Thank you; from everyone of us.

92. I simply wanted to thank you a lot more for your amazing website you have developed here. It can be full of useful tips for those who are actually interested in this specific subject, primarily this very post. Your all so sweet in addition to thoughtful of others and reading the blog posts is a great delight in my opinion. And thats a generous present! Dan and I usually have enjoyment making use of your recommendations in what we need to do in the near future. Our checklist is a distance long and tips will certainly be put to excellent use.

93. Iâ€™d should talk to you here. Which is not some thing I do! I quite like reading a post which will make people believe. Also, many thanks permitting me to comment!

94. I have to convey my respect for your kindness for all those that require guidance on this one field. Your special commitment to passing the solution up and down has been incredibly functional and has continually empowered most people just like me to achieve their dreams. Your amazing insightful information entails much to me and especially to my peers. Thanks a ton; from all of us.

95. Nice site, nice and easy on the eyes and great content too. Do you need many drafts to make a post?

96. It really is near extremely difficult to find well-aware readers on this subject, nevertheless you appear like you fully understand whatever you’re writing about! With Thanks

97. I wanted to say Appreciate providing these details, youre doing a great job with the site…

98. Some really superb info , Sword lily I found this.

99. Thanks for making the honest attempt to speak about this. I believe very robust approximately it and want to read more. If itâ€™s OK, as you gain more in depth wisdom, would you thoughts adding extra articles similar to this one with additional information? It might be extremely useful and useful for me and my friends.

100. Nice post. I learn something more challenging on distinct blogs everyday. It will always be stimulating to read content off their writers and practice a little something from their store. Iâ€™d choose to use some with all the content in my small weblog whether you do not mind. Natually Iâ€™ll provide a link on your own internet weblog. Many thanks sharing.

101. Some times its a pain in the ass to read what people wrote but this web site is very user friendly ! .

102. It truly is almost unthinkable to encounter well-aware users on this subject, and yet you look like you understand those things you’re preaching about! Appreciate It

103. Wonderful article. I learn something more challenging on different blogs everyday. It will certainly always be stimulating to review material from other authors and also practice a little something from their shop. I?d prefer to use some with the material on my blog whether you don?t mind. Natually I?ll offer you a link on your internet blog. Thanks for sharing.

104. Ultimately, a problem that I’m passionate about. I have looked for data of this caliber for the previous various hours. Your site is greatly appreciated.

105. I really intend to share it with you that I am new to online blogging and utterly adored your work. Quite possibly I am probably to save your blog post . You absolutely have excellent article blog posts. Appreciate it for giving out with us your main website report

106. I simply want to tell you that I am just all new to blogging and honestly savored you’re web blog. Very likely I’m planning to bookmark your site . You really have remarkable writings. Thanks for revealing your blog site.