This document discusses combinatorial software test design methods for selecting a small number of tests that achieve good coverage of interactions between test inputs. It explains that structured variation between tests is important to explore different areas of the software. Combinatorial testing uses algorithms to calculate the smallest number of tests needed to cover all target combinations of test inputs, like all pairs of values. This method can find many bugs with fewer tests than an unstructured approach.

1. Combinatorial
Software Test
Design
(Beyond
Pairwise
Testing)
1

2. Presented at STPCon
October, 2010
By Justin Hunter
CEO of Hexawise
Cover slide photo credit: "Mike" Michael L. Baird, flickr.bairdphotos.com
http://www.flickr.com/photos/mikebaird/2127310513
2

3. L E M
O B
P R
3

4. The Problem
You can’t test
everything.
http://www.flickr.com/photos/maisonbisson/109211670/ 2

5. The Problem
“With so many
possible tests to
choose from...”
http://www.flickr.com/photos/madabandon/344158061/
2

6. The Problem
“Things might
seem
uncontrollable...”
http://www.flickr.com/photos/maisonbisson/110867288/sizes/l/in/photostream/
2

7. The Problem
... discouraging,
even.
http://www.flickr.com/photos/cheo70/2702682262/ 2

8. The Problem
So, how can you select
just a few tests...
http://www.flickr.com/photos/krawcowicz/3830467093/
8

9. The Problem
... and still
achieve
adequate
coverage?
http://www.flickr.com/photos/editor/4861496460/ 9

10. The Problem
Wait. So this is a test design method to
achieve greater coverage in fewer tests. Right?
http://www.flickr.com/photos/johncarleton/4890239248
10

11. Not Just Fewer, Better Tests
If you prioritize But for those testers who
speed, yes. You prioritize thoroughness,
can achieve this method can also be
greater coverage used to increase
in fewer tests. thoroughness dramatically.
It can help you
prioritize around
http://www.flickr.com/photos/legofenris/4212007962/
either goal.
11

12. Not Just Fewer, Better Tests
W ay The
choice is
fewer yours to
make.
tes ts!
etter
ligh tly b
ith s than we
(W
rage
cove e now.)
hav
http://www.flickr.com/photos/thomashawk/2402194148 http://www.flickr.com/photos/thomashawk/2402190314
12

13. Not Just Fewer, Better Tests
How can you create “curiously
strong” test cases that will allow
you to achieve either of these
testing objectives?
http://www.flickr.com/photos/photobunny_earl/2309612944/
13

14. I O N
L UT
S O
14

15. The Solution
This presentation describes an
approach that will help you create
powerful test cases.
http://www.flickr.com/photos/maisonbisson/110867008/
15

16. Structured Variation
To do so, use
combinatorial
test design
methods that
build
structured
variation
into your test
cases.
16

17. Structured Variation?
“Test One”
http://www.flickr.com/photos/gelrodandjodie/148759732
17

18. Structured Variation?
This is not what I mean by structured
variation.
“Test Two”
http://www.flickr.com/photos/gelrodandjodie/148759732
18

19. Structured Variation?
This is not what I mean by structured
variation.
“Test Three”
http://www.flickr.com/photos/gelrodandjodie/148759732
19

20. Structured Variation?
So if that’s not structured
variation, what is?
20

21. Structured Variation
Structured
variation
means
http://www.flickr.com/photos/tk_five_0/2281768728 21

22. Structured Variation
...each test
will be
different
http://www.flickr.com/photos/criben/2635618391
22

23. Structured Variation
...than the
tests before
it, and
http://www.flickr.com/photos/horrgakx/3186982135 23

24. Structured Variation
...testers
will do their
testing in
different
ways,
http://www.flickr.com/photos/pixelrobber/2626213081 24

25. Structured Variation
... and
explore
from
more
angles.
http://www.flickr.com/photos/tk_five_0/2286693552 25

26. Structured Variation
“OK. Maximum variation is better than redundantly
repeated redundant repetition. Got it.
That it? Am I good to go now?”
http://www.flickr.com/photos/johncarleton/4890241530 26

27. Coverage
No.
There’s
more.
Much more.
Remember
coverage?
http://www.flickr.com/photos/30431232@N03/3610846363
27

28. Coverage
Accidentally
leaving gaps
in coverage
can lead to
painful
results.
http://www.flickr.com/photos/dwfromthepeg/240921271/
28

29. Coverage
Structured
variation
focuses on
covering as
much as
possible...
http://www.flickr.com/photos/localsurfer/38245658/
29

30. Coverage
... with as
few
resources
as
possible...
http://www.flickr.com/photos/mikebaird/4177570453/
30

31. Coverage
...and in the
most
efficiently
structured
way possible.
http://www.flickr.com/photos/dannohung/332507807
31

32. F
O &
S
U N
C IO
O T
F U L E
L ONA
O I
S T
R A 32

33. Single-Mode Faults
Most software defects
in production today
can be triggered by
just one test input.
http://www.flickr.com/photos/vaguelyartistic/292083492
33

34. Dual-Mode Faults
Software bugs caused
by the interaction of
two test inputs are
very common too.
http://www.flickr.com/photos/kanaka/1798327442
34

35. Triple-Mode Faults
Defects that require three specific test
inputs to trigger them are rare.
http://www.flickr.com/photos/kanaka/1798327442
35

36. Quadruple-Mode Faults
Defects that require four more test inputs to
trigger them are extremely rare.
http://www.flickr.com/photos/novembering/3816524907
36

37. Implication on Testing Strategy
Aha! So, to find bugs efficiently, we should
focus first on testing all possible
combinations every set of TWO test inputs!
http://www.flickr.com/photos/johncarleton/4890236762 37

38. Implication on Testing Strategy
That’s right, two.
Focus on covering all
possible two-way
defects first. In fact,
that’s so important,
can you say it a
second time?
http://www.flickr.com/photos/soulsurfingcrew/2273300213
38

39. Implication on Testing Strategy
To find bugs efficiently we should focus
first on testing
all possible combinations of TWO test inputs!
http://www.flickr.com/photos/johncarleton/4890236762 39

40. I C S
AN
E C H
M
40

41. Combinations of Test Inputs
The key
words here
are test
inputs and
combinations.
http://www.flickr.com/photos/12tribu/2750040244
41

42. Combinations of Test Inputs
Test inputs
will vary
dramatically
from project
to project.
42

43. Combinatorial Software Test Design
Well, duh!
They would,
wouldn’t they?
But how does
it all work?
http://www.flickr.com/photos/johncarleton/4377591743
43

44. Combinatorial Software Test Design
Here’s how it works: First, people
are required to identify what test
inputs should be tested.
http://www.flickr.com/photos/mikebaird/3939231212
44

45. Combinatorial Software Test Design
Second, computer algorithms will take those
inputs and quickly calculate the smallest
possible number of tests required to meet the
tester’s specified coverage objectives.
45

46. Combinatorial Software Test Design
... and again, the
algorithms will
W ay create different
solutions
fewer depending upon
tes ts! your testing
objectives.
etter
ligh tly b
ith s than we
(W
rage
cove e now.)
hav
http://www.flickr.com/photos/thomashawk/2402194148 http://www.flickr.com/photos/thomashawk/2402190314
46

47. Combinatorial Software Test Design
W ay The woman
wanting fewer
fewer tests should
tes ts! create 2-way
tests
r
bette
htly
Wi
ig
th sl n we ha
ve (AKA
( ha
age t .)
c over now “pairwise” or
“AllPairs” tests)
http://www.flickr.com/photos/thomashawk/2402194148
47

48. Combinatorial Software Test Design
This guy, wanting more
coverage, should create
more thorough
combinatorial tests.
(e.g., 3-way, 4-way, or
multi-strength tests).
http://www.flickr.com/photos/thomashawk/2402190314
48

49. OT
Y N STS
H
W NT ? E
S G ND
I A
E
D YH
B 49

50. Advantages of Combinatorial Software Test Design
Test Design: Man vs. Machine
(Once test inputs are known)
Speed: Coverage: Efficiency:
Human Weeks Gaps Way More
Tests
Computer Seconds 100%* Way Fewer
Tests
*100% of what? 100% of the desired test input combinations (e.g., all pairs of values for 2-way tests solutions, all triplets of possible
values for 3-way solutions, etc.).

51. Coverage Advantages
Whatever
your test
inputs,
combinatorial
testing makes
it easy and
fast to cover
all of the
specific
combinations
of test inputs
that you
suspect are
relatively
important.
http://www.flickr.com/photos/kanaka/1798327442 51

52. Advantages of Combinatorial Software Test Design
OK, but wouldn’t I have tested
those relatively important
combinations on my own?
http://www.flickr.com/photos/johncarleton/4890239248

53. Advantages of Combinatorial Software Test Design
Perhaps, but maybe you...
Forget Run out of time
Use more tests Accidentally
than needed repeat yourself
53

54. Coverage Advantages
Plus, using combinatorial testing, you’ll
also get additional coverage from combinations
that wouldn’t have made your list, including...
http://www.flickr.com/photos/tk_five_0/2706278432 54

55. Coverage Advantages
...specialized use cases,
No, its way
cooler than
pareto.
http://www.flickr.com/photos/subspace-eddy/2766810770/
55

56. Coverage Advantages
... unexpected
combinations,
http://www.flickr.com/photos/funky64/2875860978
56

57. Coverage Advantages
...notable,
combinations
that capture
your attention,
http://www.flickr.com/photos/brb_photography/3221049399
57

58. Coverage Advantages
...even combinations that
rarely, if ever, actually occur,
http://www.flickr.com/photos/grassvalleylarry/5536758
58

59. Coverage Advantages
... and...
combinations
that will take
you into
different
parts of the
application
(and different
states) than
you would
have
otherwise
thought to
explore.
http://www.flickr.com/photos/28041049@N00/227352411
59

60. Coverage Advantages
Quickly testing
unusual pairs is
often
surprisingly
efficient and
effective. Many
unexpected
two-way
interactions
cause
problems.
60

61. Constraint Handling
Combinations
that can’t ever
appear in reality?
Yes. No problem. You
can prevent them
from appearing in
your plans with
“Invalid Pairs.”
http://www.flickr.com/photos/kaptainkobold/4741725788/lightbox/
61

62. E T ?
Y Y
E AD
R
62

63. Ready?
I’m ready
already!
Let’s do
this!
http://www.flickr.com/photos/johncarleton/4890241530
63

64. Hazards
Not so fast.
Hazards exist.
To succeed,
you must use
common
sense.
http://www.flickr.com/photos/christianhaugen/3604106558
64

65. Limitations
And let’s
not kid
ourselves.
Not just
anyone
can do
http://www.flickr.com/photos/mikebaird/573368639
this stuff.
65

66. Prerequisites
It also requires
good critical
thinking skills.
http://www.flickr.com/photos/johncarleton/9511759/
66

67. Prerequisites
To thrive,
you must
be able to
develop
models
about
how the
systems
you’re
exploring
http://www.flickr.com/photos/10ch/3101275536/
67 work.

68. Practice Makes Perfect
Training
and practice
will help
you
succeed.
http://www.flickr.com/photos/mikebaird/2128091146

69. Questions
?
http://www.flickr.com/photos/nathaninsandiego/3647136579
69

70. Ready?
Let’s get
started!
http://www.flickr.com/photos/mikebaird/2996548709
70

71. D I X
P E N
AP
http://www.flickr.com/photos/dannohung/332507807
71

72. Thank you
Special thanks to the folks at:
and its talented contributing
photographers, including: Mike Baird
... and the inimitable John Carleton
(who, let the record show, had no input into the cartoonish
statements we added next to his striking expressions).
72

73. Why 2-way Testing is Effective
Multiple thorough studies show approximately 85% of
defects in production could have been detected by
simply testing all possible pairs of values.
Text
Source: “Combinatorial Testing” IEEE Computer, Aug, 2009. Rick Kuhn, Raghu Kacker, Jeff Lei, Justin Hunter

74. Additional Information
Excellent introductory articles and instructional videos: www.CombinatorialTesting.com
Results of a ten-project study of effectiveness:
https://www.hexawise.com/Combinatorial-Softwar-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf
Free trials of our firm’s combinatorial test design tool: www.hexawise.com/users/new (which stay free
indefinitely for companies using 1-4 licenses).
Also, please feel free to contact me if you have any questions. I’d be happy to quickly review a test
plan or two, answer your questions, give quick pointers to help you run a pilot, etc. Seriously.... I enjoy
helping people get started with this test design approach. Please don’t hesitate to reach out. There’s
no charge and no catch.
74

75. Practice Tips: 4-Step Process
Four-Step Process to Design Efficient and Effective Tests
Scope Level
of
Detail Passive
Field
Treatment
1.
Plan Be
clear
about
single
or
mul3ple: Acceptable
op3ons:
Dis3nguish
between
important
fields
(par3cularly
those
that
will
trigger
business
rules)
and
unimportant
fields
in
the
-­‐
Features
/
Func3ons
/
Capabili3es -­‐
High
level
“search
for
something” applica3on.
-­‐
User
types -­‐
Medium
level
“search
for
a
book”
Quickly
document
what
your
approach
will
be
towards
-­‐
Business
Units -­‐
Detailed
“search
for
‘Catcher
in
the
Rye’
passive
fields.
You
might
consider:
ignore
them
(e.g.,
don’t
-­‐
H/W
or
S/W
Configura3ons
by
its
3tle” select
any
Values
in
your
plan)
or
a
3
Value
approach
such
as
“valid”
“Invalid
(then
fix)”
and
“Blank
(then
fix)”
Configura3ons Users Ac3ons
2.
Create First
add
hardware
configura3ons Next,
add
mul3ple
types
of
users
(e.g.,
Start
with
Big
Common
Ac3ons
made
by
users
administrator,
customer,
special
customer) AZer
comple3ng
Big
Common
Ac3ons,
circle
back
and
add
Next
add
soZware
configura3ons
Consider
permission
/
authority
levels
of
admin
Small
Ac3ons
and
Excep3ons
users
as
well
as
business
rules
that
different
Remember
some
ac3ons
may
be
system-­‐generated
users
might
trigger
Business
Rules Gap
Filling Itera3on
3.
Refine Select
Values
to
trigger
bus.
rules Iden3fy
gaps
by
analyzing
different
states,
Refine
longest
lists
of
Values;
reduce
their
numbers
by
using
equivalence
classes,
etc.
orders
of
opera3ons,
decision-­‐tree
outcomes
Iden3fy
equivalence
classes sought
vs.
delivered
by
tests,
“gap
hun3ng”
Create
Tests
with
and
w/out
borderline
Values;
consider
cost/
conversa3ons
w/
SME’s,
etc. benefit
tradeoffs
of
addi3onal
test
design
refinements
Test
for
boundary
values
Fill
gaps
by
either
(i)
adding
Parameters
and/or
Consider
stopping
tes3ng
aZer
reaching
~80%
coverage
Mark
constraints
/
invalid
pairs
Values
or
(ii)
crea3ng
“one-­‐off”
tests.
Consider
2-­‐way,
3-­‐way,
and
Mixed-­‐Strength
op3ons
Auto-­‐Scrip3ng Expected
Results Con3nuous
Improvement
4.
Execute
Add
auto-­‐scrip3ng
instruc3ons
once;
Export
the
tests
into
Excel
when
you’re
done
If
possible
measure
defects
found
per
tester
hour
“with
and
without
Hexawise”
and
share
the
results
apply
those
instruc3ons
to
all
of
the
tests
itera3ng
the
plan
in
your
plan
instantly Add
inputs
based
on
undetected
defects
Add
complex
Expected
Results
in
Excel
post-­‐
Don’t
include
complex
Expected
Results
export Share
good,
proven,
plan
templates
with
others
in
auto-­‐scripts
© Hexawise, 2010. All rights reserved.
26

76. Practice Tips: Warning Signs
“You might be headed for trouble if...”
Scope Level
of
Detail Passive
Field
Treatment
1.
Plan ...
You
cannot
clearly
describe
both
the
...
Parameters
with
the
most
Values
have
more
...
You
cannot
clearly
describe
your
strategy
to
deal
with
scope
of
your
test
plan
and
what
will
be
Values
than
they
require.
365
values
for
“days
unimportant
details.
If
Values
will
impact
business
rules,
focus
leZ
out
of
scope. of
the
year”
is
bad.
Instead,
use
equivalence
on
them.
If
Values
don’t
impact
business
rules,
consider
class
Values
like
“weekend”
&
“weekday.”
When
ignoring
them.
in
doubt,
choose
more
Parameters
and
fewer
Values.
Configura3ons Users Ac3ons
2.
Create ...
You
have
ignored
hardware
and
...
You
have
not
included
all
the
different
types
...
You
start
entering
Small
Ac3ons
(e.g.,
“search
for
a
soZware
configura3ons
without
first
of
users
necessary
to
trigger
different
business
hardback
science
book
by
author
name”)
before
you
enter
Big
confirming
this
approach
with
rules.
What
user
types
might
create
different
Ac3ons
(e.g.,
“Put
Something
in
Cart.
Buy
it.”)
First
go
from
stakeholders. outcomes?
Authority
level?
Age?
Loca3on?
beginning
to
end
at
a
high
level.
AZer
you’ve
done
that,
feel
Income?
Customer
status? free
to
add
more
details.
Business
Rules Gap
Filling Itera3on
3.
Refine ...
You
forget
to
iden3fy
invalid
pairs.
-­‐
...
You
assume
that
the
test
condi3ons
coming
...
You
forget
to
look
at
the
Coverage
Analysis
charts.
If
you
or
-­‐
...
You
rely
only
on
Func3onal
out
of
Hexawise
will
be
100%
of
the
tests
you
achieve
80%
coverage
in
the
first
quarter
of
the
tests,
you
Requirements
and
Tech
Specs
w/out
should
run.
There
might
well
be
addi3onal
should
measure
the
cost/benefit
implica3ons
of
execu3ng
the
thinking
hard
yourself
and
asking
“one-­‐off”
things
that
you
should
test
and/or
a
last
3/4
of
the
tests.
ques3ons
to
SME’s
about
business
rules
few
nega3ve
tests
to
design
by
hand.
and
outcomes
that
are
not
yet
triggered.
Auto-­‐Scrip3ng Expected
Results Con3nuous
Improvement
4.
Execute ...
You
add
detailed
Expected
Results
in
...
You
invest
a
lot
of
3me
in
calcula3ng
and
...
You
don’t
ask
(when
defects
that
the
tests
missed
are
found
the
tests.
-­‐
documen3ng
Expected
Results
before
you
have
post-­‐tes3ng)
“What
input
could
have
been
added
to
the
test
or
-­‐
...
You
forget
that
this
feature
exists
plan
to
detected
this?”
“Should
I
add
that
input
to
the
and
find
yourself
typing
out
test-­‐by-­‐test
determined
your
“final
version”
Parameters
and
instruc3ons
one-­‐by-­‐one. Values.
Last
minute
addi3ons
to
inputs
will
Hexawise
test
plan
now
to
improve
it
in
advance
of
the
next
jumble
up
test
condi3ons
for
most
test
cases. 3me
it
is
used?”
© Hexawise, 2010. All rights reserved.
27