TestFu has now limited support for generate test suites using Combinatorial Testing as the TestFu.Operations namespace. Combinatorial Testing is an interresting testing technique that has been studied a quite some time now (see citeseer search). There already exists a few package that provide tools to generate test suites (see jenny, allpairs) but none in the .Net framework.

Combinatorial Testing description and Glossary

In this chapter, I will define the different "actors" that interact in the combinatorial testing. I will link them to their corresponding interface in TestFu.

• A domain (IDomain) is a finite set of objects.
• Ex: {1,2,3} is a domain composed of the integers 1,2,3
• The boundary of a domain D is the set of objects in D that are at the boundary of the domain.  The boundary is also a domain,
• If we decide that the first and last element of an array is it's domain, {1,3} is the boundary of {1;2,3}
• A tuple (ITuple) acts as a container for other objects,
• A product on a domain collection (IDomainCollection) is an algorithm (ITupleEnumerable) that generate a suite of tuples (ITupleEnumerator), created by combining one element of each domain, using a predefined strategy.

Currently, the only product available is the cartesian product which returns an exhaustive enumeration of the all possible combination. It is well-known that this product is not efficient at all covering the tests because of the exponential explosion of the number of suites, better strategy have been developped such as the PairWaise covering (all pairs), t-wize, etc... I haven't had the time to look at those yet.

Let's see some code

The TestFu.Operations namespace is designed to be simpler but no simpler. Let's start by defining some domains:

int[] array1 = { 1, 2 };
string[] array2 = { "a", "combinatorial", "hello", "world" };
char[] array3 = { 'a', 'b', 'c' };

Note: TestFu.Operations currently supports arrays, collection (ICollection), enumerable collection (IEnumerable) or singleton domain (1 element). Of course, this is totally extendible. You could easily think about a domain reading the rows of a DataTable, or the elements of a XmlDocument

The static class Products contains a rich set of methods that can create the different products. Let's start with the cartesian product of array1 x array2:

int i = 1;
foreach (ITuple tuple in Products.Cartesian(array1, array2))
{
    Console.WriteLine("\t{0}: {1}",i++,tuple);
}

-- output
        1: 1, a
        2: 1, combinatorial
        3: 1, hello
        4: 1, world
        5: 2, a
        6: 2, combinatorial
        7: 2, hello
        8: 2, world

As expected, each returned tuple is a collection of element from each domain. We could also decide to test the boundaries of the arrays only (first and last element each time):

i = 1;
foreach (ITuple tuple in Products.BoundaryCartesian(array1, array2))
{
    Console.WriteLine("\t{0}: {1}", i++, tuple);
}

-- output
        1: 1, a
        2: 1, world
        3: 2, a
        4: 2, world

The products can act on an arbitrary number of domains so we can add array3 in the product (don't forget about the explosion of the tests number!):

i = 1;
foreach (ITuple tuple in Products.Cartesian(array1, array2, array3))
{
    Console.WriteLine("\t{0}: {1}", i++, tuple);
}

-- output
        1: 1, a, a
        2: 1, a, b
        3: 1, a, c
        4: 1, combinatorial, a
        5: 1, combinatorial, b
        6: 1, combinatorial, c
        7: 1, hello, a
        8: 1, hello, b
        9: 1, hello, c
        10: 1, world, a
        11: 1, world, b
        12: 1, world, c
        13: 2, a, a
        14: 2, a, b
        15: 2, a, c
        16: 2, combinatorial, a
        17: 2, combinatorial, b
        18: 2, combinatorial, c
        19: 2, hello, a
        20: 2, hello, b
        21: 2, hello, c
        22: 2, world, a
        23: 2, world, b
        24: 2, world, c

Where to go from here ?

The next important step is to implement an algorithm that computes the pair-wize or t-wize suite generation. This is very important if you have a lot of domains. In a near future, I will also show how we can use these products to produce a new type of fixture in MbUnit. This fixture was suggested by Jamie Cansdale.

In this post, I have showed a new, simple, way of creating combinatorial tests using the TestFu.Operations namespace.