# Sunday, August 29, 2004

The "classic" TestFixture now supports a new type of test case: combinatorial tests. This fixture comes from an original idea of Jamie Cansdale, and it's design has been refined with the joint help of Jamie, Omer van Kloeten and the dev@mbunit.tigris.org mailing list.

A simple example

Suppose that we have a ICountX interface that defines a method that counts the number of 'x' in a string:

interface ICountX
{
   int Count(string s);
}

You have two classes that implement that interface:

class SickCountX : ICountX
{
   public int Count(string s)
   {
       return 2;
   }
}

class CountX : ICountX
{
   public int Count(string s)
   {
       int count = 0;
       foreach(char c in s)
           if (c=='x')
              count++;
       return count;
   }
}

Now you would like to test that those two interface implementations work correctly. Let's start by writing a test method inside a TestFixture. The CountIsExact method receives a string, the number of x in the string and checks that a provided ICountX implementation computes the expected x count:

[TestFixture]
public class ICountXTest
{
   public class StringX
   {
       public StringX(string value, int xcount){Value = value; XCount = xcount;}
       public string Value;
       public int XCount;
       public override string ToString() { return String.Format("{0},{1}",this.Value,this.XCount);}
   }

   [CombinatorialTest]
   public void CountIsExact(
      ICountX counter,
      StringX s
      )
   {
       Assert.AreEqual(s.XCount, counter.Count(s.Value));
   }
}

Next, we create a few strings and xcount that can serve for creating test case. By simplicity we use the new C# 2.0 iterators to implement that:

[Factory(typeof(StringX))]
public IEnumerable Strings()
{
    yield return new StringX("",0);
    yield return new StringX("x",1);
    yield return new StringX("xa",1);
    yield return new StringX("xax",2);
    yield return new StringX("aaa",0);
}

Note that we have tagged the method with Factory so that MbUnit knows this method creates StringX instances. We also create another method that will create instance of ICountX to test:

[Factory(typeof(ICountX))]
public IEnumerable Counters()
{
   yield return new SickCountX();
   yield return new CountX();
}

Now, we would like to cross product the output of Counters and StringX and feed the combination in the CountIsExact test method. To do so, we use the UsingFactories attribute to tag the parameters of the method:

[CombinatorialTest]
public void CountIsExact(
   [UsingFactories("Counters")] ICountX counter,
   [UsingFactories("Strings")] StringX s
)
{...}

The fixture is ready, so we can give it a run. The output of the test execution is as follows:

[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(,0))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(x,1))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.SickCountX),Strings(xax,2))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(aaa,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(x,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xax,2))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(aaa,0))

6 succeeded, 4 failed, 0 skipped, took 0,00 seconds.

It worked! As expected, we have 5 x 2 = 10 test case generated. Each test case name is generated out of the data source and different data values. Of course, you can "bind" data to the parameters using various other ways and the tests support more that 2 parameters as we will see in the following. We could also have added multiple Using Attributes.

Prequisites

This new features uses the TestFu.Operations framework in the background. This framework is detailed in the here (part 1) and here (part 2).

Rationale

The CombinatorialTest has the following workflow:

  • for each parameter of the test method, get all the domains D of the parameter from the UsingAttributes (there are a bunch of those). This creates a collection of domain for each parameter which is also a domain itself, which we'll call parameter domain: PD = collection of D,
  • foreach tuple in the CartesianProduct of the parameter domains
    • foreach ptuple in the PairWizeProduct of the domain in the tuple (each entry is a domain for the corresponding method parameters)
      • Create a test case that will invoke the test method with the values in ptuple.

Note that in the example above, PD = { { Counters }, { Strings } }, hence the Cartesian product has only one element {Counters} x {Strings} = (Counters,Strings). 

Custom Using Attributes

All Using attribute inherit from the abstract base class attribute UsingBaseAttribute which defines one abstract method:

public abstract class UsingBaseAttribute : Attribute
{
    public abstract void GetDomains(
        IDomainCollection domains, 
        ParameterInfo parameter,
        object fixture);
}

This method receives the tagged parameter, an instance of the fixture class and a collection of domains. It can add new domain to this fixture. For example, the following attribute, UsingLinear, generate a linearly spaced vector of integers:

public sealed class UsingLinearAttribute : UsingBaseAttribute
{
    private IDomain domain;
    public UsingLinearAttribute(int start, int stepCount)
    {
        this.domain = new LinearInt32Domain(start, stepCount);
    }
    public override void GetDomains(IDomainCollection domains, ParameterInfo parameter, object fixture)
    {
        domains.Add(domain);
    }
}

Here we have used the LinearInt32Domain shipped with TestFu.Operations and we have added that domain to the domain collection. Let's write a test that uses that attribute and see the result:

[CombinatorialTest]
public void UsingLinear(
    [UsingLinear(0, 10)] int i)
{
    Console.WriteLine(i);
}


--- output
[mbunit][Success] UsingLinear(0)
[mbunit][Success] UsingLinear(1)
[mbunit][Success] UsingLinear(2)
[mbunit][Success] UsingLinear(3)
[mbunit][Success] UsingLinear(4)
[mbunit][Success] UsingLinear(5)
[mbunit][Success] UsingLinear(6)
[mbunit][Success] UsingLinear(7)
[mbunit][Success] UsingLinear(8)
[mbunit][Success] UsingLinear(9)
10 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

Built in Using Attributes

MbUnit comes with a number of built-in using attribute that should get you started almost all situations.

UsingLinearAttribute

Already described previously.

UsingLiteralsAttribute

This attribute lets you specify a list of value separated by ';'. MbUnit will try to convert those to the parameter type using Convert.ChangeType method:

[CombinatorialTest]
public void UsingLinearAndLiterals(
    [UsingLinear(0, 3)] int i,
    [UsingLiterals("a;b")] char c
)
{
    Console.WriteLine("{0} {1}",i,c);
}
--- output
[mbunit][Success] UsingLinearAndLiterals(0,a)
[mbunit][Success] UsingLinearAndLiterals(0,b)
[mbunit][Success] UsingLinearAndLiterals(1,a)
[mbunit][Success] UsingLinearAndLiterals(1,b)
[mbunit][Success] UsingLinearAndLiterals(2,a)
[mbunit][Success] UsingLinearAndLiterals(2,b)
6 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

UsingFactoriesAttribute

This attribute will look for factory method in a specified type. If the factory type is not specified, it will default it to the fixture type. If the member name is not specified, it will look for all the methods tagged with Factory attribute and compatible with the parameter type:

  • Not specifying the member name, MbUnit looks for all the compatible factories:
    [Factory]
    public int GetZero()
    {
        return 0;
    }
    [Factory]
    public int[] GetZeroOne()
    {
        return new int[] { 0, 1 };
    }
    [Factory(typeof(int))] // we need to specify the factory type
    public IEnumerable GetZeroOneAsEnumerable()
    {
        return this.GetZeroOne();
    }
    [CombinatorialTest]
    public void UsingFactories(
        [UsingFactories] int i
        )
    {
        Console.WriteLine("{0}", i);
    }
    
    
    -- output
    
    [mbunit][Success] UsingFactories(GetZero(0))
    [mbunit][Success] UsingFactories(GetZeroOne(0))
    [mbunit][Success] UsingFactories(GetZeroOne(1))
    [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(0))
    [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(1))
    
    
  • Using external factories by specifying a factory type:
    public class IntFactory
    {
        [Factory]
        public int One()
        {
            return 1;
        }
    }
    
    
    [CombinatorialTest]
    public void UsingFactory(
        [UsingFactories(typeof(IntFactory))] int i)
    {
        Console.WriteLine("{0}", i);
    }
    
    --- output
    
    [mbunit][Success] UsingFactory(One(1))
    1 succeeded, 0 failed, 0 skipped, took 0,00 seconds.
    

UsingImplementationsAttribute

This attribute looks for all the implementation of a type in an assembly, creates an instance and feeds it to the test.

[CombinatorialTest]
public void TestCountX([UsingImplementationsAttribute(typeof(ICountX))] ICountX countX)

What about tuple filtering ?

In some situation you may want to filter out tuple. For example, we want to check that a methods throws ArgumentNullException on null arguments:

[Factory]
public string[] Strings()
{
    return new string[] { null, "hello" };
}

[CombinatorialTest]
[ExpectedArgumentNullException]
public void TestWithValidator(
    [UsingFactories("Strings")] string s1,
    [UsingFactories("Strings")] string s2
)
{
    if (s1 == null)
        throw new ArgumentNullException();
    if (s2 == null)
        throw new ArgumentNullException();
}

(Note that ExceptedException and all the other decorator work on the combinatorial tests). If we run the test now, we wil have unwanted tuple such as (null, null) and ("hello", "hello"): the first does not give much information and the second will not throw and hence, is erroneous. In this kind of situation, you can specify a "filter" method that takes the same arguments as the test method and return bool:

public bool IsValid(string s1, string s2)
{
   if (s1 == null && s2 == null)
      return false;
   if (s1 != null && s2 != null)
      return false;
   return true;
}

[CombinatorialTest(TupleValidatorMethod = "IsValid")]
[ExpectedArgumentNullException]
public void TestWithValidator(...

--- output

[mbunit][Success] TestWithValidator(Strings(),Strings(hello))
[mbunit][Success] TestWithValidator(Strings(hello),Strings())

As expected, the unwanted tuple have been dropped.

Conclusion

Combinatorial test provides a flexible, extensible and powerfull way of generating test case. It implements the classic PairWize combinatorial generation in order to avoid exponential test explosion while integrating smoothly in the MbUnit architecture.

posted on Sunday, August 29, 2004 3:51:00 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [4]