Reflector.Graph now builds and diplays the list of available addins and their menu location. Look for the Tools -> List of Reflector.Graph Addins menu item.

The TestFu Database Populator Framework (DPF presented here) is a framework for generating data for database testing. Given a DataSet, the framework can build a "smart" data generator to you can later use in your databse testing. I will present this at BENUG on sep. 9.

Why this framework ?

When you try to apply unit testing to database, you choices are not much: use transaction - clean up you "mess" after each test or even worse backup and restore the database on each test. In fact, there is an intrisic problem with unit testing of database: you cannot ensure the atomicity of each test, i.e. you cannot leave a database in the state that it was prior to the test (you could by restoring the db  from file at each test, but this is costly).

For example, transaction through enterprise services (see Roy's Rollback attribute) works great... but if you are using IDENTITY columns, then the identity counter is not reseted by the transaction and thus, the test case are correlated.

Now, what if we wanted to build unit tests for database that would not require clean up. This would mean that new data should be generated at each execution, since the previous execution would not have been cleaned up. This is where DPF comes into the picture: the DPF engine will generate new data for each of your unit test execution at no cost.

Generating random data is not difficult

That's true, the System.Random class is easy to use. However, things gets (a bit) more complicated when you generate data for database because you have to ensure that integrity constraint are enforced. This is why you will need such framework.

DPF How-to

The DPF totally relies on DataSet and comes with a few CodeSmith templates to accelerate development. The following how-to could be applied to any of your database.

1. If not done yet, create the strongly-typed DataSet of your database. You can do this by adding a new Typed DataSet to your project and drag and drop the tables in the designer. (Make sure VS has imported the constraint).
2. Open the DatabasePopulator.cst template. This template willl create a "strongly typed" database populator for a given database,
    
   where
   • Database will let you choose a database available on your machine, select the target database here,
   • Namespace  is the namespace of the strongly typed generator class,
   • TableNamePrefix is the table prefix in your database (in case there is one). This prefix will be trimmed to create the class names
3. Open the CrudPopulator.cst template and edit it in order to match the way your DAL access the database. This template will generate populator class that can apply CRUD operation to your database. You can also directly use that template and edit each "throw new NotImplementedException()" statement. (I strongly suggest you edit it the template for your needs).
4. Open the DatabasePopulatorTest.cst tempate and generate a fixture. This template will generate a fixture that will test each CRUD operation of each table in the database. The data necessary to those tests will be automatically generated by the populator.

Jamie Cansdale has also started to build new Reflector.Graph addins. This new addin creates a simple UML-like class diagram around the selected type.

Some facts:

• All the nodes of the graph are clickable wich makes it an easy way of navigation the type hierachy,
• Base class is marked in blue, which a filled arrow
• Methods and properties are marked with + (public), - (private), # (protected) and @ (internal)

Congrats Jamie!

Reflector version: 4.1.4.0
Reflector.Graph version: 4.1.4.0
Download: www.dotnetwiki.org

Before getting the new version, make sure you have a look at the release notes below!

New Addins

• StatementGraph (FlowGraph),
• Automatic Unit Test Generator (using the StatementGraph),
• Method Invokation Graph

This is the first release of the statement graph. It is surely not perfect. If you find anomalies in your graph, please prepare a code example that shows the problem and send it to me :)

TestFixtureSetUp and TestFixtureTearDown

MbUnit now supports TestFixtureSetUp and TestFixtuteTearDown attribute to mark methods to be executed at the beginning of a fixture test case execution and at the end.

[TestFixture]
public class Fixture
{
    [TestFixtureSetUp]
    public void TestFixtureSetUp()
    {...}

    ...

    [TestFixturteTearDown]
    public void TestFixtureTearDown()
    {...}
}

• Both method are optional and are allowed once per class,
• If TestFixtureSetUp invocation fails, no test case are executed and they are all marked as failures,
• TestFixtureTearDown is always invoked,
• If TestFixtureTearDown fails, all the test cases are marked as failure
• TestFixtureSetUp and TestFixtureTearDown are executed on the same instance. This is not ensured for the other methods,
• Output in those methods are recorded in the reports

AssemblyCleanUp, SetUp and TearDown

MbUnit also support test assembly setup and teardown. Those methods should be enclosed in a class that is feeded to the AssemblyCleanUpAttribute (Assembly attribute, thanks Omer), They must be public and static. The setup method must be tagged with SetUpAttribute, and the teardown method with TearDownAttribute:

[assembly: AssemblyCleanUp(typeof(AssemblyCleaner))]
...
public class AssemblyCleaner
{
    [SetUp]
    public static void SetUp()
    {
        Console.WriteLine("Setting up {0}", typeof(AssemblyCleanUp).Assembly.FullName);
    }
    [TearDown]
    public static void TearDown()
    {
        Console.WriteLine("Cleaning up {0}", typeof(AssemblyCleanUp).Assembly.FullName);
    }
}

• Only one class with AssemblyCleanUp per assembly is authorized,
• Both methods are optional,
• If SetUp fails, no tests are run and they are all marked as failure,
• TearDown is always executed.
• If TearDown fails, all the tests are marked as failure

This new Reflector.Addin creates a Method Graph where the vertex are methods, and the edges represent an invokation of a method. By clicking on the vertices, you jump to the next method, while keeping record of your previous steps.

Let's see this on FileStream.WriteByte method as shown above. The figure above is generated by right-clicking on the FileStream method and selecting "Method Invokation Graph". If I choose to to click FlushRight, the graph changes to:

I have revamped my TestFixture generator to work with the StatementGraph (i.e. FlowGraph). The generator logics has not changed:

1. Build the Statement Graph from the method,
2. Extract the path necessary to cover the entire graph (cover all statements in the method),
3. Create one method for each path
4. In each method documentation, output the decompiled code where the "target" statements have been highlighted.

Disclaimer

1. This generator is far from behing finished or optimal: the algorithm to produce the graph is not optimal, I have not implement the Newyork Street Sweeper algorithm yet,
2. If your code is buggy, the generator will generate tests for ... buggy code.

Examples

Here's a sample of what gets output from the examples in the StatementGraph article. Targetted statements are marked with /* i */ where i is the index of the tested statement

/// <summary>Test fixture for the &lt;see cref="StatementSamples"/&gt; class</summary>
/// <remarks />
[TestFixture()]
public class StatementSamplesTest
{
    /// <summary>Tests the Simple method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ Console.WriteLine("hello");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void Simple0()
    {}
    /// <summary>Tests the Block method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ Console.WriteLine("hello");
    /// /* 1 */ Console.WriteLine("world");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void Block0()
    {}
    /// <summary>Tests the If method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 83,3%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ if (value &lt; 0)
    /// {
    /// /* 1 */ Console.WriteLine("true");
    /// /* 2 */ return;
    /// }
    /// Console.WriteLine("false");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void If0()
    {}
    /// <summary>Tests the If method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 50,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ if (value &lt; 0)
    /// {
    /// Console.WriteLine("true");
    /// return;
    /// }
    /// /* 1 */ Console.WriteLine("false");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void If1()
    {}
    /// <summary>Tests the While method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ int num1 = 0;
    /// while ((num1 &lt; 10))
    /// {
    /// /* 1 */ Console.Write(num1++);
    /// }
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void While0()
    {}
}

This post presents a new Reflector Addin that creates and diplays the graph of statement inside methods: the StatementGraph. This addin is an evolution of the IL graph. (this addin is not yet available for download).

 The statement graph is built from the Reflector CodeModel where each IStatement instance is a vertex and edges are added accordingly to the code "flow" (creating the edges is the most involved task). The rest of the job is handled by the QuickGraph library. When it makes sense, the vertices are clickable and you can jump to the invoked method, etc... The next step will be to update the Automatic Unit Test generator with the StatementGraph...

Let's see some simple methods and their corresponding graphs:

Simple

• Original code:

  public void Simple()
  {
      Console.WriteLine("hello");
  }
  

• Decompiled:

  public void Simple()
  {
        Console.WriteLine("hello");
  }

• Graph:

  

Two statements in sequence

• Original code:

  public void Body()
  {
      Console.WriteLine("hello");
      Console.WriteLine("world");
  }
  

• Decompiled:

  public void Body()
  {
        Console.WriteLine("hello");
        Console.WriteLine("world");
  }

• Graph:

  

If - Then - Else

• Original code:

  public void If(int value)
  {
      if (value<0)
          Console.WriteLine("true");
      else
          Console.WriteLine("false");
  }

• Decompiled:

  public void If(int value)
  {
        if (value < 0)
        {
              Console.WriteLine("true");
              return;
        }
        Console.WriteLine("false");
  }
  

• Graph:

  

While

• Original code:

  public void While()
  {
      int i = 0;
      while(i<10)
      {
          Console.Write(i++);
      }
  }

• Decompiled:

  public void While()
  {
        int num1 = 0;
        while ((num1 < 10))
        {
              Console.Write(num1++);
        }
  }

• Graph:

  

While with break and continue

• Original code:

  public void WhileBreakContinue()
  {
      int i = 0;
      while (i < 10)
      {
          if (i == 5)
              continue;
          if (i == 7)
              break;
          Console.Write(i++);
      }
      Console.WriteLine("Finished");
  }

• Decompiled:

  public void WhileBreakContinue()
  {
        int num1 = 0;
        while ((num1 < 10))
        {
              if (num1 == 5)
              {
                    continue;
              }
              if (num1 == 7)
              {
                    break;
              }
              Console.Write(num1++);
        }
        Console.WriteLine("Finished");
  }
  
  
  

• Graph:

  

Try - Catch

• Original code:

  public void TryCatch()
  {
      try
      {
          Console.WriteLine("hello");
      }
      catch (Exception ex)
      {
          Console.WriteLine("Boom: {0}",ex);
      }
  }
  

• Decompiled:

  public void TryCatch()
  {
        try
        {
              Console.WriteLine("hello");
        }
        catch (Exception exception1)
        {
              Console.WriteLine("Boom: {0}", exception1);
        }
  }

• Graph:

  

Just finished the private presentation of the PhD... One month to go and I'll get the degree :)

For the curious, here's a link to the presentation: http://blog.dotnetwiki.org/downloads/PhDPresentationdeHalleux.pdf

The "classic" TestFixture now supports a new type of test case: combinatorial tests. This fixture comes from an original idea of Jamie Cansdale, and it's design has been refined with the joint help of Jamie, Omer van Kloeten and the dev@mbunit.tigris.org mailing list.

A simple example

Suppose that we have a ICountX interface that defines a method that counts the number of 'x' in a string:

interface ICountX
{
   int Count(string s);
}

You have two classes that implement that interface:

class SickCountX : ICountX
{
   public int Count(string s)
   {
       return 2;
   }
}

class CountX : ICountX
{
   public int Count(string s)
   {
       int count = 0;
       foreach(char c in s)
           if (c=='x')
              count++;
       return count;
   }
}

Now you would like to test that those two interface implementations work correctly. Let's start by writing a test method inside a TestFixture. The CountIsExact method receives a string, the number of x in the string and checks that a provided ICountX implementation computes the expected x count:

[TestFixture]
public class ICountXTest
{
   public class StringX
   {
       public StringX(string value, int xcount){Value = value; XCount = xcount;}
       public string Value;
       public int XCount;
       public override string ToString() { return String.Format("{0},{1}",this.Value,this.XCount);}
   }

   [CombinatorialTest]
   public void CountIsExact(
      ICountX counter,
      StringX s
      )
   {
       Assert.AreEqual(s.XCount, counter.Count(s.Value));
   }
}

Next, we create a few strings and xcount that can serve for creating test case. By simplicity we use the new C# 2.0 iterators to implement that:

[Factory(typeof(StringX))]
public IEnumerable Strings()
{
    yield return new StringX("",0);
    yield return new StringX("x",1);
    yield return new StringX("xa",1);
    yield return new StringX("xax",2);
    yield return new StringX("aaa",0);
}

Note that we have tagged the method with Factory so that MbUnit knows this method creates StringX instances. We also create another method that will create instance of ICountX to test:

[Factory(typeof(ICountX))]
public IEnumerable Counters()
{
   yield return new SickCountX();
   yield return new CountX();
}

Now, we would like to cross product the output of Counters and StringX and feed the combination in the CountIsExact test method. To do so, we use the UsingFactories attribute to tag the parameters of the method:

[CombinatorialTest]
public void CountIsExact(
   [UsingFactories("Counters")] ICountX counter,
   [UsingFactories("Strings")] StringX s
)
{...}

The fixture is ready, so we can give it a run. The output of the test execution is as follows:

[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(,0))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(x,1))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.SickCountX),Strings(xax,2))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(aaa,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(x,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xax,2))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(aaa,0))
6 succeeded, 4 failed, 0 skipped, took 0,00 seconds.

It worked! As expected, we have 5 x 2 = 10 test case generated. Each test case name is generated out of the data source and different data values. Of course, you can "bind" data to the parameters using various other ways and the tests support more that 2 parameters as we will see in the following. We could also have added multiple Using Attributes.

Prequisites

This new features uses the TestFu.Operations framework in the background. This framework is detailed in the here (part 1) and here (part 2).

Rationale

The CombinatorialTest has the following workflow:

• for each parameter of the test method, get all the domains D of the parameter from the UsingAttributes (there are a bunch of those). This creates a collection of domain for each parameter which is also a domain itself, which we'll call parameter domain: PD = collection of D,
• foreach tuple in the CartesianProduct of the parameter domains
  • foreach ptuple in the PairWizeProduct of the domain in the tuple (each entry is a domain for the corresponding method parameters)
    • Create a test case that will invoke the test method with the values in ptuple.

Note that in the example above, PD = { { Counters }, { Strings } }, hence the Cartesian product has only one element {Counters} x {Strings} = (Counters,Strings). 

Custom Using Attributes

All Using attribute inherit from the abstract base class attribute UsingBaseAttribute which defines one abstract method:

public abstract class UsingBaseAttribute : Attribute
{
    public abstract void GetDomains(
        IDomainCollection domains, 
        ParameterInfo parameter,
        object fixture);
}

This method receives the tagged parameter, an instance of the fixture class and a collection of domains. It can add new domain to this fixture. For example, the following attribute, UsingLinear, generate a linearly spaced vector of integers:

public sealed class UsingLinearAttribute : UsingBaseAttribute
{
    private IDomain domain;
    public UsingLinearAttribute(int start, int stepCount)
    {
        this.domain = new LinearInt32Domain(start, stepCount);
    }
    public override void GetDomains(IDomainCollection domains, ParameterInfo parameter, object fixture)
    {
        domains.Add(domain);
    }
}

Here we have used the LinearInt32Domain shipped with TestFu.Operations and we have added that domain to the domain collection. Let's write a test that uses that attribute and see the result:

[CombinatorialTest]
public void UsingLinear(
    [UsingLinear(0, 10)] int i)
{
    Console.WriteLine(i);
}


--- output
[mbunit][Success] UsingLinear(0)
[mbunit][Success] UsingLinear(1)
[mbunit][Success] UsingLinear(2)
[mbunit][Success] UsingLinear(3)
[mbunit][Success] UsingLinear(4)
[mbunit][Success] UsingLinear(5)
[mbunit][Success] UsingLinear(6)
[mbunit][Success] UsingLinear(7)
[mbunit][Success] UsingLinear(8)
[mbunit][Success] UsingLinear(9)
10 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

Built in Using Attributes

MbUnit comes with a number of built-in using attribute that should get you started almost all situations.

UsingLinearAttribute

Already described previously.

UsingLiteralsAttribute

This attribute lets you specify a list of value separated by ';'. MbUnit will try to convert those to the parameter type using Convert.ChangeType method:

[CombinatorialTest]
public void UsingLinearAndLiterals(
    [UsingLinear(0, 3)] int i,
    [UsingLiterals("a;b")] char c
)
{
    Console.WriteLine("{0} {1}",i,c);
}
--- output
[mbunit][Success] UsingLinearAndLiterals(0,a)
[mbunit][Success] UsingLinearAndLiterals(0,b)
[mbunit][Success] UsingLinearAndLiterals(1,a)
[mbunit][Success] UsingLinearAndLiterals(1,b)
[mbunit][Success] UsingLinearAndLiterals(2,a)
[mbunit][Success] UsingLinearAndLiterals(2,b)
6 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

UsingFactoriesAttribute

This attribute will look for factory method in a specified type. If the factory type is not specified, it will default it to the fixture type. If the member name is not specified, it will look for all the methods tagged with Factory attribute and compatible with the parameter type:

• Not specifying the member name, MbUnit looks for all the compatible factories:

  [Factory]
  public int GetZero()
  {
      return 0;
  }
  [Factory]
  public int[] GetZeroOne()
  {
      return new int[] { 0, 1 };
  }
  [Factory(typeof(int))] // we need to specify the factory type
  public IEnumerable GetZeroOneAsEnumerable()
  {
      return this.GetZeroOne();
  }
  [CombinatorialTest]
  public void UsingFactories(
      [UsingFactories] int i
      )
  {
      Console.WriteLine("{0}", i);
  }
  
  
  -- output
  
  [mbunit][Success] UsingFactories(GetZero(0))
  [mbunit][Success] UsingFactories(GetZeroOne(0))
  [mbunit][Success] UsingFactories(GetZeroOne(1))
  [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(0))
  [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(1))
  
  

• Using external factories by specifying a factory type:

  public class IntFactory
  {
      [Factory]
      public int One()
      {
          return 1;
      }
  }
  
  
  [CombinatorialTest]
  public void UsingFactory(
      [UsingFactories(typeof(IntFactory))] int i)
  {
      Console.WriteLine("{0}", i);
  }
  
  --- output
  
  [mbunit][Success] UsingFactory(One(1))
  1 succeeded, 0 failed, 0 skipped, took 0,00 seconds.
  

UsingImplementationsAttribute

This attribute looks for all the implementation of a type in an assembly, creates an instance and feeds it to the test.

[CombinatorialTest]
public void TestCountX([UsingImplementationsAttribute(typeof(ICountX))] ICountX countX)

What about tuple filtering ?

In some situation you may want to filter out tuple. For example, we want to check that a methods throws ArgumentNullException on null arguments:

[Factory]
public string[] Strings()
{
    return new string[] { null, "hello" };
}

[CombinatorialTest]
[ExpectedArgumentNullException]
public void TestWithValidator(
    [UsingFactories("Strings")] string s1,
    [UsingFactories("Strings")] string s2
)
{
    if (s1 == null)
        throw new ArgumentNullException();
    if (s2 == null)
        throw new ArgumentNullException();
}

(Note that ExceptedException and all the other decorator work on the combinatorial tests). If we run the test now, we wil have unwanted tuple such as (null, null) and ("hello", "hello"): the first does not give much information and the second will not throw and hence, is erroneous. In this kind of situation, you can specify a "filter" method that takes the same arguments as the test method and return bool:

public bool IsValid(string s1, string s2)
{
   if (s1 == null && s2 == null)
      return false;
   if (s1 != null && s2 != null)
      return false;
   return true;
}

[CombinatorialTest(TupleValidatorMethod = "IsValid")]
[ExpectedArgumentNullException]
public void TestWithValidator(...

--- output

[mbunit][Success] TestWithValidator(Strings(),Strings(hello))
[mbunit][Success] TestWithValidator(Strings(hello),Strings())

As expected, the unwanted tuple have been dropped.

Conclusion

Combinatorial test provides a flexible, extensible and powerfull way of generating test case. It implements the classic PairWize combinatorial generation in order to avoid exponential test explosion while integrating smoothly in the MbUnit architecture.