Reflector.Graph has been updated for Reflector 4.1.6.0.

Download available at www.dotnetwiki.org

This is the first release candidate for MbUnit 2.21.1.0. This release also contains QuickGraph (+NGraphviz), Refly and TestFu.

Acknowledgment
I'd like to give a special thanks to Jamie Cansdale (TestDriven.NET) which has been the main architect of the new MbUnit compilation scripts,  installers and TestDriven.NET intergration.

Installing MbUnit 2.21.1.0 RC1

[Update: follow the instructions at http://www.testdriven.net/wiki/default.aspx/MyWiki.DownLoad]

Where is MbUnit installed ?

The MbUnit assemblies are located in Program Files/TestDriven.NET/MbUnit.

New features

• TestDriven.NET integration: MbUnit is now shipped with TD.NET (see Download instructions below) and fully integrated with it. You can execute any MbUnit test, fixture, namespace directly from Visual Studio using TD.NET. The execution using TD.NET also creates an HTML that is directly browsable from VS,
• Fixture level setup and teardown: (TestFixtureSetUp and TestFixtureTearDown) and assembly level setup and teardown,
• Drag and drop Unit Testing: a new "designer friendly" way of creating your fixtures: Demo1, Demo2, Demo3
• Combinatorial tests, in MbUnit, introduction, all pairs
• Smart test tools in TestFu such as the Database Populator Framework,

Sorry, I can' resist putting little animation of the new design-time support in MbUnit. This sample shows how you can easily set up monitoring on a few PerformanceCounter inside a fixture component. (Demo solution at http://blog.dotnetwiki.org/downloads/DragAndDropUnitTesting.zip )

How-to

The procedure to monitor a PerformanceCounter is as follows:

1. Add any PerformanceCounter to the component (drag and drop from the Toolbox),
2. Make sure they are set up,
3. For each PerformanceCounter, add a PerformanceCounterChecker and associate it a PerformanceCounter instance in the property pane
4. Add the desired tests

For each of the test, the PerformanceCounterChecker will monitor the value of it's associated counter and it will assert if the value goes out of the specified bounds. In the report, a little summary of the counter values is shown (in the Console.Out view of the report).

I've recently discovered Context in .Net while reading the Programming .Net Components book (Chapter 11). This is an amazing piece of technology that I definitely needed to try. A first try was to make a Caching context:

Caching context

Image that you have methods that perform long computation, such as querying a database etc..., and return a result. Usually, the result does not change much over time, so typically you would like to cache the results in order to improve the efficiency of the application. Now imagine that there exists a Caching context that would take care of caching method calls. For example, we would like to write something like this:

[Caching]
public class CachedClass : ContextBoundObject
{
    [Cached]
    public string ReturnBigObject()
    { 
        Console.WriteLine("miss");
        Thread.Sleep(1000);
        return DateTime.Now.ToString(); 
    }
}

In that sample, we would like the output of ReturnBigObject to be cached. For example, the sample method below shows the desired behavior. The first call to ReturnBigObject is a cache miss and then, the remaining calls are cached.

miss
c.ReturnBigObject(): 5/10/2004 19:31:31
c.ReturnBigObject(): 5/10/2004 19:31:31
c.ReturnBigObject(): 5/10/2004 19:31:31

Let's see the steps to take to create the caching context:

Building the context (1): CachingAttribute

The CachingAttribute has to inherit from ContextAttribute and override two methods. The main task of the attribute is to add a IContextProperty implementation to the context properties (GetPropertiesForNewContext method). 

[AttributeUsage(AttributeTargets.Class,AllowMultiple =false,Inherited =true)]
public class CachingAttribute : ContextAttribute
{
    public CachingAttribute():base("Chaching")
    {}
    public override void GetPropertiesForNewContext(IConstructionCallMessage ctorMsg)
    {
        IContextProperty property =
            new CachingContextProperty(ctorMsg.ActivationType);
        ctorMsg.ContextProperties.Add(property);
    }
    public override bool IsContextOK(Context ctx, IConstructionCallMessage ctorMsg)
    {
        return false;
    }
}

Building the context (1.1): CachedAttribute

The CachedAttribute is used to tag method to be cached. It contains several parameters to set up the HttpRuntime.Cache object.

Building the context (2): CachingContextProperty

The task of this class is to install a server IMessageSink in the message flow that will cache the message calls. Therefore, this class implements IContextProperty, IContributeServerContextSink.

public class CachingContextProperty : 
    IContextProperty, IContributeServerContextSink
{
    private Type activationType;
    ...
    public IMessageSink GetServerContextSink(IMessageSink nextSink)
    {
        CachingSink cachingSink = new CachingSink(nextSink, activationType);
        return cachingSink;
    }
}

Building the context (3): CachingSink

This is where the real work occurs. This class filters the call to the method (IMethodMessage) and looks in a Cache (HttpRuntime.Cache) if it is stored, if stored it returns the value, otherwize it calls the next sink and store the value in the Cache. (This part of the code is a little bit more technical because a part of the IMethodReturnMessage has to be "copied").

The heart of this class is as follows:

public IMethodReturnMessage SyncProcessMessage(IMethodMessage msg)
{
    // creating unique hash value out of method, instance and paramteres
    string hash = Hash(msg);
    // looking in the cache
    IMethodReturnMessage returnMessage = HttpRuntime.Cache[hash] as IMethodReturnMessage;
    if (returnMessage != null) // cache hit
        return returnMessage;
    
    returnMessage = this.Parent.NextSink.SyncProcessMessage(msg) 
        as IMethodReturnMessage;

    // caching returned information
    CachedMethodReturnMessage cachedMessage= new CachedMethodReturnMessage(returnMessage);

    // storing in cache
    HttpRuntime.Cache.Add(hash, cachedMessage,
        this.cachedAttribute.Dependencies,
        this.cachedAttribute.AbsoluteExpiration,
        this.cachedAttribute.SlidingExpiration,
        this.cachedAttribute.Priority,
        null);
    IMethodReturnMessage r = HttpRuntime.Cache[hash] as IMethodReturnMessage;
    return returnMessage;
}

Downloads

The full source is available on www.dotnetwiki.org

I've recently discovered Context in .Net while reading the Programming .Net Components book (Chapter 11). This is an amazing piece of technology that I definitely needed to try. A first try was to make a Caching context:

Caching context

Image that you have methods that perform long computation, such as querying a database etc..., and return a result. Usually, the result does not change much over time, so typically you would like to cache the results in order to improve the efficiency of the application. Now imagine that there exists a Caching context that would take care of caching method calls. For example, we would like to write something like this:

[Caching]
public class CachedClass : ContextBoundObject
{
    [Cached]
    public string ReturnBigObject()
    { 
        Console.WriteLine("miss");
        Thread.Sleep(1000);
        return DateTime.Now.ToString(); 
    }
}

In that sample, we would like the output of ReturnBigObject to be cached. For example, the sample method below shows the desired behavior. The first call to ReturnBigObject is a cache miss and then, the remaining calls are cached.

miss
c.ReturnBigObject(): 5/10/2004 19:31:31
c.ReturnBigObject(): 5/10/2004 19:31:31
c.ReturnBigObject(): 5/10/2004 19:31:31

Let's see the steps to take to create the caching context:

Building the context (1): CachingAttribute

The CachingAttribute has to inherit from ContextAttribute and override two methods. The main task of the attribute is to add a IContextProperty implementation to the context properties (GetPropertiesForNewContext method). 

[AttributeUsage(AttributeTargets.Class,AllowMultiple =false,Inherited =true)]
public class CachingAttribute : ContextAttribute
{
    public CachingAttribute():base("Chaching")
    {}
    public override void GetPropertiesForNewContext(IConstructionCallMessage ctorMsg)
    {
        IContextProperty property =
            new CachingContextProperty(ctorMsg.ActivationType);
        ctorMsg.ContextProperties.Add(property);
    }
    public override bool IsContextOK(Context ctx, IConstructionCallMessage ctorMsg)
    {
        return false;
    }
}

Building the context (1.1): CachedAttribute

The CachedAttribute is used to tag method to be cached. It contains several parameters to set up the HttpRuntime.Cache object.

Building the context (2): CachingContextProperty

The task of this class is to install a server IMessageSink in the message flow that will cache the message calls. Therefore, this class implements IContextProperty, IContributeServerContextSink.

public class CachingContextProperty : 
    IContextProperty, IContributeServerContextSink
{
    private Type activationType;
    ...
    public IMessageSink GetServerContextSink(IMessageSink nextSink)
    {
        CachingSink cachingSink = new CachingSink(nextSink, activationType);
        return cachingSink;
    }
}

Building the context (3): CachingSink

This is where the real work occurs. This class filters the call to the method (IMethodMessage) and looks in a Cache (HttpRuntime.Cache) if it is stored, if stored it returns the value, otherwize it calls the next sink and store the value in the Cache. (This part of the code is a little bit more technical because a part of the IMethodReturnMessage has to be "copied").

The heart of this class is as follows:

public IMethodReturnMessage SyncProcessMessage(IMethodMessage msg)
{
    // creating unique hash value out of method, instance and paramteres
    string hash = Hash(msg);
    // looking in the cache
    IMethodReturnMessage returnMessage = HttpRuntime.Cache[hash] as IMethodReturnMessage;
    if (returnMessage != null) // cache hit
        return returnMessage;
    
    returnMessage = this.Parent.NextSink.SyncProcessMessage(msg) 
        as IMethodReturnMessage;

    // caching returned information
    CachedMethodReturnMessage cachedMessage= new CachedMethodReturnMessage(returnMessage);

    // storing in cache
    HttpRuntime.Cache.Add(hash, cachedMessage,
        this.cachedAttribute.Dependencies,
        this.cachedAttribute.AbsoluteExpiration,
        this.cachedAttribute.SlidingExpiration,
        this.cachedAttribute.Priority,
        null);
    IMethodReturnMessage r = HttpRuntime.Cache[hash] as IMethodReturnMessage;
    return returnMessage;
}

Downloads

The full source is available on www.dotnetwiki.org

Following the example of drag and drop unit tests. I have added support for the classic ExpectedException behavior as well as the four Setup, TearDown, FixtureSetUp, FixtureTearDown methods (sorry another screen grad).

Following the example of drag and drop unit tests. I have added support for the classic ExpectedException behavior as well as the four Setup, TearDown, FixtureSetUp, FixtureTearDown methods (sorry another screen grad).

Reflector.Graph release for Reflector 4.1.5.0 available at www.dotnetwiki.org

[Update] Changed Reflector.Graph back to an assembly

New addins:

• Add List: gives thelist of loaded addins,
• Typed DataSet structure: structure of a DataSet (experimental)
• Class Diagram (UML like)

Reflector.Graph release for Reflector 4.1.5.0 available at www.dotnetwiki.org

[Update] Changed Reflector.Graph back to an assembly

New addins:

• Add List: gives thelist of loaded addins,
• Typed DataSet structure: structure of a DataSet (experimental)
• Class Diagram (UML like)

Image that you could open the designer, drag and drop a few test case, and run them with minimal writing... now take a look at this:

(Download a demo solution at http://blog.dotnetwiki.org/downloads/DragAndDropUnitTesting.zip )

Image that you could open the designer, drag and drop a few test case, and run them with minimal writing... now take a look at this:

(Download a demo solution at http://blog.dotnetwiki.org/downloads/DragAndDropUnitTesting.zip )

Introduction

Have you ever wondered how you could animate armies of thousands of soldiers ? flocks of hunderds of fish ? etc... Well, you use what they call Autonomous Agents: agents that have a local behavior that creates interresting global behaviors. The first example of autonomous agents was brought in 1984 by Craig Reynolds which developped a small application to simulation flocks: the boids application was born.

Since then, Reynolds has published an excellent paper in 99 on the subject: Steering behavior for Autonomous Characteres. If you never had a look at this, check out his web site and look at the java demo, it is just splendid!

Previous try

During my Phd, I monitored the course of C/C++ for third year Engineer students. The autonomous agents theme was very appealing and students got very involved into that project. (If you are looking for something fun for teaching software engineering, this theme is great!). At that time, they received a mini drawing library that could render agents in real time in OpenGL and Glut. If you want to give it a try, you can download it from www.dotnetwiki.org (look for Autonomous.binaries.zip in the download section) Of course, I tested the project on myself and built an application that looked as follows:

On the picture, you can see 151 agent (green dots) that are moving toward the little black dot. They are avoiding themselves, avoiding obstacles (big gray circle). The little lines between agents show that there is a "flocking" interaction,i.e. they are avoiding themselves.

During that project, we could see a lot of interresting properties of such flocks. For example, the system has a self-organization property. If you let the simulation go long enough, agents will organize themselves in a perfect triangular tiling as show below:

We could also see waves where the agents stopped going in reverse direction from the direction of the flock (In PDE theory, those are called shocks). In fact, it is the same phenomenon that occurs in traffic jam when a wave a "zero" velocity is travelling along the highway. When you stand back in your car, just look at the way you will start run a few hundred meters and the stop, and again and again. This behavior is predicted by the hyperbolic PDE theory :) In the figure below, the agents are moving left toward the little black dot. You can visualize the waves going right. The gray lines denote flock which means that agents are stopping in order to avoid each other.

NSteer

The code C++ was written using Tempate Meta Programming. In NSteer, I'll try to build gradually a framework for C#

Introduction

Have you ever wondered how you could animate armies of thousands of soldiers ? flocks of hunderds of fish ? etc... Well, you use what they call Autonomous Agents: agents that have a local behavior that creates interresting global behaviors. The first example of autonomous agents was brought in 1984 by Craig Reynolds which developped a small application to simulation flocks: the boids application was born.

Since then, Reynolds has published an excellent paper in 99 on the subject: Steering behavior for Autonomous Characteres. If you never had a look at this, check out his web site and look at the java demo, it is just splendid!

Previous try

During my Phd, I monitored the course of C/C++ for third year Engineer students. The autonomous agents theme was very appealing and students got very involved into that project. (If you are looking for something fun for teaching software engineering, this theme is great!). At that time, they received a mini drawing library that could render agents in real time in OpenGL and Glut. If you want to give it a try, you can download it from www.dotnetwiki.org (look for Autonomous.binaries.zip in the download section) Of course, I tested the project on myself and built an application that looked as follows:

On the picture, you can see 151 agent (green dots) that are moving toward the little black dot. They are avoiding themselves, avoiding obstacles (big gray circle). The little lines between agents show that there is a "flocking" interaction,i.e. they are avoiding themselves.

During that project, we could see a lot of interresting properties of such flocks. For example, the system has a self-organization property. If you let the simulation go long enough, agents will organize themselves in a perfect triangular tiling as show below:

We could also see waves where the agents stopped going in reverse direction from the direction of the flock (In PDE theory, those are called shocks). In fact, it is the same phenomenon that occurs in traffic jam when a wave a "zero" velocity is travelling along the highway. When you stand back in your car, just look at the way you will start run a few hundred meters and the stop, and again and again. This behavior is predicted by the hyperbolic PDE theory :) In the figure below, the agents are moving left toward the little black dot. You can visualize the waves going right. The gray lines denote flock which means that agents are stopping in order to avoid each other.

NSteer

The code C++ was written using Tempate Meta Programming. In NSteer, I'll try to build gradually a framework for C#

The full source of the BENUG presentation is available for download at http://blog.dotnetwiki.org/downloads/benug.package.zip

The file contains the two presentation I did (TFU.pdf and DPF.pdf in the Showtime folder), the CodeSmith templates and the sample project solution. The first presentation gave a quick introduction to unit testing tools + basic database testing. The second paper presented a new way of handling database testing through intelligent data generators (Database Populator Framework).

If you want to do the exercise of the DPF presentation, you should install TestDriven.Net....msi on your machine. Make sure you remove NUnitAddIn before doing that. This file is a special build of NUnitAddIn (now named TestDriven.Net) that ships with MbUnit.

The full source of the BENUG presentation is available for download at http://blog.dotnetwiki.org/downloads/benug.package.zip

The file contains the two presentation I did (TFU.pdf and DPF.pdf in the Showtime folder), the CodeSmith templates and the sample project solution. The first presentation gave a quick introduction to unit testing tools + basic database testing. The second paper presented a new way of handling database testing through intelligent data generators (Database Populator Framework).

If you want to do the exercise of the DPF presentation, you should install TestDriven.Net....msi on your machine. Make sure you remove NUnitAddIn before doing that. This file is a special build of NUnitAddIn (now named TestDriven.Net) that ships with MbUnit.

This addin explores a Typed DataSet generated by the Visual Studio and creates the table structure. For the example, selecting the NorthwindDataSet, you will get this output:

This addin explores a Typed DataSet generated by the Visual Studio and creates the table structure. For the example, selecting the NorthwindDataSet, you will get this output:

Reflector.Graph now builds and diplays the list of available addins and their menu location. Look for the Tools -> List of Reflector.Graph Addins menu item.

The TestFu Database Populator Framework (DPF presented here) is a framework for generating data for database testing. Given a DataSet, the framework can build a "smart" data generator to you can later use in your databse testing. I will present this at BENUG on sep. 9.

Why this framework ?

When you try to apply unit testing to database, you choices are not much: use transaction - clean up you "mess" after each test or even worse backup and restore the database on each test. In fact, there is an intrisic problem with unit testing of database: you cannot ensure the atomicity of each test, i.e. you cannot leave a database in the state that it was prior to the test (you could by restoring the db  from file at each test, but this is costly).

For example, transaction through enterprise services (see Roy's Rollback attribute) works great... but if you are using IDENTITY columns, then the identity counter is not reseted by the transaction and thus, the test case are correlated.

Now, what if we wanted to build unit tests for database that would not require clean up. This would mean that new data should be generated at each execution, since the previous execution would not have been cleaned up. This is where DPF comes into the picture: the DPF engine will generate new data for each of your unit test execution at no cost.

Generating random data is not difficult

That's true, the System.Random class is easy to use. However, things gets (a bit) more complicated when you generate data for database because you have to ensure that integrity constraint are enforced. This is why you will need such framework.

DPF How-to

The DPF totally relies on DataSet and comes with a few CodeSmith templates to accelerate development. The following how-to could be applied to any of your database.

1. If not done yet, create the strongly-typed DataSet of your database. You can do this by adding a new Typed DataSet to your project and drag and drop the tables in the designer. (Make sure VS has imported the constraint).
2. Open the DatabasePopulator.cst template. This template willl create a "strongly typed" database populator for a given database,
    
   where
   • Database will let you choose a database available on your machine, select the target database here,
   • Namespace  is the namespace of the strongly typed generator class,
   • TableNamePrefix is the table prefix in your database (in case there is one). This prefix will be trimmed to create the class names
3. Open the CrudPopulator.cst template and edit it in order to match the way your DAL access the database. This template will generate populator class that can apply CRUD operation to your database. You can also directly use that template and edit each "throw new NotImplementedException()" statement. (I strongly suggest you edit it the template for your needs).
4. Open the DatabasePopulatorTest.cst tempate and generate a fixture. This template will generate a fixture that will test each CRUD operation of each table in the database. The data necessary to those tests will be automatically generated by the populator.

Jamie Cansdale has also started to build new Reflector.Graph addins. This new addin creates a simple UML-like class diagram around the selected type.

Some facts:

• All the nodes of the graph are clickable wich makes it an easy way of navigation the type hierachy,
• Base class is marked in blue, which a filled arrow
• Methods and properties are marked with + (public), - (private), # (protected) and @ (internal)

Congrats Jamie!

Reflector version: 4.1.4.0
Reflector.Graph version: 4.1.4.0
Download: www.dotnetwiki.org

Before getting the new version, make sure you have a look at the release notes below!

New Addins

• StatementGraph (FlowGraph),
• Automatic Unit Test Generator (using the StatementGraph),
• Method Invokation Graph

This is the first release of the statement graph. It is surely not perfect. If you find anomalies in your graph, please prepare a code example that shows the problem and send it to me :)

TestFixtureSetUp and TestFixtureTearDown

MbUnit now supports TestFixtureSetUp and TestFixtuteTearDown attribute to mark methods to be executed at the beginning of a fixture test case execution and at the end.

[TestFixture]
public class Fixture
{
    [TestFixtureSetUp]
    public void TestFixtureSetUp()
    {...}

    ...

    [TestFixturteTearDown]
    public void TestFixtureTearDown()
    {...}
}

• Both method are optional and are allowed once per class,
• If TestFixtureSetUp invocation fails, no test case are executed and they are all marked as failures,
• TestFixtureTearDown is always invoked,
• If TestFixtureTearDown fails, all the test cases are marked as failure
• TestFixtureSetUp and TestFixtureTearDown are executed on the same instance. This is not ensured for the other methods,
• Output in those methods are recorded in the reports

AssemblyCleanUp, SetUp and TearDown

MbUnit also support test assembly setup and teardown. Those methods should be enclosed in a class that is feeded to the AssemblyCleanUpAttribute (Assembly attribute, thanks Omer), They must be public and static. The setup method must be tagged with SetUpAttribute, and the teardown method with TearDownAttribute:

[assembly: AssemblyCleanUp(typeof(AssemblyCleaner))]
...
public class AssemblyCleaner
{
    [SetUp]
    public static void SetUp()
    {
        Console.WriteLine("Setting up {0}", typeof(AssemblyCleanUp).Assembly.FullName);
    }
    [TearDown]
    public static void TearDown()
    {
        Console.WriteLine("Cleaning up {0}", typeof(AssemblyCleanUp).Assembly.FullName);
    }
}

• Only one class with AssemblyCleanUp per assembly is authorized,
• Both methods are optional,
• If SetUp fails, no tests are run and they are all marked as failure,
• TearDown is always executed.
• If TearDown fails, all the tests are marked as failure

This new Reflector.Addin creates a Method Graph where the vertex are methods, and the edges represent an invokation of a method. By clicking on the vertices, you jump to the next method, while keeping record of your previous steps.

Let's see this on FileStream.WriteByte method as shown above. The figure above is generated by right-clicking on the FileStream method and selecting "Method Invokation Graph". If I choose to to click FlushRight, the graph changes to:

I have revamped my TestFixture generator to work with the StatementGraph (i.e. FlowGraph). The generator logics has not changed:

1. Build the Statement Graph from the method,
2. Extract the path necessary to cover the entire graph (cover all statements in the method),
3. Create one method for each path
4. In each method documentation, output the decompiled code where the "target" statements have been highlighted.

Disclaimer

1. This generator is far from behing finished or optimal: the algorithm to produce the graph is not optimal, I have not implement the Newyork Street Sweeper algorithm yet,
2. If your code is buggy, the generator will generate tests for ... buggy code.

Examples

Here's a sample of what gets output from the examples in the StatementGraph article. Targetted statements are marked with /* i */ where i is the index of the tested statement

/// <summary>Test fixture for the &lt;see cref="StatementSamples"/&gt; class</summary>
/// <remarks />
[TestFixture()]
public class StatementSamplesTest
{
    /// <summary>Tests the Simple method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ Console.WriteLine("hello");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void Simple0()
    {}
    /// <summary>Tests the Block method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ Console.WriteLine("hello");
    /// /* 1 */ Console.WriteLine("world");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void Block0()
    {}
    /// <summary>Tests the If method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 83,3%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ if (value &lt; 0)
    /// {
    /// /* 1 */ Console.WriteLine("true");
    /// /* 2 */ return;
    /// }
    /// Console.WriteLine("false");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void If0()
    {}
    /// <summary>Tests the If method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 50,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ if (value &lt; 0)
    /// {
    /// Console.WriteLine("true");
    /// return;
    /// }
    /// /* 1 */ Console.WriteLine("false");
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void If1()
    {}
    /// <summary>Tests the While method</summary>
    /// <remarks>
    /// <para>Test coverage (estimated): 100,0%</para>
    /// <para>Target path:</para>
    /// <code>/* 0 */ int num1 = 0;
    /// while ((num1 &lt; 10))
    /// {
    /// /* 1 */ Console.Write(num1++);
    /// }
    /// </code>
    /// </remarks>
    [Test()]
    [Ignore("NotImplemented")]
    public virtual void While0()
    {}
}

This post presents a new Reflector Addin that creates and diplays the graph of statement inside methods: the StatementGraph. This addin is an evolution of the IL graph. (this addin is not yet available for download).

 The statement graph is built from the Reflector CodeModel where each IStatement instance is a vertex and edges are added accordingly to the code "flow" (creating the edges is the most involved task). The rest of the job is handled by the QuickGraph library. When it makes sense, the vertices are clickable and you can jump to the invoked method, etc... The next step will be to update the Automatic Unit Test generator with the StatementGraph...

Let's see some simple methods and their corresponding graphs:

Simple

• Original code:

  public void Simple()
  {
      Console.WriteLine("hello");
  }
  

• Decompiled:

  public void Simple()
  {
        Console.WriteLine("hello");
  }

• Graph:

  

Two statements in sequence

• Original code:

  public void Body()
  {
      Console.WriteLine("hello");
      Console.WriteLine("world");
  }
  

• Decompiled:

  public void Body()
  {
        Console.WriteLine("hello");
        Console.WriteLine("world");
  }

• Graph:

  

If - Then - Else

• Original code:

  public void If(int value)
  {
      if (value<0)
          Console.WriteLine("true");
      else
          Console.WriteLine("false");
  }

• Decompiled:

  public void If(int value)
  {
        if (value < 0)
        {
              Console.WriteLine("true");
              return;
        }
        Console.WriteLine("false");
  }
  

• Graph:

  

While

• Original code:

  public void While()
  {
      int i = 0;
      while(i<10)
      {
          Console.Write(i++);
      }
  }

• Decompiled:

  public void While()
  {
        int num1 = 0;
        while ((num1 < 10))
        {
              Console.Write(num1++);
        }
  }

• Graph:

  

While with break and continue

• Original code:

  public void WhileBreakContinue()
  {
      int i = 0;
      while (i < 10)
      {
          if (i == 5)
              continue;
          if (i == 7)
              break;
          Console.Write(i++);
      }
      Console.WriteLine("Finished");
  }

• Decompiled:

  public void WhileBreakContinue()
  {
        int num1 = 0;
        while ((num1 < 10))
        {
              if (num1 == 5)
              {
                    continue;
              }
              if (num1 == 7)
              {
                    break;
              }
              Console.Write(num1++);
        }
        Console.WriteLine("Finished");
  }
  
  
  

• Graph:

  

Try - Catch

• Original code:

  public void TryCatch()
  {
      try
      {
          Console.WriteLine("hello");
      }
      catch (Exception ex)
      {
          Console.WriteLine("Boom: {0}",ex);
      }
  }
  

• Decompiled:

  public void TryCatch()
  {
        try
        {
              Console.WriteLine("hello");
        }
        catch (Exception exception1)
        {
              Console.WriteLine("Boom: {0}", exception1);
        }
  }

• Graph:

  

Just finished the private presentation of the PhD... One month to go and I'll get the degree :)

For the curious, here's a link to the presentation: http://blog.dotnetwiki.org/downloads/PhDPresentationdeHalleux.pdf

The "classic" TestFixture now supports a new type of test case: combinatorial tests. This fixture comes from an original idea of Jamie Cansdale, and it's design has been refined with the joint help of Jamie, Omer van Kloeten and the dev@mbunit.tigris.org mailing list.

A simple example

Suppose that we have a ICountX interface that defines a method that counts the number of 'x' in a string:

interface ICountX
{
   int Count(string s);
}

You have two classes that implement that interface:

class SickCountX : ICountX
{
   public int Count(string s)
   {
       return 2;
   }
}

class CountX : ICountX
{
   public int Count(string s)
   {
       int count = 0;
       foreach(char c in s)
           if (c=='x')
              count++;
       return count;
   }
}

Now you would like to test that those two interface implementations work correctly. Let's start by writing a test method inside a TestFixture. The CountIsExact method receives a string, the number of x in the string and checks that a provided ICountX implementation computes the expected x count:

[TestFixture]
public class ICountXTest
{
   public class StringX
   {
       public StringX(string value, int xcount){Value = value; XCount = xcount;}
       public string Value;
       public int XCount;
       public override string ToString() { return String.Format("{0},{1}",this.Value,this.XCount);}
   }

   [CombinatorialTest]
   public void CountIsExact(
      ICountX counter,
      StringX s
      )
   {
       Assert.AreEqual(s.XCount, counter.Count(s.Value));
   }
}

Next, we create a few strings and xcount that can serve for creating test case. By simplicity we use the new C# 2.0 iterators to implement that:

[Factory(typeof(StringX))]
public IEnumerable Strings()
{
    yield return new StringX("",0);
    yield return new StringX("x",1);
    yield return new StringX("xa",1);
    yield return new StringX("xax",2);
    yield return new StringX("aaa",0);
}

Note that we have tagged the method with Factory so that MbUnit knows this method creates StringX instances. We also create another method that will create instance of ICountX to test:

[Factory(typeof(ICountX))]
public IEnumerable Counters()
{
   yield return new SickCountX();
   yield return new CountX();
}

Now, we would like to cross product the output of Counters and StringX and feed the combination in the CountIsExact test method. To do so, we use the UsingFactories attribute to tag the parameters of the method:

[CombinatorialTest]
public void CountIsExact(
   [UsingFactories("Counters")] ICountX counter,
   [UsingFactories("Strings")] StringX s
)
{...}

The fixture is ready, so we can give it a run. The output of the test execution is as follows:

[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(,0))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(x,1))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.SickCountX),Strings(xax,2))
[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(aaa,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(,0))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(x,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xa,1))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xax,2))
[mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(aaa,0))
6 succeeded, 4 failed, 0 skipped, took 0,00 seconds.

It worked! As expected, we have 5 x 2 = 10 test case generated. Each test case name is generated out of the data source and different data values. Of course, you can "bind" data to the parameters using various other ways and the tests support more that 2 parameters as we will see in the following. We could also have added multiple Using Attributes.

Prequisites

This new features uses the TestFu.Operations framework in the background. This framework is detailed in the here (part 1) and here (part 2).

Rationale

The CombinatorialTest has the following workflow:

• for each parameter of the test method, get all the domains D of the parameter from the UsingAttributes (there are a bunch of those). This creates a collection of domain for each parameter which is also a domain itself, which we'll call parameter domain: PD = collection of D,
• foreach tuple in the CartesianProduct of the parameter domains
  • foreach ptuple in the PairWizeProduct of the domain in the tuple (each entry is a domain for the corresponding method parameters)
    • Create a test case that will invoke the test method with the values in ptuple.

Note that in the example above, PD = { { Counters }, { Strings } }, hence the Cartesian product has only one element {Counters} x {Strings} = (Counters,Strings). 

Custom Using Attributes

All Using attribute inherit from the abstract base class attribute UsingBaseAttribute which defines one abstract method:

public abstract class UsingBaseAttribute : Attribute
{
    public abstract void GetDomains(
        IDomainCollection domains, 
        ParameterInfo parameter,
        object fixture);
}

This method receives the tagged parameter, an instance of the fixture class and a collection of domains. It can add new domain to this fixture. For example, the following attribute, UsingLinear, generate a linearly spaced vector of integers:

public sealed class UsingLinearAttribute : UsingBaseAttribute
{
    private IDomain domain;
    public UsingLinearAttribute(int start, int stepCount)
    {
        this.domain = new LinearInt32Domain(start, stepCount);
    }
    public override void GetDomains(IDomainCollection domains, ParameterInfo parameter, object fixture)
    {
        domains.Add(domain);
    }
}

Here we have used the LinearInt32Domain shipped with TestFu.Operations and we have added that domain to the domain collection. Let's write a test that uses that attribute and see the result:

[CombinatorialTest]
public void UsingLinear(
    [UsingLinear(0, 10)] int i)
{
    Console.WriteLine(i);
}


--- output
[mbunit][Success] UsingLinear(0)
[mbunit][Success] UsingLinear(1)
[mbunit][Success] UsingLinear(2)
[mbunit][Success] UsingLinear(3)
[mbunit][Success] UsingLinear(4)
[mbunit][Success] UsingLinear(5)
[mbunit][Success] UsingLinear(6)
[mbunit][Success] UsingLinear(7)
[mbunit][Success] UsingLinear(8)
[mbunit][Success] UsingLinear(9)
10 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

Built in Using Attributes

MbUnit comes with a number of built-in using attribute that should get you started almost all situations.

UsingLinearAttribute

Already described previously.

UsingLiteralsAttribute

This attribute lets you specify a list of value separated by ';'. MbUnit will try to convert those to the parameter type using Convert.ChangeType method:

[CombinatorialTest]
public void UsingLinearAndLiterals(
    [UsingLinear(0, 3)] int i,
    [UsingLiterals("a;b")] char c
)
{
    Console.WriteLine("{0} {1}",i,c);
}
--- output
[mbunit][Success] UsingLinearAndLiterals(0,a)
[mbunit][Success] UsingLinearAndLiterals(0,b)
[mbunit][Success] UsingLinearAndLiterals(1,a)
[mbunit][Success] UsingLinearAndLiterals(1,b)
[mbunit][Success] UsingLinearAndLiterals(2,a)
[mbunit][Success] UsingLinearAndLiterals(2,b)
6 succeeded, 0 failed, 0 skipped, took 0,00 seconds.

UsingFactoriesAttribute

This attribute will look for factory method in a specified type. If the factory type is not specified, it will default it to the fixture type. If the member name is not specified, it will look for all the methods tagged with Factory attribute and compatible with the parameter type:

• Not specifying the member name, MbUnit looks for all the compatible factories:

  [Factory]
  public int GetZero()
  {
      return 0;
  }
  [Factory]
  public int[] GetZeroOne()
  {
      return new int[] { 0, 1 };
  }
  [Factory(typeof(int))] // we need to specify the factory type
  public IEnumerable GetZeroOneAsEnumerable()
  {
      return this.GetZeroOne();
  }
  [CombinatorialTest]
  public void UsingFactories(
      [UsingFactories] int i
      )
  {
      Console.WriteLine("{0}", i);
  }
  
  
  -- output
  
  [mbunit][Success] UsingFactories(GetZero(0))
  [mbunit][Success] UsingFactories(GetZeroOne(0))
  [mbunit][Success] UsingFactories(GetZeroOne(1))
  [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(0))
  [mbunit][Success] UsingFactories(GetZeroOneAsEnumerable(1))
  
  

• Using external factories by specifying a factory type:

  public class IntFactory
  {
      [Factory]
      public int One()
      {
          return 1;
      }
  }
  
  
  [CombinatorialTest]
  public void UsingFactory(
      [UsingFactories(typeof(IntFactory))] int i)
  {
      Console.WriteLine("{0}", i);
  }
  
  --- output
  
  [mbunit][Success] UsingFactory(One(1))
  1 succeeded, 0 failed, 0 skipped, took 0,00 seconds.
  

UsingImplementationsAttribute

This attribute looks for all the implementation of a type in an assembly, creates an instance and feeds it to the test.

[CombinatorialTest]
public void TestCountX([UsingImplementationsAttribute(typeof(ICountX))] ICountX countX)

What about tuple filtering ?

In some situation you may want to filter out tuple. For example, we want to check that a methods throws ArgumentNullException on null arguments:

[Factory]
public string[] Strings()
{
    return new string[] { null, "hello" };
}

[CombinatorialTest]
[ExpectedArgumentNullException]
public void TestWithValidator(
    [UsingFactories("Strings")] string s1,
    [UsingFactories("Strings")] string s2
)
{
    if (s1 == null)
        throw new ArgumentNullException();
    if (s2 == null)
        throw new ArgumentNullException();
}

(Note that ExceptedException and all the other decorator work on the combinatorial tests). If we run the test now, we wil have unwanted tuple such as (null, null) and ("hello", "hello"): the first does not give much information and the second will not throw and hence, is erroneous. In this kind of situation, you can specify a "filter" method that takes the same arguments as the test method and return bool:

public bool IsValid(string s1, string s2)
{
   if (s1 == null && s2 == null)
      return false;
   if (s1 != null && s2 != null)
      return false;
   return true;
}

[CombinatorialTest(TupleValidatorMethod = "IsValid")]
[ExpectedArgumentNullException]
public void TestWithValidator(...

--- output

[mbunit][Success] TestWithValidator(Strings(),Strings(hello))
[mbunit][Success] TestWithValidator(Strings(hello),Strings())

As expected, the unwanted tuple have been dropped.

Conclusion

Combinatorial test provides a flexible, extensible and powerfull way of generating test case. It implements the classic PairWize combinatorial generation in order to avoid exponential test explosion while integrating smoothly in the MbUnit architecture.

Reflector version: 4.1.2.0
Reflector.Graph version: 4.1.2.0
Download: www.dotnetwiki.org

Before getting the new version, make sure you have a look at the release notes below!

Important Changes:

• The graphs are now visualized in SVG by embedding Internet Explorer inside Reflector. Make sure you have a SVG viewer that works with IE installed on your machine. I strongly recommend Adobe SVG Viewer.
• The version number of the assemblies is now following Reflector version. Hence, current version number is 4.1.2.0.

Release Notes:

• The Assembly Graph and IL graph now have clickable nodes. In general, clicking on a vertex will move the selection to that element in the Reflector tree.
• The Assembly Graph now shows all the referenced even if they are not loaded. Click on a unloaded assembly will load it in Reflector,

Adobe SVG viewer tips:

If you are using Adobe SVG viewer, here are a few tips:

• Alt + left down + move move = panning the view,
• Ctrl + left click = zoom in,
• Ctrl + Shift + left click = zoom out,
• Ctrl + left click + mouse move = zoom region,
• Right click pops up the SVG viewer context menu.

Here's the new look of the Assembly graph.

In the previous post on combinatorial testing, I mentionned that Cartesian product was not a good solution when the number of domains and their dimension grew, because of obvious explosion of the number of tests.

A common approach to this problem is to consider a set of combination to create all the possible 2-tuples between the domains. It is called  PairWize or AllPairs suites generation. In TestFu.Operations, I decided to implement the algorithms found in [1]. Currently only the A1 algorithm is implemented. A2 and A4 should follow soon. Note that those algorithms are constructive, memory cheap and fast.

PairWize suite generation

The use of the PairWize algorithm is straightforward and follows the same semantics as the cartesian product. In the example below, we want to generate the allpairs for 3 domains of respective size 2,3,4.

int[] array1 = new int[] { 1, 2 };
char[] array2 = new char[] { 'a', 'b','c' };
string[] array3 = new string[] { "a", "combinatorial", "hello","world" };

int i = 1;
foreach (ITuple tuple in Products.PairWize(array1, array2, array3))
{
    Console.WriteLine("{0}: {1}",i++, tuple);
}

-- output
1: 1, a, a
2: 1, a, combinatorial
3: 2, a, a
4: 1, a, hello
5: 1, a, a
6: 1, a, world
7: 2, a, a
8: 2, b, a
9: 1, b, combinatorial
10: 2, b, combinatorial
11: 2, b, hello
12: 1, b, combinatorial
13: 2, b, world
14: 2, b, combinatorial
15: 1, c, a
16: 1, c, hello
17: 1, c, combinatorial
18: 2, c, hello
19: 1, c, hello
20: 1, c, world
21: 2, c, hello
22: 2, c, a
23: 1, c, world
24: 2, c, combinatorial
25: 2, c, world
26: 2, c, hello
27: 1, c, world
28: 2, c, world

If you wish to compare with the article, this is the output for a (k,l) = (7,3) and for the bipartite graphs A1 = {0,1,2,3} B1={4,5,6}, A2 = {0,4,2,3} B2={1,5,6} and A3 = {0,1,5,3} B3={4,2,6}. Remember that if we want to generate the test cases using the Cartesian product, we would have 3^7 = 2187 tests to run.

1: 0, 0, 0, 0, 0, 0, 0
2: 0, 0, 0, 0, 1, 1, 1
3: 1, 0, 0, 0, 0, 1, 1
4: 0, 1, 0, 0, 1, 0, 1
5: 0, 0, 0, 0, 2, 2, 2
6: 2, 0, 0, 0, 0, 2, 2
7: 0, 2, 0, 0, 2, 0, 2
8: 1, 1, 1, 1, 0, 0, 0
9: 0, 1, 1, 1, 1, 0, 0
10: 1, 0, 1, 1, 0, 1, 0
11: 1, 1, 1, 1, 1, 1, 1
12: 1, 1, 1, 1, 2, 2, 2
13: 2, 1, 1, 1, 1, 2, 2
14: 1, 2, 1, 1, 2, 1, 2
15: 2, 2, 2, 2, 0, 0, 0
16: 0, 2, 2, 2, 2, 0, 0
17: 2, 0, 2, 2, 0, 2, 0
18: 2, 2, 2, 2, 1, 1, 1
19: 1, 2, 2, 2, 2, 1, 1
20: 2, 1, 2, 2, 1, 2, 1
21: 2, 2, 2, 2, 2, 2, 2

Compairing results with AllPairs

For 10 domains with 10 values each, the TestFu algorithm generate 370 test suite. The AllPairs tool generate 177 which is roughly 50% less. This behavior was expected by the algorithm author. A greedy algorithm on the output could lower our result. Note that the tests were generate in 36.419ms which shows the efficiency of the algorithm.

References:

[1] Efficient Algorithms for Generation of Combinatorial Covering Suites, by Adrian Dumitrescu

TestFu has now limited support for generate test suites using Combinatorial Testing as the TestFu.Operations namespace. Combinatorial Testing is an interresting testing technique that has been studied a quite some time now (see citeseer search). There already exists a few package that provide tools to generate test suites (see jenny, allpairs) but none in the .Net framework.

Combinatorial Testing description and Glossary

In this chapter, I will define the different "actors" that interact in the combinatorial testing. I will link them to their corresponding interface in TestFu.

• A domain (IDomain) is a finite set of objects.
• Ex: {1,2,3} is a domain composed of the integers 1,2,3
• The boundary of a domain D is the set of objects in D that are at the boundary of the domain.  The boundary is also a domain,
• If we decide that the first and last element of an array is it's domain, {1,3} is the boundary of {1;2,3}
• A tuple (ITuple) acts as a container for other objects,
• A product on a domain collection (IDomainCollection) is an algorithm (ITupleEnumerable) that generate a suite of tuples (ITupleEnumerator), created by combining one element of each domain, using a predefined strategy.

Currently, the only product available is the cartesian product which returns an exhaustive enumeration of the all possible combination. It is well-known that this product is not efficient at all covering the tests because of the exponential explosion of the number of suites, better strategy have been developped such as the PairWaise covering (all pairs), t-wize, etc... I haven't had the time to look at those yet.

Let's see some code

The TestFu.Operations namespace is designed to be simpler but no simpler. Let's start by defining some domains:

int[] array1 = { 1, 2 };
string[] array2 = { "a", "combinatorial", "hello", "world" };
char[] array3 = { 'a', 'b', 'c' };

Note: TestFu.Operations currently supports arrays, collection (ICollection), enumerable collection (IEnumerable) or singleton domain (1 element). Of course, this is totally extendible. You could easily think about a domain reading the rows of a DataTable, or the elements of a XmlDocument

The static class Products contains a rich set of methods that can create the different products. Let's start with the cartesian product of array1 x array2:

int i = 1;
foreach (ITuple tuple in Products.Cartesian(array1, array2))
{
    Console.WriteLine("\t{0}: {1}",i++,tuple);
}

-- output
        1: 1, a
        2: 1, combinatorial
        3: 1, hello
        4: 1, world
        5: 2, a
        6: 2, combinatorial
        7: 2, hello
        8: 2, world

As expected, each returned tuple is a collection of element from each domain. We could also decide to test the boundaries of the arrays only (first and last element each time):

i = 1;
foreach (ITuple tuple in Products.BoundaryCartesian(array1, array2))
{
    Console.WriteLine("\t{0}: {1}", i++, tuple);
}

-- output
        1: 1, a
        2: 1, world
        3: 2, a
        4: 2, world

The products can act on an arbitrary number of domains so we can add array3 in the product (don't forget about the explosion of the tests number!):

i = 1;
foreach (ITuple tuple in Products.Cartesian(array1, array2, array3))
{
    Console.WriteLine("\t{0}: {1}", i++, tuple);
}

-- output
        1: 1, a, a
        2: 1, a, b
        3: 1, a, c
        4: 1, combinatorial, a
        5: 1, combinatorial, b
        6: 1, combinatorial, c
        7: 1, hello, a
        8: 1, hello, b
        9: 1, hello, c
        10: 1, world, a
        11: 1, world, b
        12: 1, world, c
        13: 2, a, a
        14: 2, a, b
        15: 2, a, c
        16: 2, combinatorial, a
        17: 2, combinatorial, b
        18: 2, combinatorial, c
        19: 2, hello, a
        20: 2, hello, b
        21: 2, hello, c
        22: 2, world, a
        23: 2, world, b
        24: 2, world, c

Where to go from here ?

The next important step is to implement an algorithm that computes the pair-wize or t-wize suite generation. This is very important if you have a lot of domains. In a near future, I will also show how we can use these products to produce a new type of fixture in MbUnit. This fixture was suggested by Jamie Cansdale.

In this post, I have showed a new, simple, way of creating combinatorial tests using the TestFu.Operations namespace.

Yesterday, Jamie Cansdale (NunitAddin) and I spent time "pair-programming" on MbUnit, NUnitAddIn and other tools like the Reflector.Graph addins. The collaboration has been very productive! I've got a lot of blog entry to write about the things we have built yesterday, I'll summarize quickly:

• MbUnit has now a much better support for NUnitAddIn:
  • this solution should be "version" proof: it should not break if NUnitAddIn updates and so on,
  • you can select a method and execute the tests involving that method,
  • MbUnit generates the Html report and puts the URL in the output, ready for you
• MbUnit has new fixtures! CrossFixture generates generate the cross products of different data source and hands it to the test...
• Reflector.Graph renders to SVG: we are now hosting IE inside Reflector to display SVD graphs...
• Reflector.Graph graphs has clickable links! You can click on an assembly to load it, and so on...

A lot of work, a lot of fun...