The Abstract Test Pattern (ATP)

I have received a few comments on my blog entry on Composite Unit Testing (CUT) arguying that this was the Abstract Test Pattern . Here's a snapshot of the definition from the definition form http://c2.com/cgi/:

A Testing Pattern describing a way to reuse test cases for multiple implementations of an Interface.

Problem

How to write a Test Suite against an Interface (or Abstract Class) that can be used to test all implementations of the interface.

Solution

• Write an AbstractTest  for every Interface and Abstract Class). The AbstractTest should have an abstract FactoryMethod that creates an object with the type of the Interface.
• Write a ConcreteTest for every implementation of the Interface. The ConcreteTest? should be a descendant of the AbstractTest and override the FactoryMethod to construct an instance of the implementation class.

Functional Compliance

Eric George's article gives a more detailled description of the pattern and describes it as functional compliance. It is easy enough for the compiler to tell whether a class is syntactically compliant with an interface. It applies a check to see if all required methods have been implemented with the correct signatures (syntaxic compliance), but the compiler cannot check functional compliance of a class with its interface. Here's the formal definition given by Eric George:

Functional Compliance is a module's compliance with some documented or published functional specification. The specification can be purely documentational, or it can be partially enforced through Interfaces or Abstract Classes. Interfaces and Abstract Classes along with their associated documentation represent a contract between the implementation code and the client (or user) code. It is this contract that needs to be fully tested. The Liskov Substitution Principle (LSP) tells us that all modules that honor a contract (usually by implementing an interface), should behave the same from the perspective of the client code. A module's functional compliance is really the degree to which it obey's the LSP.

So what about Composite Unit Testing ?

The remarks from the readers were right. Composite Unit Testing is

• an enhanced form of the Abstract Test Pattern,
• is a tool to test functional compliance

There is, however, a major difference between ATP and CUT: separation of the test code and the factory methods. In AUT, you create a ConcreteTest that inherits AbstractTest and implements a factory method, so the code that generates the tested entity is "hard-coded" into concrete test. In CUT, the framework takes care of retreiving and feeding you AbstractTest using user-specified factories (you can easily have multiple factories):

// AUT
// abstract method
public abstract class AbstractEnumerableTest
{
    public IEnumerable Create();
    public void GetEnumeratorTest()
    {
        IEnumerable en = this.Create();
        ...
    }
}

// concrete implementation
[TestFixture]
public class ArrayListEnumerableTest
{
    public override IEnumerable Create()
    { return new ArrayList();}
}

The same test as above, using CUT:

// the fixture
public class EnumerableFixture
{
    public void GetEnumeratorTest(IEnumerable en)
    {
        ...
    }
}

// the factories
public class ArrayListFactory
{
    public ArrayList Emtpy
    { get{ return new ArrayList();}}
}

// link the fixture with the factories
[CompositeFixture(typeof(EnumerableFixture), typeof(IEnumerable))]
[ProviderFactory(typeof(ArraListFactory),typeof(IEnumerable))]
public class EnumerableTest
{}

The following article on CodeProject talks about Scarified Treemaps, an interresting tree visualization. I wonder what it would look like in MbUnit...

Here's a preview of a new fixture that will be available in the next release of MbUnit. The fixtures loads XML data from files and feeds it to the different methods, hence it's Data Driven Unit Testing. I'll make a real article on this when the fixture stabilizes but here's a first example.

[DataFixture]
[XmlDataProvider("../../sample.xml","//User")]
[XmlDataProvider("../../sample.xml","DataFixture/Customers")]
public class DataDrivenTests
{
    [ForEachTest("//User")]
    public void ForEachTest(XmlNode node)
    {
        Assert.IsNotNull(node);
        Assert.AreEqual("User",node.Name);
        Console.WriteLine(node.OuterXml);
    }

    [ForEachTest("//User",DataType = typeof(User))]
    public void ForEachTestWithSerialization(User user)
    {
        Assert.IsNotNull(user);
        Console.WriteLine(user.ToString());
    }
}

The file sample.xml looks like this:

<DataFixture>
  <Employees>
    <User Name="Mickey" LastName="Mouse" />
  </Employees>
  <Customers>
    <User Name="Jonathan" LastName="de Halleux" />
    <User Name="Voldo" LastName="Unkown" />
  </Customers>
</DataFixture>

and the User class like this:

[XmlRoot("User")]
public class User
{
    private string name;
    private string lastName;
    public User()
    {}
    [XmlAttribute("Name")]
    public String Name
    {
    get{ return this.name;}
    set{ this.name = value;}
    }
    [XmlAttribute("LastName")]
    public String LastName
    {
    get{ return this.lastName;}
    set{ this.lastName = value;}
    }
}

How does it work ?

• The XmlDataProvider attributes are used to specify an XML filename that contains the data (of course, you can put several of those).
• Then a first XPath expression is applied to the data.
• Each selected node is then feeded to the ForEachTest which applies a second XPath expression on the node. This allows a fine grained selection of the data.
• You can also specify the desired output data type. XmlSerializer is then used to convert XmlNode into the desired object.

Screenshot

Here's a screenshot that shows have MbUnit loads the XML and creates the different test case.

This is a copy of an upcoming CodeProject article (number 36)

Introduction

This article presents a new way of creating unit tests. Rather than creating a fixture for each class, we split the testing effort by class functionality. Taking advantage of interface composition, we use split the unit tests for each interface and we feed those fixtures using factories. This is why I call this technique: Composite Unit Testing.

There are several advantages of using this approach:

1. Expressing requirements: interfaces are a natural place for expressing requirements, which later on translate into unit tests,
2. Test reusability: once you have design a test suite for an interface, you can apply it to any class that implements this interface,
3. Test Driven Developement: this approach fits nicely and naturally into the TDD paradigm since execution path is interface -> interface fixture -> implementation(s).
4. Separation of tests and tested instance generation: the code that generates the tested instances is located in factories that can be reused for each fixtures.

In the rest of the article, I will illustrate this technique on ArrayList and Hashtable.

Test Case

Let us consider illustrate the process with two classes of the System.Collections namespace: ArrayList and Hashtable.

The two classes belong to different families of containers: ArrayList is a sequential container, while Hashtable is an associative container. However, as the interface diagram below shows, they share a lot of  functionalities (enumeration, cloneable, serialization, etc...). These functionalities are usually represented by interface composition: ICloneable, ISerializable, etc... The interface define functionalities and requirements on those functionalities.

If you take the usual unit testing methodology, you will need to write two (huge) fixture to test the two classes. Since they share functionality, you will end up duplicating testing code, maintenance problem will increase, etc...

Composite unit testing provides a flexbile solution to those problems. In the following, I will illustrate how it is implemented in MbUnit.

Composite Unit Testing Methodology

As mentionned in the introduction, composite unit testing fits naturally in the TDD idea. The principal steps of the process are:

1. create the interface and express requirements,
2. create the interface fixture and translate the requirements into unit tests,
3. implement the interface,
4. create a class factory that provides instances of the interface,
5. link fixture to factories and run...

Step 1: Create the interface

This is where you define the functionalities and the requirements. If the documentation is clear enough, it should translate naturally into unit tests. (In this example, the job is already done).

Step 2: Create the interface fixture (for IEnumerable)

MbUnit defines a new custom attribute TypeFixture that is used to create fixture for types (classes, structs or interface). TypeFixture constructor take the type that is tested as argument. Let us start with the fixture of IEnumerable:

using System;
using System.Collections;
using MbUnit.Core.Framework;
using MbUnit.Framework;
[TypeFixture(typeof(IEnumerable))]
public class EnumerableTest
{}

The test case in EnumerableTest will receive an instance of the tested type (IEnumerable here) as argument. Therefore, the correct signature of those methods is as follows:

[TypeFixture(typeof(IEnumerable))]
public class EnumerableTest
{
    [Test]
    public void EmptyTest(IEnumerable en)
    {...}
}

The argument is the only difference with the "classic" unit test. You can use test decorators like ExpectedException, Ignore, etc... as usual. IEnumerable defines one method, GetEnumerator. The only requirement is that the IEnumerator instance is not a null reference:

[TypeFixture(typeof(IEnumerable))]
public class EnumerableTest
{
    [Test]
    public void GetEnumeratorNotNull(IEnumerable en)
    {
        Assert.IsNotNull(en.GetEnumerator());
    }
}

That's pretty short but there is nothing else to test. If you want to test the enumeration, you need to write another fixture for IEnumerator. By defining fixtures for each interface you quickly increase the coverage of the tested code.

Step 4: Create the factories

(We have skipped step 3, the implementation step)

A factory is simply a class that defines public properties or method (with no arguments) that return an object to be tested. You can use factories to provide different flavor of the same class: an empty ArrayList, randomly filled, ordered filled, etc... For example, a possible factory for ArrayList is:

public class ArrayListFactory
{
    public ArrayList Empty
    {
        get
        {
            return new ArrayList();
        }
    } 
    public ArrayList RandomFilled()
    {
        ArrayList list = new ArrayList();
        Random rnd = ...;
        for(int i=0;i<15;++i) 
            list.Add(rnd.Next());
        return list;
    } 
}

Note that a factory does not need any particular attributes. Similarly we can define HashtableFactory.

Step 5: Linking the fixtures to the factories

Linking the factories to the fixtures is simply done by using another custom attribute: ProviderFactory.

[TypeFixture(typeof(IEnumerable))]
[ProviderFactory(typeof(ArrayListFactory),typeof(IEnumerable))]
[ProviderFactory(typeof(HashtableFactory),typeof(IEnumerable))]
public class EnumerableTest
{...}

ProviderFactory takes the type of the factory, and the tested type as argument. The framework will take care of exploring by reflection the factories, select the suitable properties and feed the fixtures with created test instances.

Step 6: Running the tests

The full source of the example is available in the demo project. You need to create a new C# assembly project and add the reference to MbUnit.Core.dll and MbUnit.Framework.dll, which you can download from the MbUnit web site: http://mbunit.tigris.org. /

Launch MbUnit.GUI and load the assembly (right click -> Assemblies -> Add Assemblies...). Here are some screenshots of the application:

Conclusion

This article has presented composite unit testing, a new strategy for designing and implementing unit testing. Awaiting comments :)

Just finished a new CodeSmith template for generating a MockObject out of an interface. The template is pretty basic: it loads the type, create the methods/properties of the different interfaces. You can also specify a number of expected values and the template will generate the SetXXX methods.

Located in the MbUnit CVS (Templates folder).

Here's a sample C# application that implements structured numbers. (Practial use of this number is unknown :) )

The maths

The StrutucredNumbers sample contains an implementation of the binary tree operation developped by V. Blondel in Structured Numbers, Properties of a hierarchy of operations on binary trees, Acta Informatica, 35, 1-15, 1998.

We introduce a hierarchy of operations on (finite and infinite) binary trees. The operations are obtained by successive repetition of one initial operation. The ¯rst three operations are generalizations of the operations of addition, multiplication and exponentiation for positive integers.

In the paper, the author defines countably many internal operations on binary trees. The first operation, which he denote by .1. , is obtained by forming the binary tree whose left and right subtrees are equal to the operands. This operation is not associative. The second operation .2. is defined as follows: From the binary trees a and b we construct the binary tree a.2.b by repeating the operation .1. on the tree a with the structure dictated by b. In the same way, we define an operation .3. by repeating .2. , an operation .4. by repeating .3. , etc. We eventually obtain countably many internal operations ( .k. for k>= 1) with the definition

a .k. b = a^(k-1).a^(k-1).a^(k-1)...a^(k-1).a,

where there are b factors.

The code

The code is mainly composed of the BtNode that implements the algebra defined above and some valuators for the tree. At last, NGraphviz is used to render to trees :)

The results

Suppose that we have

x = ..
y = ..

• x:
• x.1.y:
• x.2.y:
• x.3.y:
• x.4.y:

In this "episode", we are going to build an application that generates and renders the exection graph of a method using QuickGraph and Lutz Reoder's IlReader. The execution graph is directed graph where each vertex is an IL instruction and each edge represent the transition between two instructions, it could be a jump or a simple move to the next instruction. The graph represents the different path that the application can take into your method. If you have access to the content of a method, you can potentially build.

In a future blog, I will use the execution graph to make "smart" test fixture generator by applying basic graph algorithms on the execution graph. In the following of the blog, I assume you have some basic knowledge of IL, Reflection and Reflection.Emit.

Step 1: creating the graph structures

We begin by creating a specialized type of vertex InstructionVertex that will hold a System.Reflection.Emit.Instruction instance:

using System.Reflection.Emit;
public class InstructionVertex : QuickGraph.Vertex
{
    private Reflector.Disassembler.Instruction instruction=null;
    public InstructionVertex(int id):base(id)
    {}
    public Reflector.Disassembler.Instruction Instruction
    {
        get
        {
            if (this.instruction==null)
                throw new InvalidOperationException();
            return this.instruction;
        }
        set
        {
            this.instruction = value;
        }
    }
    public override string ToString()
    {...}
}

Note that the ToString method uses the code Example.cs in the IlReader source to render an Instruction to string.

To generate a strongly-typed BidirecitonalGraph, that we call InstructionGraph, we use the CodeSmith template called AdjacencyGraph.cst located in the Templates directory.

Step 2: Loading the IL instructions of the method

The building of the graph is done by the IlGraphBuilder class. This class takes the type of the method as an argument. It contains one public method BuildGraph that takes a MethodInfo instance as argument as outputs a InstructionGraph instance containing the execution graph.

The code to extract the MethodBody from the method is almost entirely copy pasted from the IlReader sample:

public class IlGraphBuilder
{
    private Type visitedType;
    private ModuleReader reader;
    public IlGraphBuilder(Type visitedType)
    {
        this.visitedType = visitedType;
        this.reader = new ModuleReader(this.visitedType.Module, new AssemblyProvider());
    }
    public InstructionGraph BuildGraph(MethodInfo mi)
    { 
        // get method body using IlReaderr
        MethodBody methodBody = reader.GetMethodBody(mi);
        ...
    }

    private sealed class AssemblyProvider : IAssemblyProvider
    {
        public Assembly Load(string assemblyName)
        {
            return Assembly.Load(assemblyName); 
        }
        public Assembly[] GetAssemblies()
        {
            throw new NotImplementedException(); 
        }
    }
}

Step 3: Building the graph

The MethodBody instance contains the list of Instruction instances and the list of exception handlers. The building of the graph is done in 3 steps:

1. iterate the Instruction list and create the vertices in the graph. We also build a dictionary that associates the instruction offset to the corresponding vertex,

   // new field in the class
   Hashtable instructionVertices;
   ...
   foreach(Instruction i in methodBody.GetInstructions())
   {
       // avoid certain instructions
       if (i.Code.FlowControl == FlowControl.Phi || i.Code.FlowControl == FlowControl.Meta)
           continue;
       // add vertex
       InstructionVertex iv = g.AddVertex();
       iv.Instruction = i;
       // store in hashtable
       this.instructionVertices.Add(i.Offset,iv);
   }
   

2. iterate again over the instructions and add the edges that represent the transitions. This is the tricky part, I use a recursive exploration of the instructions, (this part of the code is a bit heavy for the blog)
3. iterate over the exception handlers to link the different sections: create the link to catch,finally handlers, etc...

Step 4: Drawing the graph

Drawing the graph is straight-foward using the GraphvizAlgorithm class:

GraphvizAlgorithm gv = new GraphvizAlgorithm(g); 
gv.Write(mi.Name);

Analysing some basic flow contructions:

As an application, I going to show the graph of some basic instruction flow like for, while, if, foreach, etc...

public void HelloWorld()
{
    Console.WriteLine("Hello World");
}

public void IfAlone(bool value)
{
    if (value)
        Console.Write("value is true");
}

public void IfThenElse(bool value)
{
    if (value)
        Console.Write("value is true");
    else
        Console.Write("value is false");
}

public void For()
{
    for(int i = 0;i!=10;++i)
    {
        Console.Write(i);
    }
}

public void While()
{
    int i = 0;
    while(i!=10)
    {
        i++;
    }
}

public void TryCatchFinally()
{
    try
    {
        Console.WriteLine("try");
    }
    catch(Exception)
    {
        Console.WriteLine("catch"); 
    }
    finally
    {
        Console.WriteLine("finally");
    }
}

public void TryMultiCatchFinally()
{
    try
    {
        Console.WriteLine("try");
    }
    catch(ArgumentException)
    {
        Console.WriteLine("catch(arg)"); 
    }
    catch(Exception)
    {
        Console.WriteLine("catch"); 
    }
    finally
    {
        Console.WriteLine("finally");
    }
}

public void ForEach(ICollection col)
{
    foreach(Object o in col)
    {
        Console.WriteLine(o);
    }
}

public void ForEachContinue(int[] col)
{
    foreach(int o in col)
    {
        Console.WriteLine(o);
        if (o == 0)
            continue;
    }
}

public void ForEachBreak(int[] col)
{
    foreach(int o in col)
    {
        Console.WriteLine(o);
        if (o == 0)
            break;
    }
}

public void SwitchIt(int value)
{
    switch(value)
    {
    case 0:
        Console.Write("0");
        break;
    case 1:
        Console.Write("1");
        break;
    default:
        Console.Write("default");
        break;
    }
}

I have just finished a CodeSmith template that "automatically" generates empty tests case for a given type.

This template does a little bit more than just enumerating the available methods: it explores MSIL using RAIL and creates test case following the given rules:

• if the IL instruction is a conditional branch, create a test case for both possibilities (true/false case)
• if the IL instruction throws, create a test case expecting the exception,
• if the IL instruction is returning, create a test case that checks the returned value

Those rules are quite simplistic but can already generate a "huge" number of test case automatically. Each test case comes with a piece of documentation and the method IL code where the target instruction has been set in bold. Currently, the documentation contains IL code, but in the future it would be nicer to output real code, using Reflector for example.

In order to make the template work, you need to put RAIL assemblies and the AssemblyHelper assembly in the CodeSmith directory.

Here's a quick example. The method:

public void ABitMoreComplext(bool goForIt)
{
    if (goForIt)
        throw new Exception();
    else
        Console.Write("did not throw");
        // other instruction 
    Console.Write("hello");
}

and the resulting output in the generated class:

        
        /// <summary>
        /// Tests ABitMoreComplext method when condition is executed as true
        /// (see remarks).
        /// </summary>
        /// <remark>
        /// <para>
        /// Not implemented
        /// </para>
        /// <code>
        /// ldarg.1
        /// <b>brfalse.s</b>
        /// newobj
        /// throw
        /// ldstr
        /// call
        /// ldstr
        /// call
        /// ret
        /// </code>
        /// </remarks>
        [Test]
        [Ignore]
        public void ABitMoreComplextIfTrue1()
        {
            throw new NotImplementedException();
        }
        /// <summary>
        /// Tests ABitMoreComplext method when condition is executed as true
        /// (see remarks).
        /// </summary>
        /// <remark>
        /// <para>
        /// Not implemented
        /// </para>
        /// <code>
        /// ldarg.1
        /// <b>brfalse.s</b>
        /// newobj
        /// throw
        /// ldstr
        /// call
        /// ldstr
        /// call
        /// ret
        /// </code>
        /// </remarks>
        [Test]
        [Ignore]
        public void ABitMoreComplextIfFalse1()
        {
            throw new NotImplementedException();
        }
        /// <summary>
        /// Tests that the ABitMoreComplext method throws (see remarks)
        /// </summary>
        /// <remark>
        /// <para>
        /// Not implemented
        /// </para>
        /// <code>
        /// ldarg.1
        /// brfalse.s
        /// newobj
        /// <b>throw</b>
        /// ldstr
        /// call
        /// ldstr
        /// call
        /// ret
        /// </code>
        /// </remarks>
        [Test]
        [Ignore]
        [ExpectedException(typeof(Exception))]
        public void ABitMoreComplextThrow3()
        {
            // don't forget to update the exception type.
            throw new NotImplementedException();
        }

ps: The template is called TestFixture and is in the Templates directory in the MbUnit CVS.

A co-worker, Jacques Theys, sent me this link:

"GPGPU stands for General-Purpose computation on GPUs. With the increasing programmability of commodity graphics processing units (GPUs), these chips are capable of performing more than the specific graphics computations for which they were designed. They are now capable coprocessors, and their high speed makes them useful for a variety of applications. The goal of this page is to catalog the current and historical use of GPUs for general-purpose computation."

MbUnit has a Visual Studio Add-In using the NUnitAddIn framework. Setting the framework is done through the following steps:

1. Get the latest NUnitAddIn installed on your machine,
2. Copy the MbUnit binaries to the NUnitAddIn folder,
3. Edit the NUnitAddIn.config file as follows:

   <?xml version="1.0"?>
   <configuration>
     <nunitaddin>
       <frameworktestrunners>
         <testRunner name="MbUnit"
                        typeName="MbUnit.AddIn.MbUnitTestRunner" 
                        assemblyPath="MbUnit.AddIn.dll"  />
         <testRunner name="NUnit"
                        typeName="NUnitAddIn.NUnit.TestRunner.SimpleNUnitTestRunner" 
                        assemblyPath="NUnitAddIn.NUnit.dll"  />
         ...
       </frameworktestrunners>
     </nunitaddin>
   </configuration>

That's it. You can now right click on an assembly, namespace or fixture and execute it using "Run Tests...".