With the kind help of Lutz Roeder, I have recompiled the IL Grapher into a Reflector Add-in...

In a near future, MbUnit will support storing of the test results in a SQL database (SQL server supported). The database structure is done and the BLL/DAL is finished. Here's a snapshot of the database schema.

I have added the possibility to sort fixture by importance (or severity): Critical, Severe, Default, NoOneReallyCaresAbout.

[TestFixture]
[Importance(TestImportance.Critical)]
public SomeFixture
{...}

A number of new assertions classes have been added to MbUnit since the latest post on this topic.  The new helper classes involve Arrays, collection, compiler, serialization, web...

ArrayAssert

This helper class contains method to compare arrays:

byte[] expected = ...;
byte[] actual = ...;
ArrayAssert.AreEqual(expected,actual);

The method compares the rank, the length and makes element-wize comparaison.

ColAssert

This class provides several methods to compare two ICollection instance:

ICollection expected = ...;
ICollection actual = ...;

ColAssert.IsSynchronized(actual);
ColAssert.AreCountEqual(expected,actual);
ColAssert.AreEqual(expected,actual);

The class also provides method to test the collection count, syncroot, synchronization, etc...

SerialAssert

This class contains various methods to test the "serializability" of objects.

SerialAssert.IsXmlSerializable(typeof(MyClass));

WebAssert

This class contains assertions on the properties of web control and web pages.

CompilerAssert

This class contains assertions to check that snippets are compilable:

String source = ...; // C# code to compile
// verify that source compiles
CompilerAssert.Compiles(CompilerAssert.CSharpCompiler, source);

What about your assertions ?

There is also a CodeSmith template that can let you build "strongly-typed" assertion classes out of existing types. See in the templates directory.

I just came back from Microsoft, Redmond where I attented interviews for SDE/T in the CLR team. (Interviews at Microsoft is a unique experience on its own). It looks like it worked because I'm moving in October to Redmond. :) I would like to thank Michael Corning, Harry Robinson and Holly Barbacovi for their support on this adventure.

Me and Michael Corning at Building 44 in Redmond.

Don't be fooled by the bad quality of the photo, it was pooring rain (the famous Redmond weather).

I've been recently interrested into Mutation Testing, a funny way of measure the quality of tests. This blog presents the first snapshot of a toy application that mutates any .Net program.

Mutation testing is the action of inserting "articifial" faults into the Instance Under Test and look if the tests catch this fault. The idea is that the tests are adequate if they detect all the faults. For example, a typical mutation is to negate the condition expression in a if statement:

//original
if (condition)
   DoSomething();
// mutated
if (!condition)
   DoSomething();

Jester implements Mutation testing for JUnit (there is a nice article here about Jester). In his Thesis (that Lutz Roeder kindly pointed out to me), A Multation Testing Tool for Java Programs, Matthias Bybro defines an entire framework for generating and executing mutants. In this blog, I will not focus on the theory of mutation testing but I'll show how you can get it implemented in .NET

The tools we need

As usual, before attacking the problem we can review what functionalities we need and what we have on our tool set. In this case, we need the ability to load an assembly, explore and alter the IL, and execute or write the mutated assembly. Got any idea....

RAIL! Runtime Assembly Instrumentation Library, that's exactly what we need. With RAIL, you can load an assembly, explore and alter the IL and execute or write the mutated assembly. You can even substitute types or entire functions. In fact, the powerpoint presentation of RAIL, the author shows how to play with IL.

Let's code

The AssemblyScrambler application is designed as follows: a ScramblerEngine instance contains a collection of IScrambler instances. An IScrambler instance contains a method to scramble IL code (I started this application before knowing about mutation testing. So Scrambler should be named mutators, etc...):

public interface IScrambler
{
    void Scramble(ScrambleTrace trace,RMethodDef method);
}

where trace is used to log mutations, and method is an instance of Rail.Reflect.RMethodDef which represents a method. The scramblers are used as follows in ScramblerEngine:

public void Scramble(string fileName)
{
    this.assembly = RAssemblyDef.LoadAssembly(fileName);
    foreach(RTypeDef t in this.assembly.RModuleDef.GetTypes())
    {
        foreach(RMethodDef method in t.GetMethods())
        {
            foreach(IScrambler scrambler in this.Scramblers)
            {
                scrambler.Scramble(trace,method);
            }
        }
    }
}

We are now ready to start implementing scramblers. There is currently only one implemented that swithes brtrue -> brfalse and brfalse -> brtrue. RMethodDef contains a MethodBody that contains a Code instance. Code is a mutable collection of instructions:

for(int i = 0;i<method.MethodBody.Code.InstructionCount;++i)
{
    Instruction il = method.MethodBody.Code[i];
    // selecting instruction
    if (il.OpCode.OperandType != OperandType.InlineBrTarget 
    && il.OpCode.OperandType != OperandType.ShortInlineBrTarget)
        continue;
    // il is ILBranch
    ILBranch branch = (ILBranch)il;
    // check if is brfalse
    if (il.OpCode.Name == OpCodes.Brfalse.Name)
    {
        // subtitute with brtrue
        method.MethodBody.Code[i]=new ILBranch(OpCodes.Brtrue,branch.Target);
    }
    else ...

Switching brtrue and brfalse is as simple as that. Note that here, 99% percent of the work is done by the excellent RAIL library.

Small example

Let's apply the scrambler to a small method:

public void IsTrue(bool isTrue)
{
    Console.Write("Expected: {0}, ",isTrue);
    if (isTrue)
        Console.WriteLine("Actual: true");
    else
        Console.WriteLine("Actual: false");
}

The IL code for this method is the following (using Reflector):

.method public hidebysig instance void IsTrue(bool isTrue) cil managed
{
// Code Size: 42 byte(s)
.maxstack 2
L_0000: ldstr "Expected: {0}, "
L_0005: ldarg.1 
L_0006: box bool
L_000b: call void [mscorlib]System.Console::Write(string, object)
L_0010: ldarg.1 
L_0011: brfalse.s L_001f
L_0013: ldstr "Actual: true"
L_0018: call void [mscorlib]System.Console::WriteLine(string)
L_001d: br.s L_0029
L_001f: ldstr "Actual: false"
L_0024: call void [mscorlib]System.Console::WriteLine(string)
L_0029: ret 
}

You can see that instruction at index 0011 is what we target. We have a small console application that calls this method. The code and results are:

Sandbox sandbox = new Sandbox();
sandbox.IsTrue(true);
sandbox.IsTrue(false);
-- output
Expected: True, Actual: true
Expected: False, Actual: false

After mutation

The above method is passed into the AssemblyScrambler machine, the IL code of the mutated application now looks like this:

.method public hidebysig instance void IsTrue(bool isTrue) cil managed
{
// Code Size: 42 byte(s)
.maxstack 3
L_0000: ldstr "Expected: {0}, "
L_0005: ldarg.1 
L_0006: box bool
L_000b: call void [mscorlib]System.Console::Write(string, object)
L_0010: ldarg.1 
L_0011: brtrue.s L_001f
L_0013: ldstr "Actual: true"
L_0018: call void [mscorlib]System.Console::WriteLine(string)
L_001d: br.s L_0029
L_001f: ldstr "Actual: false"
L_0024: call void [mscorlib]System.Console::WriteLine(string)
L_0029: ret 
}

Take a look now at L_0011, it is now brtrue.s.... the method is mutated. In fact, the output of the snippet gives:

Expected: True, Actual: false
Expected: False, Actual: true

In the post Fun with Graphs (3): Creating the graph of a database structure, I have presented a small application that creates the graph of a database. We are now going to improve the output by adding the different fields, primary keys, etc.. in the graph.

GraphvizRecordCell

Graphviz supports a type of vertex shape that is drawed as nested tables. This shape is called Record. NGraphviz comes with a class wrapper (GraphvizRecordCell) that lets you easily create such records. Some remarks on cells:

• A GraphvizRecordCell can also contain other nested cells,
• By default, Graphviz starts to arrange the cells horizontally and swith direction (vertical/horizontal) at each level 

Let's take the formatVertex event handler and adapt it to create records:

private void formatVertex(Object sender, FormatVertexEventArgs e)
{
    TableSchemaVertex v = (TableSchemaVertex)e.Vertex;
    GraphvizRecord record = new GraphvizRecord();
    e.VertexFormatter.Shape = GraphvizVertexShape.Record;
    e.VertexFormatter.Record = record;
    GraphvizRecordCell table = new GraphvizRecordCell();
    record.Cells.Add(table);

    GraphvizRecordCell name = new GraphvizRecordCell();
    name.Text = v.Table.Name;
    table.Cells.Add(name);
    ...

Here's a sample result on the MbUnit database:

Over the last few days, I have started to prepare MbUnit to support loading of test assemblies into separate domain. This feature is very important for a number of reasons:

• test assemblies are shadow copied,
• test assemblies can be unloaded. This means that MbUnit can detect when you have recompile the test assembly and reload it.The assembly unloading feature is very important if you plan to do Test Driven Development (test, code, test, code...).
• it is easier to control the AssemblyResolve event,

Of course, executing the tests in separate AppDomain has a big drawback: test results and notifications is transmitted by Remoting, and this cost cpu cycles. Currently there is a big performance hit (twice slower) for using separate AppDomain. A possible explanation is that there too much event notification that need to cross Remoting channel.

To be continued...

Sorting the fixture using the namespace/type is nice... but like always, there situations when you would like to sort fixture using other criteras. For example, you might want to sort the test by authors, categories, importance, etc...

While preparing for AddDomain remoting, I have totally refactored the way MbUnit populates the tree to make it totally extensible: now you can populate the anyway you like!

FixtureCategoryAttribute

This is a new attribute that can tag fixture to sort them by categories. You can describe a nested category by separting the names by dots (like a namespace) and you can tag a fixture with multiple categories (a single fixture can be part of multiple categories). For example:

[CompositeFixture(typeof(EnumerableTest))]
[ProviderFactory(typeof(ArrayListFactory),typeof(IEnumerable))]
[ProviderFactory(typeof(HashtableFactory),typeof(IEnumerable))]
[Pelikhan] -> author
[FixtureCategory("Important.Tests.Should.Be.Here")] -> categories
[FixtureCategory("A.Test.Can.Be.In.Multiple.Categories")]
[FixtureCategory("A.Test.Can.Be.In.Multiple.Categories2")]
public class CompositeTest
{
}

Screenshot

Here's a snapshot of the latest MbUnit snapshot: as you can see the tests are sorted by namespace, authors and categories.

This article will try to give an rough overview of the MbUnit vision of tests, and consequently it's architecture. It's contains some material of a previous CodeProject article.

Why MbUnit ?

Unit testing is a great tool for ensuring an application quality and frameworks like NUnit or csUnit have made it very simple to implement. However, as the number of tests begins to grow, the need for more functionalities begin to show up. The above frameworks are based on the Simple Test Pattern which is basically the sequence of SetUp, Test, TearDown actions. Although highly generic, this solution lets a lot of work to be done by the test writer. Sadely, there is no easy way to derive and include a new "fixture" type in those frameworks.

MbUnit is simply born from the fact that I wanted a new fixture and integrating it into existing frameworks was nearly impossible (I was also resting from a knee surgery at hospital with nothing else to do than coding).

Illustrating example

In order to make things clear, I will refer to an example while explaining how MbUnit works. Let me consider the Simple Test Pattern which is implemented by most test unit framework available. This is the classic way of writing unit test as described in the figure below. A TestFixture attribute tags the test class, one SetUp method, tests are done in the Test tagged method and clean up is performed in TearDown tagged method. This is illustrated in the left of the figure.

Attribute -> Run -> Invoker

The kernel of MbUnit is  composed of different components that work in a serial way. The first component is the fixture attribute. 

The fixture attribute is used to tag the classes that contain unit tests (TestFixtureAttribute is a fixture attribute). The new thing in MbUnit is that each fixture attribute contains the execution logic of the fixture which is returned at run-time under the form of a Run (IRun interface). In the case of the example, the TestFixtureAttribute is defined as a sequence of SetUp, Test and TearDown:

public class TestFixtureAttribute : TestFixturePatternAttribute 
{
     public override IRun GetRun()
     {
          SequenceRun runs = new SequenceRun();
            
          // setup
          OptionalMethodRun setup = new
                              OptionalMethodRun(typeof(SetUpAttribute),false);
          runs.Runs.Add( setup );
            
          //tests
          MethodRun test =new MethodRun(typeof(TestPatternAttribute),true,true);
          runs.Runs.Add(test);
            
          // tear down
          OptionalMethodRun tearDown = new
                           OptionalMethodRun(typeof(TearDownAttribute),false);
          runs.Runs.Add(tearDown);
            
          return runs;                        
     }
}

where

• TestFixturePatternAttribute is the abstract base class for all new fixture attribute in MbUnit,
• the GetRun method is called by the MbUnit core to know what is the execution path of the fixture. The fixture can use built-in basic attributes to build it's execution path.
• An IRun instance can represent the call to a method, or to a sequence of methods, etc...
• SequenceRun is a sequence of IRun's,
• MethodRun is a IRun instance that wraps a call to a method tagged by a predefined attribute.
• OptionalMethodRun is inherited from MethodRun and describes optional methods.

The IRun object will create an execution tree  by exploring the tagged type. Each node of the tree contains a RunInvoker (IRunInvoker interface). The RunInvoker is in charge for calling the method, garding the execptions, loading data, etc... On our sample fixture, there are two tests that the Run will extract:

When the tree is built, we just extract all the possible path from the root node to the leaves to extract the different possible tests. Each of these path is called a Pipe (RunPipe class).

In the GUI, the RunPipe instances are attached to the TreeNode nodes so you can easily select and execute separately the tests. This ensures that the test execution are isolated.

This architecture brings a lot of flexibility (and complexity) on the kind of fixtures that can be defined. Any user can define it's own fixture and use MbUnit to execute it.