I've been recently interrested into Mutation Testing, a funny way of measure the quality of tests. This blog presents the first snapshot of a toy application that mutates any .Net program.

Mutation testing is the action of inserting "articifial" faults into the Instance Under Test and look if the tests catch this fault. The idea is that the tests are adequate if they detect all the faults. For example, a typical mutation is to negate the condition expression in a if statement:

//original
if (condition)
   DoSomething();
// mutated
if (!condition)
   DoSomething();

Jester implements Mutation testing for JUnit (there is a nice article here about Jester). In his Thesis (that Lutz Roeder kindly pointed out to me), A Multation Testing Tool for Java Programs, Matthias Bybro defines an entire framework for generating and executing mutants. In this blog, I will not focus on the theory of mutation testing but I'll show how you can get it implemented in .NET

The tools we need

As usual, before attacking the problem we can review what functionalities we need and what we have on our tool set. In this case, we need the ability to load an assembly, explore and alter the IL, and execute or write the mutated assembly. Got any idea....

RAIL! Runtime Assembly Instrumentation Library, that's exactly what we need. With RAIL, you can load an assembly, explore and alter the IL and execute or write the mutated assembly. You can even substitute types or entire functions. In fact, the powerpoint presentation of RAIL, the author shows how to play with IL.

Let's code

The AssemblyScrambler application is designed as follows: a ScramblerEngine instance contains a collection of IScrambler instances. An IScrambler instance contains a method to scramble IL code (I started this application before knowing about mutation testing. So Scrambler should be named mutators, etc...):

public interface IScrambler
{
    void Scramble(ScrambleTrace trace,RMethodDef method);
}

where trace is used to log mutations, and method is an instance of Rail.Reflect.RMethodDef which represents a method. The scramblers are used as follows in ScramblerEngine:

public void Scramble(string fileName)
{
    this.assembly = RAssemblyDef.LoadAssembly(fileName);
    foreach(RTypeDef t in this.assembly.RModuleDef.GetTypes())
    {
        foreach(RMethodDef method in t.GetMethods())
        {
            foreach(IScrambler scrambler in this.Scramblers)
            {
                scrambler.Scramble(trace,method);
            }
        }
    }
}

We are now ready to start implementing scramblers. There is currently only one implemented that swithes brtrue -> brfalse and brfalse -> brtrue. RMethodDef contains a MethodBody that contains a Code instance. Code is a mutable collection of instructions:

for(int i = 0;i<method.MethodBody.Code.InstructionCount;++i)
{
    Instruction il = method.MethodBody.Code[i];
    // selecting instruction
    if (il.OpCode.OperandType != OperandType.InlineBrTarget 
    && il.OpCode.OperandType != OperandType.ShortInlineBrTarget)
        continue;
    // il is ILBranch
    ILBranch branch = (ILBranch)il;
    // check if is brfalse
    if (il.OpCode.Name == OpCodes.Brfalse.Name)
    {
        // subtitute with brtrue
        method.MethodBody.Code[i]=new ILBranch(OpCodes.Brtrue,branch.Target);
    }
    else ...

Switching brtrue and brfalse is as simple as that. Note that here, 99% percent of the work is done by the excellent RAIL library.

Small example

Let's apply the scrambler to a small method:

public void IsTrue(bool isTrue)
{
    Console.Write("Expected: {0}, ",isTrue);
    if (isTrue)
        Console.WriteLine("Actual: true");
    else
        Console.WriteLine("Actual: false");
}

The IL code for this method is the following (using Reflector):

.method public hidebysig instance void IsTrue(bool isTrue) cil managed
{
// Code Size: 42 byte(s)
.maxstack 2
L_0000: ldstr "Expected: {0}, "
L_0005: ldarg.1 
L_0006: box bool
L_000b: call void [mscorlib]System.Console::Write(string, object)
L_0010: ldarg.1 
L_0011: brfalse.s L_001f
L_0013: ldstr "Actual: true"
L_0018: call void [mscorlib]System.Console::WriteLine(string)
L_001d: br.s L_0029
L_001f: ldstr "Actual: false"
L_0024: call void [mscorlib]System.Console::WriteLine(string)
L_0029: ret 
}

You can see that instruction at index 0011 is what we target. We have a small console application that calls this method. The code and results are:

Sandbox sandbox = new Sandbox();
sandbox.IsTrue(true);
sandbox.IsTrue(false);
-- output
Expected: True, Actual: true
Expected: False, Actual: false

After mutation

The above method is passed into the AssemblyScrambler machine, the IL code of the mutated application now looks like this:

.method public hidebysig instance void IsTrue(bool isTrue) cil managed
{
// Code Size: 42 byte(s)
.maxstack 3
L_0000: ldstr "Expected: {0}, "
L_0005: ldarg.1 
L_0006: box bool
L_000b: call void [mscorlib]System.Console::Write(string, object)
L_0010: ldarg.1 
L_0011: brtrue.s L_001f
L_0013: ldstr "Actual: true"
L_0018: call void [mscorlib]System.Console::WriteLine(string)
L_001d: br.s L_0029
L_001f: ldstr "Actual: false"
L_0024: call void [mscorlib]System.Console::WriteLine(string)
L_0029: ret 
}

Take a look now at L_0011, it is now brtrue.s.... the method is mutated. In fact, the output of the snippet gives:

Expected: True, Actual: false
Expected: False, Actual: true