It is holiday time so I could spend some time to play with CCI. The result is a bunch of interesting assembly mutators. Let’s start with the first one: automatic rich assertion messages.
The problem with Assertions
When an assertion fails, the error message is usually insufficient to understand failure. The problem is that we are usually lacking the values in the expression to immediately diagnose the problem. Consider this example,
Assert.True(x == 123);
When this assertion fails, one would like to know what was the value of ‘x’. Of course, that value of x is not serialized in the test output since the assertion method only sees the ‘false’ value. What we would really like to get is something like ‘x (64) != 123’. A number of techniques have been developed to work around this issue.
Solution #1: Specialized Assertions
A common approach is to provide specialized assertion methods with enhanced logging. For example, a special method to test equality:
Assert.Equal(x , 123);
When this assertion fails, the Equal method has the value of both sides of the equality and can render it in the message: ‘expected 123, actual 64’. Unfortunately, expressions are more readable when you write them, and specialized assertions cannot cover all scenarios. This takes us to the next solution.
Solution #2: Expression Trees at Run time
Jafar Husain wrote a very interresting post on how to use Expression Trees to generate rich assertion messages (this is also supported in MbUnit).
Assert.True(() => x == 123);
Instead of taking a bool, the assert method takes an expression tree (Expression<Func<bool>>). Thus, when the expression evaluates to false, the expression tree can be traversed to extract interesting values (such as x) and generate a rich log of the failure.
Unfortunately, there is a major drawback to this technique: what used to be 3 MSIL instructions (ldloc.1, ldc.i4 123, ceq) to evaluate the condition becomes hundreds of methods calls, millions of instructions executed through the System.Linq namespaces. This performance is a overhead on the Pex whitebox analysis.
Jafar considers unit test frameworks as an extension of the compiler and uses expression trees to access what the compiler knows: expression and statements. Following his idea takes us to the CCI based solution.
Solution #3: Assembly Rewritter at Compile time
In the previous approach, expression trees were used to build a little compiler at runtime. A better solution would be to rewrite the expressions at compile time. In other words, we want to build an assembly mutator that takes the original method call and appends code that generates the logging:
Assert.True(x == 123, String.Format(“x == 123 where x = ‘{0}’”, x));
The assembly mutator extracts the expression sources from the pdb (‘x == 123’), collect the local/variable/field references by traversing the expression tree, generate a String.Format friendly message and replace the assertion method call to the assertion method that takes the additional string.
Hello Common Compiler Infrastructure (CCI) and CciSharp
Of course, to rewrite an assembly at compile time, we need a framework that can read and write MSIL, decompile expressions etc… This is where CCI (from http://ccimetadata.codeplex.com ) comes. It allows to manipulate .NET assemblies using an object model and save them back to disk. The assertion message mutator is just one of other mutations that have been or will be implemented as part of CciSharp, a post compiler for .NET that is built on top of CCI.
(There’s already a bunch of useful operators in CciSharp that i’ve been coding over the holidays: assigning the value of auto-properties, making auto-properties readonly or lazy (or weakly lazy), or even implementing the DependencyProperty stuff automatically. We’ll talk about this later.)
Where can I find it?
CciSharp binaries and sources are avaiable on http://ccisamples.codeplex.com/wikipage?title=CciSharp. Grab it!