# Wednesday, December 30, 2009

It is holiday time so I could spend some time to play with CCI. The result is a bunch of interesting assembly mutators. Let’s start with the first one: automatic rich assertion messages.

The problem with Assertions

When an assertion fails, the error message is usually insufficient to understand failure. The problem is that we are usually lacking the values in the expression to immediately diagnose the problem. Consider this example,

Assert.True(x == 123);

When this assertion fails, one would like to know what was the value of ‘x’. Of course, that value of x is not serialized in the test output since the assertion method only sees the ‘false’ value. What we would really like to get is something like ‘x (64) != 123’. A number of techniques have been developed to work around this issue.

Solution #1: Specialized Assertions

A common approach is to provide specialized assertion methods with enhanced logging. For example, a special method to test equality:

Assert.Equal(x , 123);

When this assertion fails, the Equal method has the value of both sides of the equality and can render it in the message: ‘expected 123, actual 64’. Unfortunately, expressions are more readable when you write them, and specialized assertions cannot cover all scenarios. This takes us to the next solution.

Solution #2: Expression Trees at Run time

Jafar Husain wrote a very interresting post on how to use Expression Trees to generate rich assertion messages (this is also supported in MbUnit).

Assert.True(() => x == 123);

Instead of taking a bool, the assert method takes an expression tree (Expression<Func<bool>>). Thus, when the expression evaluates to false, the expression tree can be traversed to extract interesting values (such as x) and generate a rich log of the failure.

Unfortunately, there is a major drawback to this technique: what used to be 3 MSIL instructions (ldloc.1, ldc.i4 123, ceq) to evaluate the condition becomes hundreds of methods calls, millions of instructions executed through the System.Linq namespaces. This performance is a overhead on the Pex whitebox analysis.

Jafar considers unit test frameworks as an extension of the compiler and uses expression trees to access what the compiler knows: expression and statements. Following his idea takes us to the CCI based solution.

Solution #3: Assembly Rewritter at Compile time

In the previous approach, expression trees were used to build a little compiler at runtime. A better solution would be to rewrite the expressions at compile time. In other words, we want to build an assembly mutator that takes the original method call and appends code that generates the logging:

Assert.True(x == 123, String.Format(“x == 123 where x = ‘{0}’”, x));

The assembly mutator extracts the expression sources from the pdb (‘x == 123’), collect the local/variable/field references by traversing the expression tree, generate a String.Format friendly message and replace the assertion method call to the assertion method that takes the additional string.

Hello Common Compiler Infrastructure (CCI) and CciSharp

Of course, to rewrite an assembly at compile time, we need a framework that can read and write MSIL, decompile expressions etc… This is where CCI (from http://ccimetadata.codeplex.com ) comes. It allows to manipulate .NET assemblies using an object model and save them back to disk. The assertion message mutator is just one of other mutations that have been or will be implemented as part of CciSharp, a post compiler for .NET that is built on top of CCI.

(There’s already a bunch of useful operators in CciSharp that i’ve been coding over the holidays: assigning the value of auto-properties, making auto-properties readonly or lazy (or weakly lazy), or even implementing the DependencyProperty stuff automatically. We’ll talk about this later.)

Where can I find it?

CciSharp binaries and sources are avaiable on http://ccisamples.codeplex.com/wikipage?title=CciSharp. Grab it!

posted on Wednesday, December 30, 2009 8:45:27 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Friday, December 18, 2009

We just released the version 0.20 of Pex. This version brings a Beavers, an extension of Stubs to write models, a global event view in Visual Studio and many bug fixes. Beavers are a small programatic models that build upon Moles…

  • Beavers: With stubs and moles, we provided a framework to write record/replay tests (i.e. mock-based tests) against any .NET type, interface or not. Beavers take you to the next level and allow you to write simple implementations with behavior (hence the ‘B’eaver) that can be used to replace the external system during testing. The benefit of using Beavers is that the resulting test cases are much more robust to code changes since the tests specify the state of the system, rather than a sequence of method calls and outcomes. We will talk a lot about Beavers in the future. Let’s take a look at an example. The following snippet uses Moles to redirect the constructor of a Bank class, then sets up the GetAccountById(int) method to return an instance of IAccount, etc… This is a typical test that is based record and replay. The problem with this technique is that it is really fragile with respect to code changes: if the code calls different methods, the test will break.
    The same behavior can be specified using Beavers. Beaver are simple implementations that simulate that capture enough behavior of the original code for testing. Using Beavers, one can specify the state of the system. The functional behavior is re-factored away in a Beaver class and reused in all tests!
  • Global Event View: when executing multiple explorations, it was cumbersome to go through each exploration to investigate the events (instrumentation, object creation, etc…). We have added a view in the Pex Explorer tree view that centralizes the events for the entire run.
  • Beavers for SharePoint: The samples that come with Pex contains a number of Beavers for the SharePoint object model (SPList, SPListItem, SPField, etc…) that should help you get started with testing SharePoint with Beavers. As an example, you can compare a mole-based test (left) and a beaver-based test (right) for a simple SharePoint service:
    image image
  • Bug fixes and Improvements:
    • The HostType crashes non-deterministically.
    • Visual Studio 2010 crashes when loading with a ‘DocumentSite already has a site’ error message.
    • The Stubs code generator uses a lot of memory and may crash Visual Studio. The Stubs generator now runs out of process and does not take down Visual Studio when it runs out of memory.
    • The Stubs code generator detects version changes of the Stubs framework to regenerate the stubs assemblies.
    • The Pex execution produces less warnings. We’ve reviewed our warnings and moved some, who did not matter, as messages.
    • When a Contract.Requires is failed multiple time in a stacktrace, do not allow it.
    • The object factory guesser has partial support for internal types and internal visible to (signed assemblies not supported).
    • The ‘add’ and ‘remove’ of events can be moled to deal with events.
    • Digger does not work in Visual Studio 2010 when no test project exists.
  • Breaking Changes: The Beavers introduced a number of refactoring that might impact users of Stubs and Moles. The .stubx file should regenerate themselves automatically, otherwise right click on the file and select ‘Run Custom Tool’.
    • Most types of the Stubs framework have been moved to sub namespaces.
    • IStubBehavior was renamed to IBehavior
    • StubFallbackBehavior was renamed to BehaviorFallbackBehavior
    • PexChooseAsStubFallbackBehaviorAttribute was renamed to PexChooseAsBehavedFallbackBehaviorAttribute
    • PexChooseStubBehavior was renamed to PexChooseBehavedBehavior
    • PexExpectedGoalsAttribute and PexGoal.Reached(…) were renamed and refactored into PexAssertReachEventuallyAttribute and PexAssert.ReachEventually(…);
posted on Friday, December 18, 2009 6:14:57 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Thursday, December 17, 2009

Je viens du publier une video en francais sur les Code Contracts et Pex avec Manuel.


posted on Thursday, December 17, 2009 8:35:26 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Wednesday, December 16, 2009

I’ll be presenting the PDC talk on Code Contracts and Pex at on January 20th, 2010 in Celestijnenlaan 200A, Heverlee (Leuven). Things change fast, so there will probably a couple changes to the presentation as well. I’ll probably talk a little but more about the tools to mine Code Contracts, etc…


posted on Wednesday, December 16, 2009 3:22:06 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Tuesday, December 15, 2009

I give a visit to Jeffrey Van Gogh who works for the Reactive Extensions framework, where they have been using Pex and writing Parameterized Unit Tests to test it. See the full video on Channel 9 at


posted on Tuesday, December 15, 2009 9:36:19 AM (Pacific Standard Time, UTC-08:00)  #    Comments [2]
# Sunday, December 13, 2009

J’aurai le plaisir de presenter ‘Stubs, Moles and Pex’ au premier rendez-vous de DotNetHub 20 janvier 2010 a LLN. DotNetHub est une communaute .NET francophone recemment cree. Au programme, beaucoup de demo a propos de Pex, Stubs et Moles, et aussi d’un nouvel animal – les Beavers.

Ce sera ma premiere presentation 100% en francais, ce qui en soit est un challenge assez interressant!

ps: Pas d’accent sur les claviers qwerty, ce qui rend l’ecriture du francais beaucoup plus facile :)

posted on Sunday, December 13, 2009 8:39:18 PM (Pacific Standard Time, UTC-08:00)  #    Comments [4]
# Thursday, December 03, 2009

I’ll be presenting the latest development on using Moles and Pex to unit test SharePoint 2010 (or 2007) Services at the SharePoint Connnections in Amsterdam, 18th-19th of January.

MSC26: Pex - Unit Testing of SharePoint Services that Rocks!
SharePoint Services are challenging for unit testing because it is not possible execute the SharePoint Service without being connected to a live SharePoint site. For that reason, most of the unit tests written for SharePoint are actually integration tests as they need a live system to run. In this session, we show how to use Pex, an automated test generation tool for .NET, to test SharePoint Services in isolation. From a parameterized unit test, Pex generates a suite of closed unit tests with high code coverage. Pex also contains a stubbing framework, Moles, that allows to detour any .NET method to user-defined delegates, e.g. replace any call to the SharePoint Object Model by a user-defined delegate.

posted on Thursday, December 03, 2009 10:45:36 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]