# Saturday, October 25, 2008


I’ll be at the MSR@PDC booth the entire week, don’t forget to say hello and get your demo.


posted on Saturday, October 25, 2008 7:29:17 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, October 21, 2008

Wonder what we’ve been up to for the last months… We’ve been building a very cool development experience on top of Pex that we call Code Digging. Check out Nikolai’s blog post on what it means to you!       .


posted on Tuesday, October 21, 2008 9:49:02 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]

Have you ever written code that directly used the .NET File APIs? We probably all did although we knew it would make the code less testable and dependent on the file system state. As bad as it sounds, it really requires a lot of discipline and work to avoid this: one would need to create an abstraction layer over the file system, which is not a short task (think long/tedious).

// in how many ways can this break?
public static void CleanDirectory(string path)
    if (Directory.Exists(path))
        Directory.Delete(path, true);


Fortunately, there always someone else who got motivated at some point. Ade Miller digged an abstraction of the File System, the IFileSystem interface, that Brad Wilson had written for the CodePlex client project. Very nice since it provides a solid foundation for cleanly abstracting from the File System, and thus increase the testability of our code.

// a little better, testable code at least
public static void
CleanDirectory(IFileSystem fs, string path) { if (fs.DirectoryExists(path)) fs.DeleteDirectory(path, false); fs.CreateDirectory(path); }


So with this interface we can write code that we’ll be able to test in isolation from the physical file system. That’s great but there is still a lot of work on the should of the developer: the developer will have write intricate scenarios involving mocks to simulate the different possible configurations of the file system. No matter which mock framework (Moq, Rhino, Isolator, …), he’ll be using, (1) it’s going to be painful, (2) he’ll miss cases. It’s probably easy to write a single “happy path”, but especially with the file system there are quite some realistic “unhappy paths”.

This test case uses Moq to create the scenario where there is a directory already. Although Moq has a very slick API to set expectations, it is still a lot of work to write this basic scenario. (And what exactly is the meaning of “Expect”, the delegate or expression inside, “Returns” and “Callback”?)

public void DeletesAndCreateNewDirectory()
    var fs = new Mock<IFileSystem>();
    string path = "foo";

    fs.Expect(f => f.DirectoryExists(path)).Returns(true);
    fs.Expect(f => f.DeleteDirectory(path, false)).Callback( () => Console.WriteLine("deleted"));
    fs.Expect(f => f.CreateDirectory(path)).Callback(() => Console.WriteLine("created"));

    DirectoryExtensions.CleanDirectory(fs.Object, path);


We had our intern, Soonho Kong, work on a Parameterized Model of the File System, built on top of the IFileSystem interface (yes that same interface Brad Wilson published on CodePlex). We say that the model is parameterized because it uses the Pex choices API to create arbitrary initial File System states; Pex “chooses” each such state (actually, Pex carefully computes the state using a constraint solver) to trigger different code paths in the code. You can think of each choice as a new parameter to the test. Or to put this with an example: if your code checks that the file “foo.txt” exists, then the parameterized model would choose a file system state that would contain a “foo.txt” file (or not, in another state, to cover both branches of the program).

So what does it mean for you? Well, the way you write tests that involve the file system changes radically. You simply need to pass the file system model to your implementation. The model is an under-approximation of the real file system (which means that we didn’t model every single nastiness that can occur when the moon is full), but it definitely captures more practically relevant corner cases than we (humans) usually think about. Let’s see this in the following test:

public void CleanDirectory()
    var fs = new PFileSystem();
    string path = @"\foo";
        DirectoryExtensions.CleanDirectory(fs, path);

        // assert: the directory exists and is empty
        Assert.AreEqual(0, fs.GetFiles(path).Length);

When we run Pex, we get 7 generated tests. In fact, Pex finds an interesting bug that occurs when a file with the name of the directory to clean already exists. In the Pex Exploration Results window, you can see a ‘dir’-like output of the file system model associated with a particular test case (the fs.Dir() method call outputs that text to the console which Pex captures).


This bug is the kind of corner-case that makes testing the file system so fun/hard. Thanks to the parameterized model (and Soonho), we got it for free. Note also that the assertion in our test is pretty powerful since it must be true for any configuration of the file system (it almost smells like a functional specification to me):

// assert: the directory exists and is empty
Assert.AreEqual(0, fs.GetFiles(path).Length);

Happy modeling!

The full source of PFileSystem will be available in the next version of Pex (0.8)..

posted on Tuesday, October 21, 2008 12:15:09 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [6]
# Friday, October 17, 2008

In this post, you’ll find the XSL stylesheets to render MSTest 2008 reports to CruiseControl.NET. CC.NET comes with stylesheets for MSTest 2005 but it seems that there were some changes in the xml output or at least they did not work for me. Please refer to the CC.NET documentation on how to integrate them into your build.

The templates are minimalistic since a deeper investigation can be done by importing the .trx file into Visual Studio.

The summary view:


And the details view:


Downloads – "AS IS" with no warranties, and confers no rights – :

posted on Friday, October 17, 2008 10:28:51 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Monday, October 13, 2008

In any unit testing activity, there always comes a point when one needs to isolate the code under test from the environment. This usually involves adding interfaces at the environment boundaries, which can be used to test the code in isolation. Because implementing those interfaces ‘by hand’ for the test can be tedious, an ecosystem of mock frameworks has flourished (NMock, Rhino Mocks, Moq, there’s more…). In fact, it’s a pretty competitive area.

The dark side of Mocks frameworks

Under their friendly API, mock frameworks usually hide a complex machinery: dynamic code (through Reflection.Emit or a rewriting profiler), expression trees or reflection.

From a Pex point of view, this ‘machinery’ puts a large burden on the dynamic whitebox code analysis; Pex does a precise data flow and control analysis for any code that is executed, this means that it also has to analyze the mock infrastructure.  

Therefore, we decided to build a framework to define stubs that would try to avoid the overhead of dynamic code generation (just rely on source code) and still provide the same level of flexibility of existing mock frameworks: defining the behavior of the stub inside the test, strongly typed, intellisense support, first-class debugging experience, etc…

Low tech Stubs using delegates

With C# 3.0 lambdas, it is quite easy to define delegates. Therefore, delegates seemed an elegant and simple way for building stub methods (we’re not the first ones to think about this). For example, a stub of the IServiceProvider would simply be

class SServiceProvider : IServiceProvider {
    // this field can be set to ‘override’ the behavior of GetService
    public Func GetServiceHandler;

    object IServiceProvider.GetService(Type type) {
        var d = this.GetServiceHandler;
        if (d != null) return d(type); // i’m stubbed :)
        else throw new NotImplementedException();

Enter the Stubs framework

We took this simple idea and turned it into the Stubs framework. The Stubs framework comes as a enhanced single file generator for Visual Studio which generates stubs for selected assemblies. Special attention was paid to make sure the code generation did not get into the feet for the unit test writer (we track the build events and regenerate when needed).

More about stubs can be found at http://research.microsoft.com/pex/stubs.pdf. This is a preview of what’s coming up on the next release of Pex.

Happy stubbing.

posted on Monday, October 13, 2008 8:07:23 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Thursday, October 02, 2008

We are very excited to announce that Pex has a session at PDC 2008. We will be talking about code contracts and Pex, and how they play nicely together. Book it now in your conference agenda!!! (look ‘Research’ or ‘Pex’ in the session list).


See you there and don’t forget to swing by our booth.

posted on Thursday, October 02, 2008 10:42:09 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]