Peli's Farm

My first Channel9 video (behind the camera)

Jonathan de Halleux — Tue, 02 Dec 2008 20:53:26 GMT

Very exciting, I just published my first video on channel9. In the future, I’ll posting more interviews about researchers and projects that happens in our group in Redmond. Expect to get more videos on Pex, Code Contracts, CHESS, etc… over there.

http://channel9.msdn.com/posts/Peli/The-RiSE-of-Research-in-Software-Engineering/

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex: Code Digger in the CodeProject.Com

Jonathan de Halleux — Tue, 25 Nov 2008 16:13:29 GMT

We just published a tutorial on the code digger on CodeProject, enjoy…

http://www.codeproject.com/KB/testing/pex.aspx

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex at the Seattle Code Camp – this Saturday at 2.45pm

Jonathan de Halleux — Fri, 14 Nov 2008 21:35:49 GMT

We will give a session on Pex at the Seattle Code Camp – Saturday 11/15 at 2.45pm. We will show the new Code Digger and how Pex can help you…

This posting is provided "AS IS" with no warranties, and confers no rights.

Code Digger Session on Channel 9

Jonathan de Halleux — Thu, 06 Nov 2008 07:10:22 GMT

Check out the session on Code Contracts and Pex on Channel 9. You will learn about the new cool API to express pre-conditions, post-conditions and invariants in your favorite language – i.e. design by contracts (DbC) for .NET and the new Code Digger experience in Pex, and most importantly how DbC and Pex play well together.

http://channel9.msdn.com/pdc2008/tl51/

This posting is provided "AS IS" with no warranties, and confers no rights.

PDC 2008 – See the digger at the Pex booth

Jonathan de Halleux — Sun, 26 Oct 2008 02:29:17 GMT

I’ll be at the MSR@PDC booth the entire week, don’t forget to say hello and get your demo.

This posting is provided "AS IS" with no warranties, and confers no rights.

Enter the Digger

Jonathan de Halleux — Wed, 22 Oct 2008 04:49:02 GMT

Wonder what we’ve been up to for the last months… We’ve been building a very cool development experience on top of Pex that we call Code Digging. Check out Nikolai’s blog post on what it means to you! .

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex It: The File System, Abstraction, Mocking , Modeling

Jonathan de Halleux — Tue, 21 Oct 2008 19:15:09 GMT

Have you ever written code that directly used the .NET File APIs? We probably all did although we knew it would make the code less testable and dependent on the file system state. As bad as it sounds, it really requires a lot of discipline and work to avoid this: one would need to create an abstraction layer over the file system, which is not a short task (think long/tedious).

// in how many ways can this
break? public static void CleanDirectory(string path)
{ if (Directory.Exists(path)) Directory.Delete(path, true); Directory.CreateDirectory(path);
}

Abstraction

Fortunately, there always someone else who got motivated at some point. Ade Miller digged an abstraction of the File System, the IFileSystem interface, that Brad Wilson had written for the CodePlex client project. Very nice since it provides a solid foundation for cleanly abstracting from the File System, and thus increase the testability of our code.

// a little better, testable
code at least

public static void CleanDirectory(IFileSystem fs, string path)
{ if (fs.DirectoryExists(path)) fs.DeleteDirectory(path, false);
fs.CreateDirectory(path); }

Mocking

So with this interface we can write code that we’ll be able to test in isolation from the physical file system. That’s great but there is still a lot of work on the should of the developer: the developer will have write intricate scenarios involving mocks to simulate the different possible configurations of the file system. No matter which mock framework (Moq, Rhino, Isolator, …), he’ll be using, (1) it’s going to be painful, (2) he’ll miss cases. It’s probably easy to write a single “happy path”, but especially with the file system there are quite some realistic “unhappy paths”.

This test case uses Moq to create the scenario where there is a directory already. Although Moq has a very slick API to set expectations, it is still a lot of work to write this basic scenario. (And what exactly is the meaning of “Expect”, the delegate or expression inside, “Returns” and “Callback”?)

[TestMethod] public
void DeletesAndCreateNewDirectory() { var fs
= new Mock<IFileSystem>>(); string path
= "foo"; fs.Expect(f => f.DirectoryExists(path)).Returns(true);
fs.Expect(f => f.DeleteDirectory(path, false)).Callback(
() => Console.WriteLine("deleted"));
fs.Expect(f => f.CreateDirectory(path)).Callback(() => Console.WriteLine("created")); DirectoryExtensions.CleanDirectory(fs.Object,
path); }

Modeling

We had our intern, Soonho Kong, work on a Parameterized Model of the File System, built on top of the IFileSystem interface (yes that same interface Brad Wilson published on CodePlex). We say that the model is parameterized because it uses the Pex choices API to create arbitrary initial File System states; Pex “chooses” each such state (actually, Pex carefully computes the state using a constraint solver) to trigger different code paths in the code. You can think of each choice as a new parameter to the test. Or to put this with an example: if your code checks that the file “foo.txt” exists, then the parameterized model would choose a file system state that would contain a “foo.txt” file (or not, in another state, to cover both branches of the program).

So what does it mean for you? Well, the way you write tests that involve the file system changes radically. You simply need to pass the file system model to your implementation. The model is an under-approximation of the real file system (which means that we didn’t model every single nastiness that can occur when the moon is full), but it definitely captures more practically relevant corner cases than we (humans) usually think about. Let’s see this in the following test:

[PexMethod] public
void CleanDirectory() { var fs = new PFileSystem(); string path
= @"\foo"; try { DirectoryExtensions.CleanDirectory(fs,
path); // assert: the directory exists and is empty Assert.IsTrue(fs.DirectoryExists(path)); Assert.AreEqual(0,
fs.GetFiles(path).Length); } finally { fs.Dir();
} }

When we run Pex, we get 7 generated tests. In fact, Pex finds an interesting bug that occurs when a file with the name of the directory to clean already exists. In the Pex Exploration Results window, you can see a ‘dir’-like output of the file system model associated with a particular test case (the fs.Dir() method call outputs that text to the console which Pex captures).

This bug is the kind of corner-case that makes testing the file system so fun/hard. Thanks to the parameterized model (and Soonho), we got it for free. Note also that the assertion in our test is pretty powerful since it must be true for any configuration of the file system (it almost smells like a functional specification to me):

// assert: the directory exists and is empty
Assert.IsTrue(fs.DirectoryExists(path));
Assert.AreEqual(0, fs.GetFiles(path).Length);

Happy modeling!

The full source of PFileSystem will be available in the next version of Pex (0.8)..

This posting is provided "AS IS" with no warranties, and confers no rights.

MSTest 2008 Reports for CruiseControl.NET

Jonathan de Halleux — Fri, 17 Oct 2008 17:28:51 GMT

In this post, you’ll find the XSL stylesheets to render MSTest 2008 reports to CruiseControl.NET. CC.NET comes with stylesheets for MSTest 2005 but it seems that there were some changes in the xml output or at least they did not work for me. Please refer to the CC.NET documentation on how to integrate them into your build.

The templates are minimalistic since a deeper investigation can be done by importing the .trx file into Visual Studio.

The summary view:

And the details view:

Downloads – "AS IS" with no warranties, and confers no rights – :

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex gets a Stubs framework (I said stubs, not mocks)

Jonathan de Halleux — Mon, 13 Oct 2008 15:07:23 GMT

In any unit testing activity, there always comes a point when one needs to isolate the code under test from the environment. This usually involves adding interfaces at the environment boundaries, which can be used to test the code in isolation. Because implementing those interfaces ‘by hand’ for the test can be tedious, an ecosystem of mock frameworks has flourished (NMock, Rhino Mocks, Moq, there’s more…). In fact, it’s a pretty competitive area.

The dark side of Mocks frameworks

Under their friendly API, mock frameworks usually hide a complex machinery: dynamic code (through Reflection.Emit or a rewriting profiler), expression trees or reflection.

From a Pex point of view, this ‘machinery’ puts a large burden on the dynamic whitebox code analysis; Pex does a precise data flow and control analysis for any code that is executed, this means that it also has to analyze the mock infrastructure.

Therefore, we decided to build a framework to define stubs that would try to avoid the overhead of dynamic code generation (just rely on source code) and still provide the same level of flexibility of existing mock frameworks: defining the behavior of the stub inside the test, strongly typed, intellisense support, first-class debugging experience, etc…

Low tech Stubs using delegates

With C# 3.0 lambdas, it is quite easy to define delegates. Therefore, delegates seemed an elegant and simple way for building stub methods (we’re not the first ones to think about this). For example, a stub of the IServiceProvider would simply be

class SServiceProvider : IServiceProvider {
    // this field can be set to ‘override’ the behavior of GetService
    public Func GetServiceHandler;
    object IServiceProvider.GetService(Type type) {
        var d = this.GetServiceHandler;
        if (d != null) return d(type); // i’m stubbed :)
        else throw new NotImplementedException();
    }
}

Enter the Stubs framework

We took this simple idea and turned it into the Stubs framework. The Stubs framework comes as a enhanced single file generator for Visual Studio which generates stubs for selected assemblies. Special attention was paid to make sure the code generation did not get into the feet for the unit test writer (we track the build events and regenerate when needed).

More about stubs can be found at http://research.microsoft.com/pex/articles/stubs.pdf. This is a preview of what’s coming up on the next release of Pex at the end of the month.

Happy stubbing.

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex at PDC2008 – We will be there!

Jonathan de Halleux — Fri, 03 Oct 2008 05:42:09 GMT

We are very excited to announce that Pex has a session at PDC 2008. We will be talking about code contracts and Pex, and how they play nicely together. Book it now in your conference agenda!!! (look ‘Research’ or ‘Pex’ in the session list).

See you there and don’t forget to swing by our booth.

This posting is provided "AS IS" with no warranties, and confers no rights.