We just released a new version of Pex and Moles that brings a number of bug fixes and various improvements to the behaviors for SharePoint and Asp.NET.

Improvements

• Interaction with the Source Control provider has been significantly improved for Moles
• Support for Moles of nested types.
• Improved logging in the Host Type
• Improved Behaved collections
• Added Behaved types for mscorlib and System.Web types.

Bug fixes

• Updated several outdated section in the documentation
• Updated missing classes in the SharePoint samples
• Fixed a limitation of the profiler that would silently fail to instrument certain methods

Breaking Changes

• We have formalized the naming convention of Moles and Stubs and found some holes along the way. Some mole method might have a new name under this version of the compiler.

We just released Pex 0.21.50115.2. This release brings bug fixes, a big renaming from “Stubs” to “Moles” and improved infrastructure to build behaved types (formerly known as beavers).

Bug Fixes

• The Moles VsHost fails to execute unit tests in different assemblies.
• Pex deletes the report folder it is currently writing to.
• Support for named indexers in Stubs
• Fixed bugs in how Pex reasons about System.Convert.ToBase64, DateTime
• Invalid support for protected members in the stubs generation

Breaking changes

• The Stubs framework was renamed to Moles framework. We have decided to make the Moles the center of the framework and as a consequence, renamed ‘Stubs’ to ‘Moles’. (This does not mean that we encourage writing untestable code, as Mole help to make it testable. You should still refactor your code to make it testable whenever possible, and only use Moles when that’s the only choice). The impact is that
  • Microsoft.Stubs.Framework was renamed to Microsoft.Moles.Framework
  • The moles and stubs get generated in subnamespaces ‘.Moles’ rather ‘.Stubs’.
  • See below for the list of steps to upgrade your applications.
• BaseMembers in Moles have been deprecated: this helper is not useful as it can be acheive in a better way through a constructor. We decided to remove it to reduce code size. The second reason is that BaseMembers would only work for types inside of the same assembly, which might seem inconsistent.
• PexGoal.Reached is replaced by PexAssert.ReachEventually(). The PexGoal class has been integrated into PexAssert through the ReachEventually method which should be used with the [PexAssertReachEventually] attribute.
• PexChoose simplified: we’ve simplified the PexChoose API; you can now get auxiliary test inputs with a single method call: PexChoose.Value<T>(“foo”).

Migrating from previous version of Pex

Since we’ve renamed Stubs to Moles, any existing .stubx files will not work anymore.

Take a deep breath, and apply the following steps to adapt your projects:

• change the project reference from Microsoft.Stubs.Framework.dll to Microsoft.Moles.Framework.dll
• rename all .stubx files to .moles, and
  • rename the top <Stubs xml element to <Moles.
  • Change the XSD namespace to http://schemas.microsoft.com/moles/2010/
  • Right click on the .moles file in the Solution Explorer and change the Custom Tool Name to ‘MolesGenerator’.
  • Delete all the nested files under the .moles files
• Remove references to any compiled .Stubs.dll files in your project
• In general, remove all .Stubs.dll, .Stubs.xml files from your projects.
• Rename .Stubs namespace suffixes to .Moles.
• replace all [HostType(“Pex”)] attribute with [HostType(“Moles”)]
• in PexAssemblyInfo.cs,
  • rename using Microsoft.Pex.Framework.Stubs to Microsoft.Pex.Framework.Moles
  • rename [assembly: PexChooseAsStubFallbackBehavior] to [assembly: PexChooseAsBehavedCurrentBehavior]
  • rename [assembly: PexChooseAsStubFallbackBehavior] to [assembly: PexChooseAsMoleCurrentBehavior]
• In general, the ‘Fallback’ prefix has been dropped in the following methods:
  • rename FallbackAsNotImplemented() to BehaveAsNotImplemented()
  • rename class MoleFallbackBehavior to MoleBehaviors
  • rename class StubFallbackBehavior to BehavedBehavors

We just released the version 0.20 of Pex. This version brings a Beavers, an extension of Stubs to write models, a global event view in Visual Studio and many bug fixes. Beavers are a small programatic models that build upon Moles…

• Beavers: With stubs and moles, we provided a framework to write record/replay tests (i.e. mock-based tests) against any .NET type, interface or not. Beavers take you to the next level and allow you to write simple implementations with behavior (hence the ‘B’eaver) that can be used to replace the external system during testing. The benefit of using Beavers is that the resulting test cases are much more robust to code changes since the tests specify the state of the system, rather than a sequence of method calls and outcomes. We will talk a lot about Beavers in the future. Let’s take a look at an example. The following snippet uses Moles to redirect the constructor of a Bank class, then sets up the GetAccountById(int) method to return an instance of IAccount, etc… This is a typical test that is based record and replay. The problem with this technique is that it is really fragile with respect to code changes: if the code calls different methods, the test will break.
  
  The same behavior can be specified using Beavers. Beaver are simple implementations that simulate that capture enough behavior of the original code for testing. Using Beavers, one can specify the state of the system. The functional behavior is re-factored away in a Beaver class and reused in all tests!
  
• Global Event View: when executing multiple explorations, it was cumbersome to go through each exploration to investigate the events (instrumentation, object creation, etc…). We have added a view in the Pex Explorer tree view that centralizes the events for the entire run.
  
• Beavers for SharePoint: The samples that come with Pex contains a number of Beavers for the SharePoint object model (SPList, SPListItem, SPField, etc…) that should help you get started with testing SharePoint with Beavers. As an example, you can compare a mole-based test (left) and a beaver-based test (right) for a simple SharePoint service:
  
• Bug fixes and Improvements:
  • The HostType crashes non-deterministically.
  • Visual Studio 2010 crashes when loading with a ‘DocumentSite already has a site’ error message.
  • The Stubs code generator uses a lot of memory and may crash Visual Studio. The Stubs generator now runs out of process and does not take down Visual Studio when it runs out of memory.
  • The Stubs code generator detects version changes of the Stubs framework to regenerate the stubs assemblies.
  • The Pex execution produces less warnings. We’ve reviewed our warnings and moved some, who did not matter, as messages.
  • When a Contract.Requires is failed multiple time in a stacktrace, do not allow it.
  • The object factory guesser has partial support for internal types and internal visible to (signed assemblies not supported).
  • The ‘add’ and ‘remove’ of events can be moled to deal with events.
  • Digger does not work in Visual Studio 2010 when no test project exists.
• Breaking Changes: The Beavers introduced a number of refactoring that might impact users of Stubs and Moles. The .stubx file should regenerate themselves automatically, otherwise right click on the file and select ‘Run Custom Tool’.
  • Most types of the Stubs framework have been moved to sub namespaces.
  • IStubBehavior was renamed to IBehavior
  • StubFallbackBehavior was renamed to BehaviorFallbackBehavior
  • PexChooseAsStubFallbackBehaviorAttribute was renamed to PexChooseAsBehavedFallbackBehaviorAttribute
  • PexChooseStubBehavior was renamed to PexChooseBehavedBehavior
  • PexExpectedGoalsAttribute and PexGoal.Reached(…) were renamed and refactored into PexAssertReachEventuallyAttribute and PexAssert.ReachEventually(…);

J’aurai le plaisir de presenter ‘Stubs, Moles and Pex’ au premier rendez-vous de DotNetHub 20 janvier 2010 a LLN. DotNetHub est une communaute .NET francophone recemment cree. Au programme, beaucoup de demo a propos de Pex, Stubs et Moles, et aussi d’un nouvel animal – les Beavers.

Ce sera ma premiere presentation 100% en francais, ce qui en soit est un challenge assez interressant!

ps: Pas d’accent sur les claviers qwerty, ce qui rend l’ecriture du francais beaucoup plus facile :)

We just released the Pex 0.19.41110.1 just in time for TechEd session. This version brings an updatedsupport for Visual Studio 2010 Beta2 and .NET 4.0 and many improvements to the Moles.

• Better Dev10 Beta 2 support! We’ve upgraded our support for Visual Studio 2010 Beta 2. This includes full support for .NET 4.0 Beta 2, the Visual Studio Unit Test HostType and the Visual Studio integration.
  Known Beta 2 issues: 1) When running Pex for the first time, the path to Visual Studio Unit Test is not found. You need to specify the Visual Studio ‘PublicAssemblies’ path (“c:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PublicAssemblies”), 2) Don’t run Pex on v2 projects, Pex will crash.
• Smooth generation of Stubs and Moles: when you create a stubs and moles assembly that is not a project reference, like one of the system assemblies, the stubs and moles code generator will automatically compile it into an assembly and add it to the project. This dramatically increases the performance of projects using Stubs and Moles.
  
• Stubs and Moles for VisualBasic.NET: the Stubs and Moles may be used in VisualBasic.NET projects. In that case, they are embedded as a compiled assembly.
• Moles support for NUnit, xUnit.Net and others: Unit tests involving Moles require the Pex profiler to work. We’ve added a new mode to our command line runner that allows to execute your favorite unit test framework runner under Pex. For example, the command “pex /runner=xunit.console.exe tests.dll” will launch the xunit runner for test.dll under Pex. x64 assemblies are still not supported.
  
• Nicer Moles Syntax: we’ve added some little tweaks to the Moles API, such as an implicit conversion operator to the moled type, that simplifies the syntax. Here is a small example that showcases the current syntax: step 1 moles the Bank constructor, step 2 moles the GetAccount method and returns a mole of Account, step 3 moles the getter and setter of the Balance property.
   
• Fluent Mole Binding: The Bind method returns the mole itself which allows it to be used in a fluent fashion. Interface binding is specially useful when you need to mole custom collections – just attach an array! For example, we attach an array of integers to a the MyList type that implements IEnumerable<int>.
   
• Breaking Changes in Stubs and Moles:
  • The stubs and moles code generator tool is now ‘StubsGenerator’. You need to path this value in the property pages of your .stubx pages.
  • No more AsStub and AsMole: the AsStub and AsMole helpers were confusing and pretty useless so we decided to reduce the amount of generated code by removing them.
  • The naming convention for Stubs delegate fields always mangles the parameter types of the stubbed method, i.e. Foo(int) yields to a field called FooInt32. We used to be smart about adding parameters but we decided to make the scheme stable and symmetric with respect to the Moles.
  • The ‘FallbackBehavior’ property has been renamed to ‘InstanceFallbackBehavior’ in both Stubs and Moles to make it more explicit.

As usual, we’ve also fixed a number of bugs and added features that were reported through our MSDN forums.

I’ll be presenting Stubs and Moles at the Visual Studio User Group on November 3rd. See you there…

Stubs is a lightweight framework for test stubs and detours in .NET that is entirely based on delegates, type safe, Refactorable, debuggable and source code generated.
Stubs also allows to replace  any .NET method with a user-defined delegate, including non-virtual/static methods in sealed types. Stubs is fully integrated into Pex , an automated white box test generation tool for .NET.
http://research.microsoft.com/en-us/projects/stubs/

We have just release a new version of Pex (v0.17.xxx) which brings new features for Moles and more bug fixes.

Per-Instance Moles

Instance methods may be moled differently per object instance. This is particularly useful when multiple instances of same type need to behave differently in the same test. For example, let’s consider a simple class Bar that has a ‘Run’ method to check that the values are not the same. The important point here is that we would not be able to deal with such cases if we could not mole the behavior on a per-instance basis.

In the test case of Run, we create to instance of MBar (the mole type of Bar) and assign two different delegates to the ‘Value’ getter. Moles are active as soon as they are attached (unlike the previous version) so we can just call the ‘Run’ method once the moles are set up. In the code below, we are using the C# Object Initializers feature that allows us to set properties of a newly created object within curly braces after the ‘new’ call.

As you may have noticed, we just wrote a parameterized unit test that takes arbitrary ‘left’ and ‘right’ values. We then simply execute Pex to find the relevant values to cover the ‘Run’ method:

Moles for Constructors

In same cases, objects created inside of other methods need to be moled to test the actual program logic. With moles you can replace any constructor with your own delegate, which you can use to attach a mole to the newly created instance. Let’s see this with another simple example where a  ‘Run’ method creates and uses an instance of a ‘Foo’ class.

Let’s say we would like to mole the call to the ‘Value’ property to return any other value. To do so, we would need to attach a mole to a future instance of Foo, and then mole the Value property getter. This is exactly what is done in the method below.

We then run Pex to find that 2 different values are needed to trigger all code paths.

Mole interface binding

All the member implementations of an interface may be moled at once using the new ‘Bind’ method. This smells like duck typing with type safety as the Bind methods are strongly typed. For example, we want to mole a collection type (Foes) to return a custom list of Foo elements (which need to be moled too). The goal is to test the sum method…

In the parameterized unit test case, we create an instance of Vector, which implements IEnumerable<int>. Then, we can simply bind the ‘values’ array to the mole to make Vector iterate over the array. The call to Bind will mole all methods of MVector that are implemented by the ‘values’ parameters, effectively redirecting all the enumeration requests to the ‘values’ array.

When Pex runs through the sample, it correctly understand the data flow of the parameters through the moles and finds inputs that break the ‘sum >= 0’ assertion:

Compilation of Stubs assemblies

When one needs stubs or moles for a system assembly, it does not really make sense to re-generate the stubs each time the solution is loaded. To that end, Stubs ships with a command line tool that lets you easily compile a stubs assembly into a .dll:

stubs.exe mscorlib

Once the stubs assembly is compiled, i.e. mscorlib.Stubs.dll is created, you simply have to add a reference to it in your test project to start using it. In future versions of Pex, we will provide a better experience to support this scenario.

Bug Fixes

• A number of fixes from the issues reported through the forums and bugs reported to pexbug@microsoft.com .

Breaking Changes

• The code generation of Moles was significantly changed. This might mean that you will have to recompile your solution, and adapt all existing uses of Moles.

We just released v0.16.40915.5 of Pex. The major highlights of this release are Moles, a lightweight detour framework, and better support for test frameworks.

• Moles, a lightweight detour framework: Moles is a new extension of the Stubs framework: it lets you replace any .NET method (including static methods) with your own delegate.(sample outdated)
  Want read more about this sample? here is the full walkthrough.
  The moles framework is not yet feature complete; moles does not support constructors and external methods. Some types of mscorlib cannot be moled as they interact too deeply with the CLR.
• Pex gets its own HostType: A HostType is a feature for the Visual Studio Unit Test framework that lets specific unit tests be run under specialized hosts such as Asp.Net or VisualStudio iself. In order to create reproducible test cases using Moles, we had to implement a HostType that lets tests be run under the Pex profiler. This is very exciting because it also opens the door for many uses of Pex such as fault injection, dynamic checking, etc… in future versions of Pex. When generating test cases with Pex, all the necessary annotations will be created automatically. To turn the Pex HostType on hand-written (non-parameterized) unit tests, simply add [HostType(“Pex”)] on your test case.
  
  This feature only works with Visual Studio Unit Test 2008.
• Select your test framework: the first time you invoke ‘Pex’ on your code, Pex pops up a dialog to select your test framework of choice. You can select which test framework should be used by default and, more importantly, where it can be found on disk.
  
  If you do not use any test framework, Pex ships with what we call the “Direct test framework”: in this “framework”, all methods are considered as unit tests without any annotation or dependencies.
  
  These settings are stored in the registry and Pex should not bug you again. If you want to clear these settings, go to ‘Tools –> Options –> Pex –> General’ and clear the TestFramework and TestFrameworkDirectory fields:
  
  
• Thumbs up and down: We’ve added thumbs up and down buttons in the Pex result view. We are always looking for feedback on Pex, so don’t hesitate to click them when Pex deserves a thumbs up or a thumbs down.
  
• Performance: Many other performance improvements under the hood which should avoid Pex hog too much memory in long running scenarios.
• Miscellanous improvements and bug fixes:
  • Support for String.GetHashCode(): we now faithfully encode the hashing function of strings, which means Pex can deal with dictionaries where the key is a string.
  • Fewer “Object Creation” messages that are not actually relevant.
  • In VS 2010, the package used to trigger an exception when the solution loaded. This issue is now fixed.
  • In Visual Studio when an assembly had to be resolved manually (i.e. that little dialog asking you where an assembly is), Pex remembers that choice and does not bug you anymore.
  • And many other bugs that were reported through the forums.

This is a cool new feature that builds on Stubs, Pex  and Code Contracts  (in fact, it is really just a consequence of the interaction of the 3 technologies): stubs that act as parameterized models while complying to the interface contracts.

• Code Contracts provide contracts for interfaces. All implementations of that particular interface will be instrumented with the runtime checks by the rewriter at compile time.
• Stubs is really just a code generator that provides a minimal implementation of interfaces and relies on the C# compiler to build the code. Therefore, if a stub implements an interface with contracts, it will automatically be instrumented by the runtime rewriter as well.
• Pex choices (i.e. dynamic parameter generation) may be used as the fallback behavior of stubs. In other words, whenever a stubbed method without user-defined behavior is called, Pex gets to pick the return value. Since the stub implementation itself may have been instrumented with contracts, we’ve added special handling code so that post-condition violation coming from a stub are considered as an assumption. This effectively forces Pex to generate values that comply with the interface contracts.

Let’s see how this works with an example. The IFoo interface contains a single ‘GetName’ method. This method has one precondition, i should be positive and one postcondition, the result is non null. Since interfaces cannot have a method body, we use a ‘buddy’ class to express those contracts and bind both classes using the [ContractClass] and [ContractClassFor] attributes:

We then turn on runtime rewriting for both the project under test and the test project (Project properties –> Code Contracts –> Runtime checks). As a result, the stub of IFoo automatically gets instrumented with the contracts. We can clearly see that from the generated code of GetName in Reflector below:

The last step is to write a test for IFoo. In this case, we write a parameterized unit test that takes an IFoo instance and an integer, then simply call GetName.

Since the contracts of IFoo.GetName specifies that the result should not be null, we should not see any assertion violation in this test. Moreover, we should see a test for the precondition considered as an expected exception:

Note that all those behaviors rely on extensions to Pex that the wizard automatically inserted in the PexAssemblyInfo.cs file:

where

• [PexAssumeContractEnsuresFailureAtStubSurface] filters post-condition violations in stubs as assumptions,
• [PexChooseAsStubFallbackBehavior] installs Pex choices as the fallback behavior of stubs,
• [PexStubsPackage] load the stubs package (this is a standard procedure for Pex packages),
• [PexUseStubsFromTestAssembly] specifies to Pex that it should considers stub types when trying to test interfaces or abstract classes.

We just made a quick release to fix another installer issue related to missing packages. Along the way, we’ve added an Exploration Tree View and Partial Stubs support.

Exploration Tree View

The exploration tree view displays the list of explorations to be executed, running and finished. It serves as a progress indicator but also as a smooth result explorer. When browsing through the tree, Pex will synchronize the exploration result view and the tree view.

The tree view populates each namespace with the fixture types and exploration methods, and provide a visual feedback on the progress of Pex.

When you browse through the exploration and generated test nodes, Pex automatically synchronizes the exploration result display. This makes it really easy to start from an high-level view of the failures and drill into a particular generated test, with stack trace and parameter table.

Partial Stubs

Stubs lets you call the base class implementation of methods as a fallback behavior. This functionality is commonly referred as Partial Mocks or Partial Stubs and is useful to test abstract classes in isolation. Stubs inheriting from classes have a “CallBase” property that can turn this mode on and off.

Let see this with the RhinoMocks example on partial mocks. Given an abstract ProcessorBase class,

we write a test for the Inc method. To do so, we provide a stub implementation of Add that simply increment a counter.

Miscellaneous

• PexAssume/PexAssert.EnumIsDefine checks that a particular value is defined in an enum.
• Missing OpenMP files in the installer break Pex.

Poll: should we skip 0.13 and go straight for 0.14? :)

Update: Andrew Kazyrevich does the final performance analysis on his blog.
Update II: Stubs v0.12.40430.3 has Partial Stubs support.

Stubs is a lightweight stub framework that is entirely based on delegates. We designed it so that it brings as little overhead as possible, and as a side effect, is very effective with Pex. In this post, we’ll see how Stubs ‘look’ with respect to other frameworks.

Mocking-framework-compare

Andrew Kazyrevich started a project, mocking-framework-compare, back in February that compared Moq, Rhino, NMock2 and Isolator. This is project compares various ‘mocking scenarios’ across all those frameworks. I added Stubs to the mix today (go check it out).

What’s a stub in Stubs anyway?

Before we dig in, let’s recap how stubs look like when you use Stubs. Stubs uses delegate to ‘hook’ behavior to interface members.For every stubbed interface and class, code is generated at a compile time: One stub class per interface and class. For each stubbed method, the stub class contains a field. The field has a delegate type matching the stubbed method. As a user, you can simply attach delegates (or lambdas) to the delegate fields to assign*any* behavior to each member. If no user delegate is set, Stubs calls into a fallback behavior, i.e. throw or return the default value.

Let’s take one of the examples of mocking-framework-compare. The little snippet below ‘stubs’ the TouchIron() method (which is implemented explicitly) by attaching a lambda to the TouchIron field.

SIHand is the stub type of the IHand interface, it was generated at compile time by the Stubs framework. Its implementation looks more or less like this:

The TouchIron methods really shows the main idea behind stubs: delegates are all we need to move behavior around.

Stubs: relying on the language rather than an API

One of the differences of Stubs with other mock frameworks is that Stubs does not have any API: delegates, lambdas, closures are all features of the programming language, while assertions are part of the test framework. This is quite obvious when we compare the BrainTest using Moq and using Stubs. In this test, we need to ensure that the brain coordinates the hand and the mouth so that when a hot iron is touched by the hand, the mouth yells.

• Moq: we set up the hand to throw a burn exception when touched with a hot iron and verify that the mouth yelled.

Moq, as Rhino or Isolate, make smart use of expression trees and/or Reflection.Emit to define the mock behavior. The expectation and behavior are set through nice and fluent APIs which really make the developer’s life’s easier.

• Stubs: same scenario, we attach a lambda that throws if iron is hot and use a local to verify that Yell is called (this local is pushed to the heap by the compiler).

Stubs does not have any API. It relies on lambdas (to define new behaviors), closures (to track side effects) which are given for free by the C# 3.0 language.

Performance numbers

Does performance matter when it comes to mocking? It’s not really the question we are trying to answer here. We’re just looking at the benchmark results from the mocking-framework-compare project. When looking at the results, you’ll notice that Stubs may be 100x to 1000x faster than other frameworks. This is no surprise since stubs boil down to a virtual method call, while other frameworks do much more work (in fact we expected worse numbers).

Data units of msec resolution = 0.394937 usec

Mocking methods.
Moq      : 100 repeats:  37.471 +- 12%   msec
Rhino    : 100 repeats:  38.030 +- 7%    msec
NMock2   : 100 repeats:  24.035 +- 4%    msec
Stubs    : 100 repeats:   0.115 +- 8%    msec

Mocking events.
Moq      : 100 repeats:  86.913 +- 7%    msec
Rhino    : 100 repeats:  61.142 +- 6%    msec
NMock2   : 100 repeats:  27.378 +- 6%    msec
Stubs    : 100 repeats:   0.071 +- 6%    msec

Mocking properties.
Moq      : 100 repeats:  82.434 +- 6%    msec
Rhino    : 100 repeats:  47.471 +- 5%    msec
NMock2   : 100 repeats:  11.334 +- 10%   msec
Stubs    : 100 repeats:   0.042 +- 15%   msec

Mocking method arguments.
Moq      : 100 repeats: 142.668 +- 4%    msec
Rhino    : 100 repeats:  45.118 +- 5%    msec
NMock2   : 100 repeats:  22.344 +- 7%    msec
Stubs    : 100 repeats:   0.078 +- 4%    msec

Partial mocks.
Moq      : 100 repeats: 117.581 +- 5%    msec
Rhino    : 100 repeats:  58.827 +- 6%    msec
Stubs : 100 repeats:   0.054 +- 6%    msec

Recursive mocks.
Moq      : 100 repeats:  92.482 +- 4%    msec
Rhino    : 100 repeats:  40.921 +- 3%    msec
Stubs    : 100 repeats:   0.493 +- 18%   msec
Press any key to continue . . .

Stubs limitations

There are many areas where the Stubs fall short or simply don’t support it. Currently, Stubs are only emitted for C# and partial mocks will be supported in the next version.

Getting started with Stubs

Stubs comes with the Pex installer. If you’re interested on using it, check out the getting started page on the Stubs project page where you can also download the full Stubs primer.

Cheers, Peli

Download the new version!

Today we’ve released a new release of Pex on DevLabs and on our academic downloads. This highlights of this release are: NUnit, MbUnit and xUnit.net support out of the box, writing parameterized unit tests in VisualBasic.NET and F#, better Code Contracts support. As always, if we encourage you to send us feedback, bugs, stories on our forums at http://social.msdn.microsoft.com/Forums/en-US/pex/threads/ .

NUnit, MbUnit and xUnit.net supported out of the box

Pex now supports MSTest, NUnit, MbUnit and xUnit.net out of the box. Pex will automatically detect which framework you are using by inspecting the assembly reference list, and automatically save the generated tests decorated with the correct attributes for that framework.

The default test framework can also be specified through the global options (Tools –> Options –> Pex –> enter the test framework name in TestFramework).

Writing Parameterized Unit Tests in VisualBasic.NET

While the Pex white box analysis engine works at the MSIL level, Pex only emits C# code for now. In previous releases, this limitation made it impossible to use Pex parameterized unit tests from non-C# code. In this release, we have worked around this problem by automatically saving the generated tests in a ‘satellite’ C# project.

Let’s see this with an example. The screenshot below shows a single VisualBasic.NET test project with a Pex parameterized unit test:

We can right-click in the HelloTest.Hello method and select “Run Pex Explorations”:

At this point, Pex will start exploring the test in the background as usual. This is where the new support comes in: When a generated test comes back to Visual Studio, Pex will save it in a separate C# project automatically (after asking you where to drop the new project):

The generated tests are now ready to be run just as any other unit tests!

Writing Parameterized Unit Tests from F#

Similarly to VisualBasic.NET, we’ve made improvements in our infrastructure to enable writing parameterized unit tests in F#. Let’s see this with a familiar example. We have a single F# library that has xUnit.net unit tests and reference Microsoft.Pex.Framework (project Library2 below). In that project, we add a parameterized unit test (hello_test):

We can right-click on the test method name and Pex will start the exploration of that test in the background. Because of the limitations of the F# project system, you absolutely need to right-click on the method name in F# if you want contextual test selection to work. Because the project is already referencing xunit.dll, Pex will also automatically detect that you are using xUnit.net and use that framework. When the first test case comes back to VisualStudio, Pex saves it in a separate C# project:

The tests are saved in the generated test project and ready to be run by your favorite test runner!

PexObserve: Observing values, Asserting values

We’ve completely re-factored the way values can be logged on the table or saved as assertions in the generated tests. The following example shows various ways to log and assert values:

In the Observe method, we use the return value and out parameter output to automatically log and assert those values. Additionally, we add “view input” on the fly to the parameter table through the ValueForViewing method, and we add “check input” to be asserted through the ValueAtEndOfTest method. After running Pex, we get the following results:

As expected, input, ‘view input’, output and result show up in the parameter table.

In the generated test, we see assertions for the return value, out parameters and other values passed through the ValueAtEndOfTest method.

Code Contracts : Reproducible generated tests

When Pex generates a unit test that relied on a runtime contract, Pex also adds a check to the unit test which validates that the contracts have been injected into the code by the contracts rewriter. If the code is not rewritten when re-executing the unit test, it is marked as inconclusive. You will appreciate this behavior when you run your unit tests both in Release and in Debug builds, which usually differ in how contracts get injected.

Code Contracts:  Automatic filtering of the contract violations

When Pex generates a test that violates a Code Contract pre-condition (i.e. Contract.Requires), there are basically two scenarios: the precondition was on top of the stack and should be considered as an expected exception; or it is a nested exception and should be considered as a bug. Pex provides a default exception filtering that implements this behavior.

Stubs: simplified syntax

We’ve considerably simplified the syntax of stubs by removing the ‘this’ parameter from the stub delegate definition. Let’s illustrate this with a test that stubs the ‘ReadAllText’ method of a fictitious ‘IFileSystem’ interface. 

Stubs: generic methods

The Stubs framework now supports stubbing generic methods by providing particular instantiations of that method. In the following example, the generic Bar<T> method is stubbed for the particular Bar<int> instantiation:

Stubs and Pex: Pex will choose the stubs behavior by default

We provide a new custom attribute, PexChooseAsStubFallbackBehaviorAttribute, that hooks Pex choices to the Stub fallback behavior. To illustrate what this means, let’s modify slightly the example above by removing the stub of ReadAllText:

If this test was to be run without the PexChooseAsStubFallbackBehavior attribute, it would throw a StubNotImplementedException. However, with the PexChooseAsStubFallbackBehavior attribute, the fallback behavior calls into PexChoose to ask Pex for a new string. In this example in particular, on each call to ReadAllText, Pex will generate a new string for the result. You can see this string as a new parameter to the parameterized unit test. Therefore, when we run this test under Pex, we see different behavior happening, including the “hello world” file:

Note that all the necessary attributes are added at the assembly level by the Pex Wizard.

Miscellanous bug fixes and improvements

• [fixed] Dialogs do not render correctly under high DPI
• When a generic parameterized unit tests does not have any generic argument instantiations, Pex makes a guess for you.
• When a test parameter is an interface or an abstract class, Pex now searches the known assemblies for implementations and concrete classes. In particular, that means that Pex will often automatically use the automatically generated Stubs implementations for interfaces or abstract classes.
• Static parameterized unit tests are supported (if static tests are supported by your test framework)
• Better solving of decimal and floating point constraints. We will report on the details later.

Breaking Changes

• The PexFactoryClassAttribute is no longer needed and has been removed. Now, Pex will pick up object factory methods marked with the PexFactoryMethodAttribute from any static class in the test project containing the parameterized unit tests. If the generated tests are stored in a separate project, that project is not searched.
• The PexStore API has been renamed to PexObserve.
• Pex is compatible with Code Contracts versions strictly newer than v1.1.20309.13. Unfortunately, v1.1.20309.13 is the currently available version of Code Contracts. The Code Contracts team is planning on a release soon.

Happy Pexing!