We just released a new version of Pex, v0.15.40714.1. This version does not contain any new features but… it brings better memory management and bug fixes for issues that were reported through our MSDN forums. Happy testing!

It contains a couple breaking changes too:

• Microsoft.Pex.Stubs.dll does not exist anymore and has been integrated into Microsoft.Pex.Framework.dll
• PexExplorableFromFactoriesAttribute renamed to PexExplorableFromFactoriesFromAssemblyAttribute

We have just released a new version of Pex, v0.14.040610.2. This version brings a range of Bug Fixes that were reported on the MSDN forums and a couple new features: Fix Suggestions for Invariant Methods and Natural properties  for Stubs.

Fix Suggestions for Invariant Methods

When you use Code Contracts, then Pex can now help you to add missing invariants, just as Pex already inferred missing preconditions in the past. Take for example the ArrayList class that comes with the samples of our recently updated tutorial slide deck. This class has the following fields:

When you add an empty invariant method,

and enable Runtime Checking of contracts, then Pex will create instances of this class using reflection while making sure that the invariants holds (we announced this feature in the last release). However, with an empty invariant method, Pex will be able to create illegal states, e.g. where the _items array is null, which may cause a NullReferenceException, and other issues. To get the following test cases, we let Pex explore the Add method of our ArrayList class:

When you look at the details of the failing test cases, then you will see that Pex now offers the option to add invariants directly into your [ContractInvariantMethod]:

Natural Properties for Stubs

The logic to handle properties with setters and getters generated by Stubs has been slightly improved so that it is possible to force properties to have as if they had a backing field.

When a getter or a setter of a property is invoked, Stubs checks whether the getter delegate or the setter delegates have been set. If not, i.e. no custom user behavior, it queries the fallback behavior for an initial value of the property. If the fallback behavior successfully returns a value, Stubs generates a closure and binds the getter and setter.

For example, here’s a simple IFoo interface with a Bar property:

One could write the following parameterized unit test:

Since we expect the Bar property to behave as it was wrapping a field, we don’t expect the assertion to be raised. When we execute this test with Pex, and the Pex fallback behavior, Pex will choose a value for ‘Bar’, and this value will be ‘stored’ the property. Therefore, we get as expected 2 test cases:

Non default constructor support for Stubs

In previous versions, Stubs could not generate stubs for classes with no default constructor. This is no longer the case, Stubs will generate one public constructor for each public or protected base constructor.

Bug Fixes

• Various code generation tweaks to make StyleCop happier
• Visual Studio addin crashes when loading in VS2010
• Visual Studio addin does not find attributes when they are globally qualified
• Added missing substitutions for System.Threading.Interlocked members
• Stubs automatically imports missing references
• When an object has an invariant method (from Code Contracts), Pex would emit assertions using the private fields
• Editorial issues to patterns paper
• Contracts no longer kill the process when the runtime rewriter was not executed (.net 4.0)
• Fixed bug in generic type guesser
• Fixed samples for the latest version of Stubs

As always, don’t hesitate to send us your feedback, good or bad, through our MSDN forums.

Inversion of Control (IoC) is a very important practice to make code testable. It usually relies on defining interfaces for each external dependency. Various frameworks and techniques, i.e. Dependency Injection (DI), Service Locator, exist to bind the interface implementations to the components. In this blog post, we will *not* focus on this aspects of IoC but rather on it’s main building block: interfaces. This article describes how Contracts for interfaces improves IoC… regardless of which DI framework/technique you are using. The Contracts for interfaces are a feature of the Code Contracts for .Net tool.

Interfaces are not Contracts

Interfaces are often referred to as Contracts in the context of IoC: they define the methods and properties that a service should implement and are a key factor in achieving the loose coupling: one can build the code against the interface and can plug in various implementation with no risks.

Unfortunately, interfaces are not contracts: while they specify how the methods signatures should be, interfaces do not specify the functional behavior. To illustrate this, let’s take a look at a well-known simple interface of the BCL:

For this interface, I could naively implement as follows:

While my implementation is really really wrong, it fulfills all requirements of the IServiceProvider interface: a method GetService that returns object. Of course, if I would have read the MSDN documentation of GetService, I would have known that object should implement serviceType. Unfortunately, the interface did not tell me anything about that and compilers don’t understand MSDN documentation either.

Contracts for interfaces

Code Contracts provides an API for design-by-contracts (pre-condition, post-condition, etc…). It also supports defining contracts for interface or abstract classes. Contracts for interfaces can be used to specify the functional behavior of interface members. While they also serve as documentation, those contracts can also be turned into runtime checks or leveraged by static checkers.

• since interface members cannot have method bodies, the contracts are stored in a ‘buddy’ type.
• The [ContractClass]/[ContractClassFor] attributes bind the interface type and the contract type together.
• Contract.Result<object>() is a helper method to refer to the result value, since this is not supported in C#.

Let’s take a closer look at the body of IServiceProviderContract.GetService. It contains a pre-condition (Requires) that the serviceType should not be null and a post-condition (Ensures) that the return value should be null or should be assignable to the serviceType:

In this simple case, the contracts captured precisely the specification that we found in MSDN. The critical difference is that they are stored in a format that compilers and runtimes know pretty well: MSIL byte code. The benefits are huge:

• documentation: the contracts are stored in a programming-language agnostic format that mined and rendered for your favorite programming language,
• static checking: static analysis tools can (and will) use contract to try to find bugs before you execute the code, or prove that it is correct with respect to the contracts,
• runtime checking: the runtime checker will instrument all implementations of the interface with the interface contracts automatically. Once you’ve specified how an interface should behave, you do not have to repeat yourself when re-implementing it, the re-writer takes care of that.
• automated white box testing: tools like Pex can leverage the runtime contract checks to try to find inputs that violate the contracts.
• IoC/DI framework agnostic: It does not matter which DI framework you use, as soon as you use interface, you could also provide contracts for it.

We’ve just released the Pex 0.13.40528.2 (Nikolai has all the details about the new features). In this release, a number of attributes were added to force Pex produce more friendly inputs.

The problem with Enums

When you apply Pex to code that uses Enumerations, you quickly learn something: the CLR does not constraints the enumeration values to be defined. This means than whenever you take an Enum as a parameter, you should validate it is properly defined:

void Test(SomeEnum e) {
    if (!Enum.IsDefined(typeof(SomeEnum), e)) throw new ArgumentException(“…”);
}

Look at the BCL code, they actually do this kind of validation. For good or bad, some of our users have been complaining that Pex should restrict the enumeration to defined values, in fact considering the generated failing test case as noise. To restrict generated enumartion values to be defined, you can add the [PexEnumValuesDefined(typeof(SomeEnum))] attribute on your test/fixture/assembly. Internally, this attribute attaches an invariant to the given enumeration type that tells Z3 the allowed values for that type.

What about multi-dimensional arrays?

Ever wakened a day wanting to iterate arrays starting from –12345? Well you can with multidimensional arrays… The Array.CreateInstance method has an overload that takes length and lower bounds to instantiate multidimensional arrays:

int[,] array = (int[,])Array.CreateInstance(typeof(int),
    new int[] {10, 10}, // lengths
    new int[] {–12345, – 789} // lower bounds
    );
a[-12345, –789] = –1;

Of course, this feels a bit weird because most languages won’t allow to create such arrays, but the Array.CreateInstance method is still available nonetheless. It feels a bit weird but this behavior is perfectly valid from the runtime point of view. To restrict Pex from generating such bounds, you can add the [PexZeroMdArrayLowerBounds(0)] attribute (one attribute per dimension).

What about Booleans?

Booleans is another interesting citizen of the CLR. At the MSIL level, the Boolean type is really a byte, widened as an integer when on the stack. Compilers try really hard to keep Booleans as 0 or 1 but… it is possible to craft Boolean values anywhere the byte range using safe MSIL code (not from C# though or any BCL method I know of). The implication of this are quite disturbing, for example, consider the following parameterized test:

[PexMethod] {
void BooleanMadness(bool a, bool b) {
     if (a && b && a != b)
         throw new Exception(“this is madness!”);
}

Clearly, in a word of Booleans constrained to 0 or 1, the branch that throws the exception should be unreachable. If you execute Pex on this test, it will generate 4 test one of which raises the exception:

Notice the actual values (0x02, 0x03) of the Booleans in the pretty printing. If you want Pex to generate Boolean that are restricted to 0 or 1, use the [PexBooleanAsZeroOrOne] attribute. This attribute forces Z3 to pick boolean values as 0 or 1. Btw, there’s already a connect bug for this behavior.

If you’ve been thinking about presenting Pex to your co-workers or your local .NET community, you can use our slide decks at http://research.microsoft.com/en-us/projects/pex/documentation.aspx . The slide decks are there to help you, don’t hesitate to shuffle them, cut them or pick whatever you need in them (and of course, tell us about it).

Cheers, Peli.

If you are (or you've been) blogging about Pex, let us know and drop the link in this following forum thread:

http://social.msdn.microsoft.com/Forums/en-US/pex/thread/7e6c26df-ccfb-48a4-878d-b3b01f5ba8d5

We are building a list of links that will be hosted our project web site.

Not sure what’s going to come up at of this but you can find me on twitter at http://twitter.com/pelikhan.

This is a cool new feature that builds on Stubs, Pex  and Code Contracts  (in fact, it is really just a consequence of the interaction of the 3 technologies): stubs that act as parameterized models while complying to the interface contracts.

• Code Contracts provide contracts for interfaces. All implementations of that particular interface will be instrumented with the runtime checks by the rewriter at compile time.
• Stubs is really just a code generator that provides a minimal implementation of interfaces and relies on the C# compiler to build the code. Therefore, if a stub implements an interface with contracts, it will automatically be instrumented by the runtime rewriter as well.
• Pex choices (i.e. dynamic parameter generation) may be used as the fallback behavior of stubs. In other words, whenever a stubbed method without user-defined behavior is called, Pex gets to pick the return value. Since the stub implementation itself may have been instrumented with contracts, we’ve added special handling code so that post-condition violation coming from a stub are considered as an assumption. This effectively forces Pex to generate values that comply with the interface contracts.

Let’s see how this works with an example. The IFoo interface contains a single ‘GetName’ method. This method has one precondition, i should be positive and one postcondition, the result is non null. Since interfaces cannot have a method body, we use a ‘buddy’ class to express those contracts and bind both classes using the [ContractClass] and [ContractClassFor] attributes:

We then turn on runtime rewriting for both the project under test and the test project (Project properties –> Code Contracts –> Runtime checks). As a result, the stub of IFoo automatically gets instrumented with the contracts. We can clearly see that from the generated code of GetName in Reflector below:

The last step is to write a test for IFoo. In this case, we write a parameterized unit test that takes an IFoo instance and an integer, then simply call GetName.

Since the contracts of IFoo.GetName specifies that the result should not be null, we should not see any assertion violation in this test. Moreover, we should see a test for the precondition considered as an expected exception:

Note that all those behaviors rely on extensions to Pex that the wizard automatically inserted in the PexAssemblyInfo.cs file:

where

• [PexAssumeContractEnsuresFailureAtStubSurface] filters post-condition violations in stubs as assumptions,
• [PexChooseAsStubFallbackBehavior] installs Pex choices as the fallback behavior of stubs,
• [PexStubsPackage] load the stubs package (this is a standard procedure for Pex packages),
• [PexUseStubsFromTestAssembly] specifies to Pex that it should considers stub types when trying to test interfaces or abstract classes.

We just made a quick release to fix another installer issue related to missing packages. Along the way, we’ve added an Exploration Tree View and Partial Stubs support.

Exploration Tree View

The exploration tree view displays the list of explorations to be executed, running and finished. It serves as a progress indicator but also as a smooth result explorer. When browsing through the tree, Pex will synchronize the exploration result view and the tree view.

The tree view populates each namespace with the fixture types and exploration methods, and provide a visual feedback on the progress of Pex.

When you browse through the exploration and generated test nodes, Pex automatically synchronizes the exploration result display. This makes it really easy to start from an high-level view of the failures and drill into a particular generated test, with stack trace and parameter table.

Partial Stubs

Stubs lets you call the base class implementation of methods as a fallback behavior. This functionality is commonly referred as Partial Mocks or Partial Stubs and is useful to test abstract classes in isolation. Stubs inheriting from classes have a “CallBase” property that can turn this mode on and off.

Let see this with the RhinoMocks example on partial mocks. Given an abstract ProcessorBase class,

we write a test for the Inc method. To do so, we provide a stub implementation of Add that simply increment a counter.

Miscellaneous

• PexAssume/PexAssert.EnumIsDefine checks that a particular value is defined in an enum.
• Missing OpenMP files in the installer break Pex.

Poll: should we skip 0.13 and go straight for 0.14? :)

I recorded a Channel9 video last week with Manuel Fahndrich on the interaction of Code Contracts and Pex. The demo gives a glimpse at the nice interaction between Contracts (design by contracts) and Pex (automated white box).

Code Contracts gives you a great way to specify what your code is supposed to do. These contracts can be leverage for documentation, static checkers but also – tada – by Pex! Contracts can also be turned into runtime checks which Pex will try to explore. Pex will try to cover the post conditions, assertions or in other words, find inputs that violates of your contracts etc… By adding contracts to your code, you give Pex a ‘direction’ to search for (and a test oracle).

Drop me a note if you want more of those movies.