# Friday, April 10, 2009

Download the new version!

Today we’ve released a new release of Pex on DevLabs and on our academic downloads. This highlights of this release are: NUnit, MbUnit and xUnit.net support out of the box, writing parameterized unit tests in VisualBasic.NET and F#, better Code Contracts support. As always, if we encourage you to send us feedback, bugs, stories on our forums at http://social.msdn.microsoft.com/Forums/en-US/pex/threads/ .

NUnit, MbUnit and xUnit.net supported out of the box

Pex now supports MSTest, NUnit, MbUnit and xUnit.net out of the box. Pex will automatically detect which framework you are using by inspecting the assembly reference list, and automatically save the generated tests decorated with the correct attributes for that framework.

image

The default test framework can also be specified through the global options (Tools –> Options –> Pex –> enter the test framework name in TestFramework).

image

 

Writing Parameterized Unit Tests in VisualBasic.NET

While the Pex white box analysis engine works at the MSIL level, Pex only emits C# code for now. In previous releases, this limitation made it impossible to use Pex parameterized unit tests from non-C# code. In this release, we have worked around this problem by automatically saving the generated tests in a ‘satellite’ C# project.

Let’s see this with an example. The screenshot below shows a single VisualBasic.NET test project with a Pex parameterized unit test:

clip_image002

We can right-click in the HelloTest.Hello method and select “Run Pex Explorations”:

clip_image004

At this point, Pex will start exploring the test in the background as usual. This is where the new support comes in: When a generated test comes back to Visual Studio, Pex will save it in a separate C# project automatically (after asking you where to drop the new project):

clip_image006

The generated tests are now ready to be run just as any other unit tests!

Writing Parameterized Unit Tests from F#

Similarly to VisualBasic.NET, we’ve made improvements in our infrastructure to enable writing parameterized unit tests in F#. Let’s see this with a familiar example. We have a single F# library that has xUnit.net unit tests and reference Microsoft.Pex.Framework (project Library2 below). In that project, we add a parameterized unit test (hello_test):

clip_image008

We can right-click on the test method name and Pex will start the exploration of that test in the background. Because of the limitations of the F# project system, you absolutely need to right-click on the method name in F# if you want contextual test selection to work. Because the project is already referencing xunit.dll, Pex will also automatically detect that you are using xUnit.net and use that framework. When the first test case comes back to VisualStudio, Pex saves it in a separate C# project:

clip_image010

The tests are saved in the generated test project and ready to be run by your favorite test runner!

PexObserve: Observing values, Asserting values

We’ve completely re-factored the way values can be logged on the table or saved as assertions in the generated tests. The following example shows various ways to log and assert values:

image

In the Observe method, we use the return value and out parameter output to automatically log and assert those values. Additionally, we add “view input” on the fly to the parameter table through the ValueForViewing method, and we add “check input” to be asserted through the ValueAtEndOfTest method. After running Pex, we get the following results:

image

As expected, input, ‘view input’, output and result show up in the parameter table.

image

In the generated test, we see assertions for the return value, out parameters and other values passed through the ValueAtEndOfTest method.

Code Contracts : Reproducible generated tests

When Pex generates a unit test that relied on a runtime contract, Pex also adds a check to the unit test which validates that the contracts have been injected into the code by the contracts rewriter. If the code is not rewritten when re-executing the unit test, it is marked as inconclusive. You will appreciate this behavior when you run your unit tests both in Release and in Debug builds, which usually differ in how contracts get injected.

clip_image012

Code Contracts:  Automatic filtering of the contract violations

When Pex generates a test that violates a Code Contract pre-condition (i.e. Contract.Requires), there are basically two scenarios: the precondition was on top of the stack and should be considered as an expected exception; or it is a nested exception and should be considered as a bug. Pex provides a default exception filtering that implements this behavior.

Stubs: simplified syntax

We’ve considerably simplified the syntax of stubs by removing the ‘this’ parameter from the stub delegate definition. Let’s illustrate this with a test that stubs the ‘ReadAllText’ method of a fictitious ‘IFileSystem’ interface. 

Stubs: generic methods

The Stubs framework now supports stubbing generic methods by providing particular instantiations of that method. In the following example, the generic Bar<T> method is stubbed for the particular Bar<int> instantiation:

image

Stubs and Pex: Pex will choose the stubs behavior by default

We provide a new custom attribute, PexChooseAsStubFallbackBehaviorAttribute, that hooks Pex choices to the Stub fallback behavior. To illustrate what this means, let’s modify slightly the example above by removing the stub of ReadAllText:

image

If this test was to be run without the PexChooseAsStubFallbackBehavior attribute, it would throw a StubNotImplementedException. However, with the PexChooseAsStubFallbackBehavior attribute, the fallback behavior calls into PexChoose to ask Pex for a new string. In this example in particular, on each call to ReadAllText, Pex will generate a new string for the result. You can see this string as a new parameter to the parameterized unit test. Therefore, when we run this test under Pex, we see different behavior happening, including the “hello world” file:

image

Note that all the necessary attributes are added at the assembly level by the Pex Wizard.

Miscellanous bug fixes and improvements

  • [fixed] Dialogs do not render correctly under high DPI
  • When a generic parameterized unit tests does not have any generic argument instantiations, Pex makes a guess for you.
  • When a test parameter is an interface or an abstract class, Pex now searches the known assemblies for implementations and concrete classes. In particular, that means that Pex will often automatically use the automatically generated Stubs implementations for interfaces or abstract classes.
  • Static parameterized unit tests are supported (if static tests are supported by your test framework)
  • Better solving of decimal and floating point constraints. We will report on the details later.

Breaking Changes

  • The PexFactoryClassAttribute is no longer needed and has been removed. Now, Pex will pick up object factory methods marked with the PexFactoryMethodAttribute from any static class in the test project containing the parameterized unit tests. If the generated tests are stored in a separate project, that project is not searched.
  • The PexStore API has been renamed to PexObserve.
  • Pex is compatible with Code Contracts versions strictly newer than v1.1.20309.13. Unfortunately, v1.1.20309.13 is the currently available version of Code Contracts. The Code Contracts team is planning on a release soon.

 

Happy Pexing!

posted on Friday, April 10, 2009 12:06:31 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [12]
# Thursday, October 02, 2008

We are very excited to announce that Pex has a session at PDC 2008. We will be talking about code contracts and Pex, and how they play nicely together. Book it now in your conference agenda!!! (look ‘Research’ or ‘Pex’ in the session list).

image 

See you there and don’t forget to swing by our booth.

posted on Thursday, October 02, 2008 10:42:09 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Monday, September 22, 2008

How do you write good parameterized unit tests?
Where do they work the best?
Are there some Test Patterns ? Anti Patterns?

This is the kind of questions that we have received many times from Pex users. We just released Pex 0.7 which contains a list patterns and anti-patterns for parameterized unit testing (this is still a draft but we feel that we already have a number of good patterns that would be helpful for anyone giving a shot at Pex):

Note that most of the patterns in this document are not Pex specific and apply to parameterized unit tests in general; including MbUnit RowTest/CombinatorialTest/DataTest, NUnit RowTest, MSTest Data Test, etc…

The amazing ‘quadruple A’ pattern

The ‘triple A’ pattern is a common way of writing a unit test: Arrange, Act, Assert. Even more ‘A’crobatic, we propose the ‘quadruple A’ where we added one more ‘A’ for assumption:

image

Pex is an automated white box testing tool from Microsoft Research.
More information at http://research.microsoft.com/pex.

posted on Monday, September 22, 2008 3:41:15 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, August 26, 2008

Update: renamed project to YUnit to avoid clashing with other frameworks.

I've been playing with custom test types for team test lately and the result of this experiment is YUnit. A microscopic test framework that lets you write tests anywhere since it only uses the Conditional attribute. That's right, any public static parameterless method tagged with [Conditional("TEST")] becomes a test :)

If you've always dreamed of implementing your own custom test type, this sample could be helpful to you. Remember that this is only a sample and comes with no guarantees of support.

Sources and installer available at: http://code.msdn.microsoft.com/yunit.

 

posted on Tuesday, August 26, 2008 8:26:36 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, October 16, 2007

Update: I will not be at the Seattle Code Camp, too much rescheduling.

I'll be presenting Pex at the Seattle Code Camp in Nov.

Pex – Automated White Box Unit Testing

Parameterized unit testing is becoming a mainstream feature of most unit test frameworks; MbUnit RowTest (and more), VSTS data tests, xUnit.net Theories, etc... Unfortunately, it is still the responsibility of the developer to figure out relevant parameter values to exercise the code. With Pex, this is no longer true. Pex is a unit test framework addin that can generate relevant parameter values for parameterized unit tests. Pex uses an automated white box analysis (i.e. it monitors the code execution at runtime) to systematically explore every branches in the code. In this talk, Peli will give an overview of the technology behind Pex (with juicy low-level .NET profiling goodness), then quickly jump to exiting live demos.

posted on Tuesday, October 16, 2007 6:36:52 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Friday, September 07, 2007

In a previous, we were looking at partial trust the lack of support for it. In this post, I'll show the key 'fixes' that we did to make Pex 'partial trust' aware.

Simulating Partial Trust

The easiest way to run under partial trust is to run your .net application from the network. However, in the context of a test framework, this would not work since many required permissions would not be granted (reflection, i/o, etc...). So we need a new AppDomain whose security policy considers the test framework assemblies as fully trusted.

  • Get a new AppDomain:
string trust = "LocalIntranet";
AppDomain domain = AppDomain.CreateAppDomain(trust);
  • Load the named permission set
PermissionSet permission = GetNamedPermissionSet(trust);
  • Create the code group structure that associate the partial trust permission to any code
UnionCodeGroup code= new UnionCodeGroup(
    new AllMembershipCondition(),
    new PolicyStatement(permission, PolicyStatementAttribute.Nothing));
  • give full trust to each test framework assembly:
StrongName strongName = CreateStrongName(typeof(TestFixtureAttribute).Assembly);
PermissionSet fullTrust = new PermissionSet(PermissionState.Unrestricted);
UnionCodeGroup fullTrustCode = new UnionCodeGroup(
new StrongNameMembershipCondition(strongName.PublicKey, strongName.Name, strongName.Version),
new PolicyStatement(fullTrust, PolicyStatementAttribute.Nothing));
code.AddChild(fullTrustCode);
  • Assign the policy to the AppDomain
PolicyLevel policy = PolicyLevel.CreateAppDomainLevel();
policy.RootCodeGroup = code;
domain.SetAppDomainPolicy(policy);

This is basically it (the rest of the details are left as an exercise :)).

Let them call you

Make sure to add the AllowPartiallyTrustedCallers to the test framework assembly otherwize users won't be allowed to call into it...

A twist...

Pex is bit invasive when it comes to partial trust. Pex rewrites the IL at runtime and turns all method bodies into... unsafe code (that is unverifiable). At this point, any will not run because of the SkipVerification permission.

No problemo, just add it to the permissionset:

permission.AddPermission(
    new SecurityPermission(SecurityPermissionFlag.SkipVerification)
    );

 

posted on Friday, September 07, 2007 11:39:22 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Thursday, August 23, 2007

A common requirement for unit test framework is the ability to test internal types.

That's easy! use InternalsVisibleToAttribute

With .Net 2.0 and up, this is a fairly easy task thanks to the InternalsVisibleToAttribute: add it to the product assembly to give 'visibility rights' to the test assembly.

// in assembly Foo
internal class Foo {}
// giving visibility rights to the Foo.Tests assembly
[assembly:InternalsVisibleTo("Foo.Tests")]

On the test assembly side, this works because unit test are 'closed' methods which do not expose any internal types.


[Test]
public void FooTest() {
    Foo foo = new Foo(); // we're using the internal type Foo
                         // but it's hidden in the unit test
}

What about parameterized tests? Make them internal as well

If one of the parameters of the test is internal, the test method will have to be internal as well in order to compile:

[PexTest]
internal void FooTest(Foo foo) {
   ...
}

Not pretty but still gets the job done. Pex will generate public unit test methods that invoke the internal parameterized test method, and we'll be happy:

[Test]
public void FooTest_12345() {
    this.FooTest(null);
}

What about MbUnit RowTest?

This issue was never faced by MbUnit RowTest because it only accepts intrinsic types such as int, long, etc... Those types are obviously public :)

posted on Thursday, August 23, 2007 10:24:08 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, May 26, 2007

.Net 2.0 has been out for a while and it seems that 'generics' have not made it into unit test frameworks (that I know of). When I write unit tests for generics, I don't want to have to instantiate them!

For example, if I have an generic interface,

interface IFoo<T> {...}

then I'd really like to write this kind of test and let the test framework figure out an interresting instantiation (i.e. choice of T):

[Test]
public void Test<T>(IFoo<T> foo) { ... }

In the example above, System.Object can be trivially used to instantiate IFoo<T>. Of course, things get more interresting when mixing type argument, method argument and constraints :)

interface IFoo<T>
{
    R Bar<R>(T t)
     where R : IEnumerable<T>
}

In Pex, we've started to look at this problem... stay tuned.

posted on Saturday, May 26, 2007 11:09:00 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [3]
# Tuesday, April 10, 2007

MbUnit supports different flavors of parameterized unit tests: RowTest, CombinatorialTest, etc... If you are already using those features, it would be very easy for you to 'pexify' them:

namespace MyTest
{
     using MbUnit.Framework;
     using Microsoft.Pex.Framework;

     [TestFixture, PexClass]
     public partial class MyTestFixture
{
[RowTest]
[Row("a", "b")]
[PexTest]

public void Test(string a, string b)
{
...
}


}
}

Isn't this nice? :)

Some little notes:

  • 'partial' is helpfull to emit the generated unit test in the same class... but not the same file. Pex also support another mode where partial is not required.
  • the Pex attributes do not 'interfere' with the MbUnit ones. Your unit tests will still run exactly the same with MbUnit.
posted on Tuesday, April 10, 2007 9:47:27 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Thursday, March 08, 2007

Pex

I'm thrilled to present the project I joined last October: 'Pex' (for Program EXploration). Pex is a powerfull plugin for unit test frameworks that let the user write parameterized unit tests**. Pex does the hard work of computing the relevant values for those parameters, and serializing them as classic unit tests.

Here's a short screencast where we test and implement a string chunker. In the screencast, we use a parameterized unit test to express that for *any* string input and *any* chunk length, the concatenation of the chunks should be equal to the original sting.

http://research.microsoft.com/pex/screencast.aspx

More info on Pex is available at http://research.microsoft.com/pex/.

** It's actually much more than that... but let's keep that for later :)

posted on Thursday, March 08, 2007 9:58:36 AM (Pacific Standard Time, UTC-08:00)  #    Comments [5]
# Saturday, September 09, 2006

After 2 years in the CLR, I'm moving job (and building) to Microsoft Research. I will be working on Parametrized Unit Testing.

 

posted on Saturday, September 09, 2006 11:21:39 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [3]
# Wednesday, July 21, 2004

The API Design Guidelines encourage developers to check all their arguments and thereby avoid throwing a NullReferenceException.  If an argument is null and the contract of the method forbids null arguments an ArgumentNullException should be thrown. Brad Adams 

So you agree with Brad (I do) and you always check that arguments are not null before using them. This means a little bit more of code but it is worth it. But this means also a lot more of test code because, ideally, you should test that all your methods check all their arguments. This means writing hundreds of boring, repetitive test cases.... and you don't want to do that.

At least I don't so I added a new feature to MbUnit that does it for me.

Test for ArgumentNullException, first iteration:

 Let's see how it works with an example:

public class ArgumentNullDummyClass
{
    public object ClassicMethod(Object nullable, Object notNullable, int valueType)
    {
        if (notNullable == null)
            throw new ArgumentNullException("notNullable");
        return String.Format("{0}{1}{2}",nullable,notNullable,valueType);
    }
}

As one can see, the nullable parameter can be null, while the notNullable parameter is tested. Now, let's create a fixture that tests this method. We will be using the TestSuiteFixture because we will build a TestSuite:

[TestSuiteFixture]
public class MethodTestSuiteDemo
{
    public delegate object MultiArgumentDelegate(Object o,Object b, int i);
    
    [TestSuite]
    public ITestSuite AutomaticClassicMethodSuite()
    {
        ArgumentNullDummyClass dummy = new ArgumentNullDummyClass();

        MethodTester suite = new MethodTester(
            "ClassicMethod",
            new MultiArgumentDelegate(dummy.ClassicMethod),
            "hello",
            "world",
            1
            );
            suite.AddAllThrowArgumentNull();
        return suite.Suite;
    }
}

The MethodTester class takes the following argument: a name, a delegate  and valid parameters of the delegate. By valid I mean parameters that should not make the delegate invokation fail. The AddAllThrowArgumentNull looks for nullable parameters and create a TestCase that will invoke the delegate with the corresponding parameter nulled. In the example, this means that ClassicMethod will be called with:

  • null, "world", 1
  • "hello", null, 1

Test for ArgumentNullException, second iteration:

There are things I don't like in the example above:

  • you need to create a delegate (tedious),
  • you need to create 1 method tester per method (tedious),

Ok, so let's build a ClassTester class that does that for us... The test code now looks as follows:

[TestSuiteFixture]
public class ClassTesterDemo
{
    [TestSuite]
    public ITestSuite AutomaticClassSuite()
    {
        ArgumentNullDummyClass dummy = new ArgumentNullDummyClass();
        ClassTester suite = new ClassTester("DummyClassTest",dummy);
        suite.Add("ClassicMethod","hello","world",1);
        return suite.Suite;
    }
}

That's much better: delegate is gone and we could add more methods to be tested in a single call.

Test for ArgumentNullException, third iteration:

There is still one problem with this technique: there is no way to tell that an argument is authorized to be nulled! In the example, the nullable parameter can be null and the TestCase will always fail because it does not throw ArgumentNullException.

The solution of this problem is done in two steps: first, you, the developper, tag the parameters that can be nulled with a NullableAttribute attribute (could be any of your attributes). In the example, we add a SmartMethod method and the MyNullableAttribute:

[AttributeUsage(AttributeTargets.Parameter,AllowMultiple=false,Inherited=true)]
public class MyNullableAttribute : Attribute
{}

public class ArgumentNullDummyClass
{
    public object ClassicMethod(Object nullable, Object notNullable, int valueType)
    {...}
    public object SmartMethod([MyNullable]Object nullable, Object notNullable, int valueType)
    {...}
}

Next, you must tell MbUnit which attribute is used to tag nullable parameters. This is done with the NullableAttributeAttribute at the assembly level:

[assembly: NullableAttribute(typeof(MbUnit.Demo.MyNullableAttribute))]

Ok, now we just need to update our test case to load the SmartMethod:

[TestSuite]
public ITestSuite AutomaticClassSuite()
{
    ArgumentNullDummyClass dummy = new ArgumentNullDummyClass();
    ClassTester suite = new ClassTester("DummyClassTest",dummy);

    suite.Add("ClassicMethod","hello","world",1);
    suite.Add("SmartMethod","hello","world",1);

    return suite.Suite;
}

The result in MbUnit GUI is as follows: the parameters of ClassicMethod were all tested, nullable included which we want to avoid. The parameters of SmartMethod were all tested excluded nullable because it was tagged. :)

Test for ArgumentNullException, fourth iteration:

 

The more I think about this problem, the more I think FxCop should do that for us...

posted on Wednesday, July 21, 2004 10:37:00 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]