Wednesday, December 05, 2007

In the previous post, we went through the exploration testing process to exercise a simple method, CheckPositive. In this post, we'll try the same exploration testing, but will let Pex do it.

// mehod under test
1 void CheckPositive(int i, bool @throw) {
2     if (i < 0) {      
3          Console.WriteLine("not ok");
4          if (@throw)
5             throw new ArgumentException();
6     }
7     else
8         Console.WriteLine("ok");
9 }
// hand-crafted unit tests
[TestMethod] void Zero() {
     CheckPositive(0, false);
}
[TestMethod] void MinusOne() {
     CheckPositive(-1, false);
}
[TestMethod] void MinusOneAndThrow() {
     CheckPositive(-1, true);
}

Exploration testing with Pex

To let Pex explore the CheckPositive, we write a little test wrapper around that method:

[TestClass, PexClass]
public partial class ExplorationTesting {
    [PexMethod]
    public void Test(int i, bool @throw) {
        CheckPositive(i, @throw);
    }

We also instrumented the original method with additional methods to track down the path conditions that Pex computes along the execution traces. Pex generates 3 pairs of values which are equivalent to what the test we manually created:

  • 0, false
  • int.MinValue, false
  • int.MinValue, true (throws)
posted on Thursday, December 06, 2007 12:54:15 AM UTC  #    Comments [1]
 Tuesday, December 04, 2007

In the previous post, we clarified what Pex was not doing. So what does it do?

Pex performs some kind of automated exploration testing. I'll dive deeper into the details of this, but let's start with an example that gives the high level idea of the methodology.

Exploration Testing

Let's start by doing some exploration testing on a simple method that checks that an integer is positive, CheckPositive:

1 void CheckPositive(int i, bool @throw) {
2     if (i < 0) {      
3          Console.WriteLine("not ok");
4          if (@throw)
5             throw new ArgumentException();
6     }
7     else
8         Console.WriteLine("ok");
9 }

One way to 'explore' this method would be to throw different values at CheckPositive and use the debugger to see what's happening.

Iteration 1: pick the default

Let's create a unit test that does exactly that and step into the debugger. Since we don't really know anything about CheckPositive yet, we simply pick 0 for i (actually default(int)).

[TestMethod]
void Zero() {
     CheckPositive(0, false);
}

When we reach the statement "Console.WriteLine..." on line 8, we can figure out that we took this branch because the condition "i < 0" on line 2 evaluated to false. Let's remember this and continue on.
          line 2, i < 0 == false, uncovered branch

The execution continues and the test finished successfully.

Iteration 2: flip the last condition

In the previous run, we've remembered that some code was not covered on line 3. We also know that this code path was not covered because the condition "i < 0" evaluated to false. At this point, we usually intuitively figure out a value of 'i' in our brain, to make this condition true. In this case, we need to solve "find i such that i < 0". Let's pick -1.

[TestMethod]
void MinusOne() {
     CheckPositive(-1, false);
}

We run the test under the debugger. As expected on line 2, the condition evaluates to true, and the program takes the other branch that in the previous test.
The program continues and reaches line 4 where another if statement branch. Since the condition  "@throw" evaluates to false, we take the branch that throws and remember the condition:
          line 4, @throw == false, uncovered branch

The program continues to run and finishes.

Iteration 3: path condition + flipped condition

We've still some uncovered branch to cover in the method, 'guarded' by the condition at line 4. To be able to cover this code, we need 2 things:
      1) reach line 4: i < 0
      2) make the condition in line 4 evaluate to true: @throw == true

So to cover the last statement in the method, we need to find parameter values such that
                  i < 0 && @throw == true

Let's pick -1, and true.

[TestMethod]
void MinusOneAndThrow() {
     CheckPositive(-1, true);
}

The test executes and now throws an exception as we wanted. At this point, we've fully covered to behavior of CheckPositive.

What about Pex?

Pex uses the same 'exploration' methodology as above. Pex executes a parameterized unit tests over and over and tries to cover each branch of the program. As it executes more code, it learns about new branches to cover etc... To find the input, Pex uses a constraint solver, Z3.

posted on Wednesday, December 05, 2007 3:17:38 AM UTC  #    Comments [2]

Nikolai Tillmann published a paper on DySy, a tool that can infer likely invariants by monitoring the code that is running.

DySy is was built on top of the infrastructure that Pex uses to generate test cases. In fact, it is one of our sample application :)

posted on Wednesday, December 05, 2007 12:53:17 AM UTC  #    Comments [0]
 Friday, November 30, 2007

I realized that I had not talked much about how Pex computes the values for the test parameters... the most important part of the tool!

It's not ...

Let's start by getting to wrong ideas out of the way. Parameterized tests are nothing new, they exist in MbUnit, VSTS, XUnit.Net, FIT, etc... so what's different with Pex?

  • it is not random: when it comes to generate data, the easiest solution is to plugin a random generator. If the state space is big enough (e.g. integers 2^32), it highly unlikely that random tests will find the interesting corner cases,

if (i == 123456)
     throw new Exception(); <---------- random won't find this

  • it does not require ranges or hints for the data: Pex does not require annotation to specify the range of particular inputs. All the relevant input values are inferred from the code itself (we'll see later how).
  • it is not pairwize testing: following the comment above, Pex does use a pairwize approach.
  • it is not data testing: data testing such as FIT or MbUnit RowTest usually consists of rows containing a set of inputs and the expected output. In Pex, you cannot provide the expected output as a 'concrete' value, you need to express it as code (through assertions for example). This is a subtle difference that radically changes the way you write your tests.

[RowTest, Row(0, 1, 1), Row(1, 0, 1)] // data test for the addition
void AddTest(int a, int b, int result)
{   Assert.AreEqual(result, a + b);  }

[PexMethod] // 0 is the neutral of the addition operation
void ZeroNeutralTest(int b)
{   Assert.AreEqual(b, 0 + b);  }

  • it is not a static analysis tool: Pex does a dynamic analysis of the code; it analyses the code that *is* running. Pex does this by rewriting the IL before it's jitted and instrumenting it with (many) callbacks to track precisely which IL instruction is being run by the CLR. So yes, Pex analyses the IL but on the fly rather than 'statically'.

Ok, now we've got a better idea of what Pex is not. So how does it work? ....

posted on Friday, November 30, 2007 5:19:05 PM UTC  #    Comments [3]
 Friday, October 19, 2007

In it's Weekly Source Code, Scott Hanselman presents a new CodePlex project, NDepend.Helpers.FileDirectoryPath from Patrick Smacchia. Nice, better path handling should have been part of the BCL a while ago. 

Path stuff is hard

Path normalization and parsing is not an easy task so when Patrick Smacchia mentions that his code "100% unit-tested", I decided to see if Pex could not find a little bug over there.

A dumb parameterized unit test

So I added wrote the following parameterized unit test, which 'simply' calls the constructor of FilePathRelative. Under the hood, there is some string manipulation done by the library to normalize the path, it should be interresting to see what comes out of this. I also added calls to PexComment.XXX to log the input/output values (Pex will build a table out of this):

    [PexMethod]
    public void FilePathRelativeCtor(string path) {
        PexComment.Parameters(path);
        FilePath result = new FilePathRelative(path);
        PexComment.Value("result", result.FileName);
    }

Ooops

So Pex starts running and soon enough an assert pops up. Pex had just found a neat little path that broke an assertion in the library:

[Test]
public void FilePathRelativeCtor_String_71019_003302_0_05() {
    this.FilePathRelativeCtor("/");
}

Popping up the reports, I went for the parameter table (remember the PexComment calls) that shows one row for each generated test. In fact, the 5-th test that Pex generated was triggering the assert:

Note that "//" also triggers the bug which seems to indicate that any path finishing by "/" will have this behavior.

The path condition

Lastly, I took a quick look at the path condition that Pex solved to discover the bug (see red below). Luckily this one is fairly easy and one can clearly see 'path[0] == '/' in there.

 What did we learn today?

Handling paths is hard :)

Also, we saw that a dumb parameterized unit test (just calling a ctor), could find a bugs. If you use assertions, it will help Pex look for bugs in your code.

posted on Friday, October 19, 2007 3:46:37 PM UTC  #    Comments [3]
 Wednesday, October 17, 2007

Update: I will not be at the Seattle Code Camp, too much rescheduling.

I'll be presenting Pex at the Seattle Code Camp in Nov.

Pex – Automated White Box Unit Testing

Parameterized unit testing is becoming a mainstream feature of most unit test frameworks; MbUnit RowTest (and more), VSTS data tests, xUnit.net Theories, etc... Unfortunately, it is still the responsibility of the developer to figure out relevant parameter values to exercise the code. With Pex, this is no longer true. Pex is a unit test framework addin that can generate relevant parameter values for parameterized unit tests. Pex uses an automated white box analysis (i.e. it monitors the code execution at runtime) to systematically explore every branches in the code. In this talk, Peli will give an overview of the technology behind Pex (with juicy low-level .NET profiling goodness), then quickly jump to exiting live demos.

posted on Wednesday, October 17, 2007 8:36:52 AM UTC  #    Comments [0]
 Thursday, October 11, 2007

When someone is writing a book that contains code snippets, the question of (automatically) keeping those in sync quickly becomes very imporant. There's already lots of different solutions to this problem (every author has probably it's own), here's yet another one for C# that we've developed to author the Pex documentation.

Goals

A couple things that we wanted to acheive with this tool:

  • snippets are always compilable and run as expected,
  • snippets can be full classes, methods or even partial statements
  • simple :)

'#region' based solution

This solution uses the #region directive to define a snippet. The region describe contains the snippet name, which will be used to dump it into a file. For example, given this piece of C#,

...
#region snippet StackExamplePart3
stack.Push(new object);
#endregion
...

Our parse will extract the code in the region and write it to StackExamplePart3.tex, which gets pulled in our LaTeX scripts.

\begin{verbatim}
stack.Push(new object);
\end{verbatim}

That's it?

Yes, you can author snippets that stay compilable and up to date:

  • we can author all the snippets in Visual Studio and we are sure they always compile
  • it's very easy to parse the #region's (left as exercise ;))
  • #region are very flexible in terms what they contain so we can have snippets containing partial methods, statement, etc...
  • the scheme aslo supports nested regions which is usefull when one explains an example line by line, and integrate the entire sample at the end. For example, DeclaringUnitTest is a 'sub'-snippet of UnitTest:
#region snippet UnitTest
#region snippet DeclaringUnitTest
[TestMethod]
void Test(int i)
#endregion
{
   
}
#endregion
  • we can integrate our snippets in unit tests and verify they work as expected
  • the tool can be integrated into the build process as a post-command build

 

posted on Thursday, October 11, 2007 1:41:32 PM UTC  #    Comments [2]
 Sunday, September 30, 2007

In Pex, we have our own version of all (and more) collection classes from the BCL. This duplication may sound stupid but there's a very good reason behind it: since Pex instruments code (i.e. rewrites IL), the BCL collections might be instrumented as well... leading to a poorer performance (instrumented code is slower for many reasons). Therefore, to avoid this situation, we ended up using our own implementation which never gets instrumented.

While using our collections, we added 2 features that we like a lot: readonly interfaces and small collections.

Readonly interfaces

Readonly interfaces are handy to 'safely' expose collections, once you start using them you get hooked:

  • ICountable<T>, a readonly enumerable with a Count.
interface ICountable<T> : IEnumerable<T>
{
    int Count {get;}
}
  • ICopyable<T>, a countable that can be copied around,
interface ICopyable<T> : ICountable<T>
{
    void CopyTo(T[] array, int index);
}
  • IIndexable<T>, a indexed collection
interface IIndexable<T> :  ICopyable<T>
{
    T this[int index] { get; }
}

We also have a couple more interfaces for dictionary style collections but you get the idea.

Small Collections

Small collections are optimized collections with 0, 1 or 2 elements. Depending on your scenario, these can improve the performance of your application while lowering the pressure on the GC (create less objects -> less stuff to garbage collect). If you create a lot of objects which each have collections, ask yourself: are you lazy allocating the collections? what should be the initial size of the collection? do you expect more than one element on average?

posted on Sunday, September 30, 2007 1:33:15 PM UTC  #    Comments [0]
 Sunday, September 23, 2007

xUnit, the new variation on the 'unit test framework' theme comes with support for data driven tests: 'Theories' (funny name by btw). Pex is a plugin for test frameworks, so we've added support for xUnit as well.

[PexClass] // xUnit does not have fixture attributes
public class MyTests
{
    [Theory, DataViaXXXX] // xUnit theories
    [PexTest] // let pex help you 
    public void Test(int a, ....)
    {}
}

posted on Sunday, September 23, 2007 11:26:32 PM UTC  #    Comments [0]
 Friday, September 14, 2007

Z3 is the high performance theorem prover that Pex uses to solve constraint systems (and find bugs). You can now play with it too! 

http://research.microsoft.com/projects/z3/

Z3 is written in C++, but it has a .Net API to make you happy.

posted on Friday, September 14, 2007 7:01:41 AM UTC  #    Comments [1]
 Saturday, September 08, 2007

In a previous, we were looking at partial trust the lack of support for it. In this post, I'll show the key 'fixes' that we did to make Pex 'partial trust' aware.

Simulating Partial Trust

The easiest way to run under partial trust is to run your .net application from the network. However, in the context of a test framework, this would not work since many required permissions would not be granted (reflection, i/o, etc...). So we need a new AppDomain whose security policy considers the test framework assemblies as fully trusted.

  • Get a new AppDomain:
string trust = "LocalIntranet";
AppDomain domain = AppDomain.CreateAppDomain(trust);
  • Load the named permission set
PermissionSet permission = GetNamedPermissionSet(trust);
  • Create the code group structure that associate the partial trust permission to any code
UnionCodeGroup code= new UnionCodeGroup(
    new AllMembershipCondition(),
    new PolicyStatement(permission, PolicyStatementAttribute.Nothing));
  • give full trust to each test framework assembly:
StrongName strongName = CreateStrongName(typeof(TestFixtureAttribute).Assembly);
PermissionSet fullTrust = new PermissionSet(PermissionState.Unrestricted);
UnionCodeGroup fullTrustCode = new UnionCodeGroup(
new StrongNameMembershipCondition(strongName.PublicKey, strongName.Name, strongName.Version),
new PolicyStatement(fullTrust, PolicyStatementAttribute.Nothing));
code.AddChild(fullTrustCode);
  • Assign the policy to the AppDomain
PolicyLevel policy = PolicyLevel.CreateAppDomainLevel();
policy.RootCodeGroup = code;
domain.SetAppDomainPolicy(policy);

This is basically it (the rest of the details are left as an exercise :)).

Let them call you

Make sure to add the AllowPartiallyTrustedCallers to the test framework assembly otherwize users won't be allowed to call into it...

A twist...

Pex is bit invasive when it comes to partial trust. Pex rewrites the IL at runtime and turns all method bodies into... unsafe code (that is unverifiable). At this point, any will not run because of the SkipVerification permission.

No problemo, just add it to the permissionset:

permission.AddPermission(
    new SecurityPermission(SecurityPermissionFlag.SkipVerification)
    );

 

posted on Saturday, September 08, 2007 1:39:22 PM UTC  #    Comments [0]
 Wednesday, September 05, 2007

Writing code that will run in partial trust is tricky... and writing unit tests for it turns out to be even trickier. Does your favorite unit test framework support partial trust?

After looking on NUnit, MbUnit or VSTT, it did not look like so (if I overlooked this feature, don't hesitate to point it out to me). No magic command line flag or custom attribute to get the work done.

Even more interesting, the test framework assemblies don't have the AllowPartiallyTrustedCallersAttribute... so you can't even use their assertion classes in partial trust as it will trigger a SecurityException... Good stuff.

posted on Wednesday, September 05, 2007 12:01:05 PM UTC  #    Comments [1]
 Thursday, August 23, 2007

A common requirement for unit test framework is the ability to test internal types.

That's easy! use InternalsVisibleToAttribute

With .Net 2.0 and up, this is a fairly easy task thanks to the InternalsVisibleToAttribute: add it to the product assembly to give 'visibility rights' to the test assembly.

// in assembly Foo
internal class Foo {}
// giving visibility rights to the Foo.Tests assembly
[assembly:InternalsVisibleTo("Foo.Tests")]

On the test assembly side, this works because unit test are 'closed' methods which do not expose any internal types.


[Test]
public void FooTest() {
    Foo foo = new Foo(); // we're using the internal type Foo
                         // but it's hidden in the unit test
}

What about parameterized tests? Make them internal as well

If one of the parameters of the test is internal, the test method will have to be internal as well in order to compile:

[PexTest]
internal void FooTest(Foo foo) {
   ...
}

Not pretty but still gets the job done. Pex will generate public unit test methods that invoke the internal parameterized test method, and we'll be happy:

[Test]
public void FooTest_12345() {
    this.FooTest(null);
}

What about MbUnit RowTest?

This issue was never faced by MbUnit RowTest because it only accepts intrinsic types such as int, long, etc... Those types are obviously public :)

posted on Friday, August 24, 2007 12:24:08 AM UTC  #    Comments [0]

Finally found the time and motivation to upgrade the Reflection.Emit language to Reflector 5.0. Available now at http://www.codeplex.com/reflectoraddins .

image

posted on Thursday, August 23, 2007 7:26:55 AM UTC  #    Comments [1]
 Saturday, July 14, 2007

Pex can analyze regular expressions*** and generate strings that matches them automatically!

What does it mean? Well, maybe, somewhere deep in your code your are validating some string with a regex (for example a url). In order to test the validation code, one needs to craft inputs that does not match (easy) and matches (harder) the regex.

Let Pex do it:

So what if Pex could be smart enough to understand a regex and craft inputs accordingly? In the example below, it would be very hard for a random generate to generate a string that matches the regex.

[PexTest]
public void Url([PexAssumeIsNotNull]string s)
{
    if (Regex.IsMatch(s, "(?<Protocol>\w+):\/\/(?<Domain>[\w@][\w.:@]+)\/?[\w\.?=%&=\-@/$,]*"))
        throw new PexCoverThisException(); // random won't find this
}

"" would be a failing match and foo://foo.com a good match. (To generate the correct match, my brain simulated the regex automaton and estimated one possible path). Interestingly, Pex generates....

Ā://Āā

Pretty ugly... but correct! This reminds us that while regex are used to validate input, what they'll let through is sometimes scary.

Compiled Regex + Pex = Love

The great part about supporting the regular expressions is that it comes for free (almost) since Regex can be compiled to IL in .Net. When the BCL generates the regex IL code, it effectily builds the automaton... which can be analyzed by Pex!!!

Refresher: Pex works analyzing the MSIL being executed.

Hey but not all Regex are compiled!

That's true. Compiling the regex is optional so Pex needs to do a little bit of 'plumbing' to make sure all regular expressions are compiled. This is simply done by substituting the real .ctor of the Regex class with a customized version that compiles the regex. I'll talk about substitutions deeper in the future.

*** Of course, the bigger the regex is, the harder it is going to be for Pex to craft a successful match.

posted on Sunday, July 15, 2007 4:49:53 AM UTC  #    Comments [1]
 Sunday, July 08, 2007

I finally posted a project that I had been developing last year in my daily commutes: Toad (www.codeplex.com/toad).

Toad is an (experimental) compiler infrastructure to build languages top of .Net, without knowing much about IL. Mostly it was a way for me to learn first hand all the little subtleties of writing multiple language on top of .Net.

What does Toad give you?

  • A dynamic console that supports multiple languages,
  • An Expression/Statement tree that can be translated back to IL,
  • Built-in debugging capabilities,
  • A set of Visitors on top this AST that implement various optimization and language constructs (this is the fun part),
  • There's a lot of stuff, I used to take the bus twice a day. :)
  • Toad does not use the DLR maybe in the future.

What is it good for?

  • Toy around with IL and try to build your micro language.
  • Don't use it for anything serious.

Good For Nothing Language

 The 'Good for nothing' language is a toy language that Joel Pobar used to illustrate managed IL compilers. With Gfn, you can write simple program such as:

var i = 123;
print i;
for j = i to 10 do print i+j; end;

The parsing stuff

I took Joel's example and wrote a grammar for the Gold parser. It looks like something like this that is then compiled by Gold:

<Program> ::= <BlockStatement>

<BlockStatement> ::= <BlockStatement> <Statement> ';'
            | <Statement> ';'

<Statement> ::= 'var' Identifier '=' <Expression>
    | Identifier '=' <Expression>
    | 'for' Identifier '=' <Expression> 'to' <Expression> 'do' <BlockStatement> 'end'
    | 'read_int' Identifier
    | 'print' <Expression>
    | 'return' <Expression>

At the end of the process, you get a visitor for the code AST. This is the boiler plate code to start using Toad to build the compiler:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected virtual object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
     throw new NotSupportedException("RuleStatementforIdentifierEqtodoend");
}

Implementing the 'for' loop

Let's take a look at the implementation of the for loop. We need to:

  • extract the from, to expressions,
  • declare a variable for the index,
  • build the predicate i < to etc...,
  • emit symbols for debugging,

The code below shows the full implementation of that statement:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected override object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
    // from, to
    Expression from = (Expression)this.VisitNode(node[3]);
    Expression to = (Expression)this.VisitNode(node[5]);

    // x = from
    VariableDeclarationStatement init = Stm.Var(node[1].Data, from);

    // body
    this.Context.PushScope();
    this.Context.PushVariable(init.Variable);

    Unit body = (Unit)this.VisitNode(node[7]);

    // i < to;
    Expression ilto = Expr.LessThanOrEqual(Expr.Var(init), to);
    ilto.Symbol = FromNode(node[1]);
    // ++i
    Statement inc = Stm.Expr(Expr.PrePlusPlus(Expr.Var(init)));

    this.Context.PopScope();

    ForStatement forstm = Stm.For(init, ilto, inc);
    forstm.Body = Stm.FromUnit(body);

    return forstm;
}

Hooking up the visitor in Toad

The last step is to package our visitor as a 'language' and return the generated statements or expression to Toad. We can spin up the interactive console, load Gfn and try it out:

image

Nothing really mind blowing here. Let's turn on live IL debugging to see what happening under the hood. This mode emits the IL source of the method, adds a bunch of 'nop' instructions to force the debugger to step on each IL instruction:

image

You can see in the generated code how the expressions got translated into IL instructions. 

posted on Monday, July 09, 2007 5:23:32 AM UTC  #    Comments [0]
 Thursday, July 05, 2007

Pex has a couple samples that get bundled in our installer. We have them in our main solution so that we get the benefits of Refactoring (that they still compiled :)). This approach has one little problem: the sample projects are referencing the Pex projects... which are not shipped! If packaged as is, they'll never compile on the user's machine.

ProjectReference -> Reference

We solved this problem by implementing an MSBuild task that 'patches' .csproj project files by replacing ProjectReference nodes with Reference:

<ProjectReference Include="..\..\..\Pex\Framework\Microsoft.Pex.Framework.csproj">
  ...
</ProjectReference>

becomes

<Reference Include="Microsoft.Pex.Framework">...</Reference>

There's a bit of coding involved to it (resolving the assembly name, selecting the appropriate projects, etc...) but that's basically it :)

posted on Thursday, July 05, 2007 11:02:38 PM UTC  #    Comments [0]
 Wednesday, July 04, 2007

One of the problem with Pex is that.... it's yet another dependency in your test project. Pex has it's own attributes which makes it very difficult to 'strip it out' of the test source. Many teams won't allow to check-in assemblies, they won't install external components on the build machine and just forget about touching the source code!

So how do you strip Pex?

In "Condition" lies the answer

An elegant solution uses a bit of Reflection, CodeDom and the MSBuild Condition attribute: generate Attribute stubs (shadows) and bind them to the project.

Pex modifies the project file and uses MSBuild conditions to conditionally include files and assembly references in the generated test project.

  • A boolean property PexShadows to controls the shadowing state: true if shadowed, false or missing otherwize.
<Project DefaultTargets="Build" ...>
  <PropertyGroup>
    ...
    <PexShadows>true</PexShadows>
  </PropertyGroup>
  • a conditional reference to Microsoft.Pex.Framework:
    <Reference 
        Include="Microsoft.Pex.Framework" 
        Condition="$(PexShadows) != 'true'" />

when the property PexShadows evalutes to true, the Microsoft.Pex.Framework assembly is not referenced anymore.

  • a file containing all the custom attributes 'stubs' (for all attributes in the Microsoft.Pex.Framework) is generated (automatically of course :)). Pex also dumps the source of several other helper classes. Each generated file is added to the test project conditionally:
    <Compile Include="Properties\PexAttributes.cs" Condition="$(PexShadows) == 'true'">
      <AutoGen>True</AutoGen>
      <DependentUpon>AssemblyInfo.cs</DependentUpon>
    </Compile>

That's it! Flip the value of PexShadows to bind/unbing your test project from Pex.

So what does Visual Studio says ?

Visual Studio is totally fooled by the maneuver. It shows the conditional files and references as if nothing happened. Add a menu item in the project context menu to switch it on and off and we are good to go :)

posted on Wednesday, July 04, 2007 12:28:12 PM UTC  #    Comments [0]
 Saturday, June 30, 2007

In Pex, we added the possibility to specify the type under test of a given fixture:

public class Account {...}


[TestFixture, PexClass(typeof(Account))]
public class AccountTest {...}  

That's nice but why would it be useful... Beyond the fact that it clearly expresses the 'target' of the fixture, this kind of information can be leverage by tools like Pex.

For example, since we know that Account is the type under test, we can tune Pex to prioritize the exploration of the Account type.

Another interesting side effect is the targeted code coverage data. Instead of getting coverage information over the entire assembly, we can directly provide coverage over the type under test: the AccountTest covered xx% of Account.

Still toying around the concept, one can add a special filtering mode to the command line to execute all tests that target 'Account':

pex.exe /type-under-test:Account Bank.Tests.dll
posted on Saturday, June 30, 2007 2:23:24 PM UTC  #    Comments [0]
 Sunday, June 03, 2007

A nice side effect of the Pex infrastructure is that it computes coverage data while exploring a parameterized test. What does this mean to the user? Well, what about this specifying that a parameterized test should reach 100% coverage**:

[PexTest]
[PexExpectedCoverage(100)] // expect 100% coverage
public void UberTest(int i)
{ ... }
** What kind of coverage are we talking about?

Pex internally tracks implicit branch coverage (which can never be covered :)). So we usually refer to basic block coverage. By default, the coverage is reported for the code that was run by the parameterized test. This is a very important difference with usual coverage tools, which give numbers for the entire assembly.

posted on Sunday, June 03, 2007 12:24:56 PM UTC  #    Comments [0]
 Saturday, June 02, 2007

This post gives a little introduction on the integrated mock support in Pex (it should also make clear why we couldn't reuse existing (and excellent) mock framework).

As other frameworks, Pex support mocking interfaces or virtual methods in non sealed classes.

Choose and replay

Pex Mocks have 2 modes of execution that are quite different:

  • when Pex is looking for new input, they act as a source of parameter choices, (this will be clearer in the example below).
  • when executing the generated unit tests, they act as stubs.
Micro example

Let us illustrate this by mocking a simple interface that returns the name of an instance:

interface INamed
{
    string GetName();
}
Writing the mock

The mock for this interface looks as follows:

[PexMock] // specifies to pex that this type is a mock
public class MNamed : INamed
{
    public string GetName()
    {
        IPexMethodCallOracle call = PexOracle.MethodCall(this);
        // let's ask Pex for a name    
        return call.ChooseResult<string>(); 
    }
}

So what did we do really: The mock implementation of GetName fetches a value oracle,  an interface from which new values can be queried:

IPexMethodCallOracle call = PexOracle.MethodCall(this); 

Then, this oracle can be used to choose new values. It's like adding parameters to the test, but not through the test signature. In this case, we ask Pex to get a new result value. Pex will choose that value as if it was a parameter of the test:

return call.ChooseResult<string>();
Let's try it!

The following test takes a INamed instance (which we assumed to be non-null) and displays a message on the console if the name is equal to'Marc':

[PexTest]
public void GetName(INamed name)
{
    if (name.GetName() == "Marc")
        Console.WriteLine("hello marc");
}
Generated tests

Pex generates 2 unit tests that fully covers the code of method: Pex tracked the data returned by ChooseResult and computed that this value had to be equal to "Marc" to hit a different path in the execution.

The first test has GetName return a default value. The second generated test is more interresting as it shows how Pex sets up the behavior of the replay stub:

// test 2
MockTest.MNamed mn0 = new MockTest.MNamed();
IPexOracleRecorder oracle = PexOracle.NewTest();
oracle.OnCall(0, typeof(MockTest.MNamed), "GetName")
      .Returns("Marc");
this.GetName(mn0);

The oracle instance is used to set up the result value on the first call to GetName:

// on the call '0' to the method GetName of MockType.MNamed,
oracle.OnCall(0, typeof(MockTest.MNamed), "GetName")
      // return "Marc"
      .Returns("Marc");
Adding behavior

Behavior can simply be added to mock as code since Pex will explore the mock code as well. For example, let's refine the MNamed implementation to always return the same name:

[PexMock] // specifies to pex that this type is a mock
public class MNamed : INamed
{
    private string name;
    public string GetName()
    {
        if(this.name == null)
            this.name = PexOracle.MethodCall(this)
                              .ChooseNotNull<string>("name");
        return this.name;
    }
}
Wrapping Up

In this post, we've given a brief glimpse at the pex oracles. These are a special breed of mocks specific to Pex.

posted on Saturday, June 02, 2007 2:00:50 PM UTC  #    Comments [0]
 Sunday, May 27, 2007

.Net 2.0 has been out for a while and it seems that 'generics' have not made it into unit test frameworks (that I know of). When I write unit tests for generics, I don't want to have to instantiate them!

For example, if I have an generic interface,

interface IFoo<T> {...}

then I'd really like to write this kind of test and let the test framework figure out an interresting instantiation (i.e. choice of T):

[Test]
public void Test<T>(IFoo<T> foo) { ... }

In the example above, System.Object can be trivially used to instantiate IFoo<T>. Of course, things get more interresting when mixing type argument, method argument and constraints :)

interface IFoo<T>
{
    R Bar<R>(T t)
     where R : IEnumerable<T>
}

In Pex, we've started to look at this problem... stay tuned.

posted on Sunday, May 27, 2007 1:09:00 PM UTC  #    Comments [3]
 Friday, May 25, 2007

MbUnit 2.4 is out, check it out at

http://weblogs.asp.net/astopford/archive/2007/05/24/mbunit-2-4-rtm.aspx

If you are using MbUnit parameterized tests (i.e. RowTest, CombinatorialTest) then migrating to Pex should be a piece of cake :)

posted on Friday, May 25, 2007 10:25:02 PM UTC  #    Comments [2]
 Sunday, April 22, 2007

3 years after being released on CodeProject, it was about time to dust off QuickGraph and give it generics support.

posted on Sunday, April 22, 2007 3:49:21 PM UTC  #    Comments [2]
 Friday, April 20, 2007

With parameterized unit tests, it is not uncommon to generate a large number of exceptions. An basic exception log usually looks like this: a small message and the stack trace.

Whith this kind of output, a lot of work is still left to the user since he has, *for each frame*,  to manually open the source file and move to the line refered by the trace.

Give me the source context!

In the Pex reports we added a couple lines of code to read the source for each frame and display it in the reports (no rocket science). The cool thing is that you now can get a pretty good idea of what happened without leaving the test report. Multiply that by dozens of exceptions and you've won a loooot of time.

Here are some screenshots to illustrate this: an exception was thrown in some arcane method. The error message is not really useful (as usual).

If we expand the source and actually see the code, things become much clearer...

posted on Friday, April 20, 2007 7:12:40 AM UTC  #    Comments [0]
 Wednesday, April 11, 2007

MbUnit supports different flavors of parameterized unit tests: RowTest, CombinatorialTest, etc... If you are already using those features, it would be very easy for you to 'pexify' them:

namespace MyTest
{
     using MbUnit.Framework;
     using Microsoft.Pex.Framework;

     [TestFixture, PexClass]
     public partial class MyTestFixture
{
[RowTest]
[Row("a", "b")]
[PexTest]

public void Test(string a, string b)
{
...
}


}
}

Isn't this nice? :)

Some little notes:

  • 'partial' is helpfull to emit the generated unit test in the same class... but not the same file. Pex also support another mode where partial is not required.
  • the Pex attributes do not 'interfere' with the MbUnit ones. Your unit tests will still run exactly the same with MbUnit.
posted on Wednesday, April 11, 2007 11:47:27 AM UTC  #    Comments [2]
 Sunday, April 08, 2007

The bank sample is a small example that serves as a quickstart for NUnit. In this post, we will revisit the sample and see how existing unit tests can be refactored into Pex parameterized tests.

The original unit test:

Let's start with the initial implementation of the Account class, and the first unit test case from the sample:

namespace bank
{
  public class Account
  {
    private float balance;
    public void Deposit(float amount)
    {
      balance+=amount;
    }

    public void Withdraw(float amount)
    {
      balance-=amount;
    }

    public void TransferFunds(Account destination, float amount)
    {}

    public float Balance
    {
      get{ return balance;}
    }
  }
}
namespace bank
{
  using NUnit.Framework;

  [TestFixture]
  public class AccountTest
  {
    [Test]
    public void TransferFunds()
    {
      Account source = new Account();
      source.Deposit(200.00F);
      Account destination = new Account();
      destination.Deposit(150.00F);

      source.TransferFunds(destination, 100.00F);
      Assert.AreEqual(250.00F, destination.Balance);
      Assert.AreEqual(100.00F, source.Balance);
	
    }
  }
}

Refactoring the unit test and hooking Pex

An easy oportunity to refactor a unit test into parameterized unit tests is to extract constants as parameters, namely 200F as sourceDeposit, 150F as destinationDeposit, etc... We also add the PexClass and PexTest custom attributes to tell Pex that there are some parameterized tests: 

    using Microsoft.Pex.Framework;
    [TestFixture, PexClass]
    public partial class AccountTest
    {
        [PexTest]
        public void TransferFunds(
            float sourceDeposit,
            float destinationDeposit,
            float transfer)
        {
            Account source = new Account();
            source.Deposit(sourceDeposit);
            Account destination = new Account();
            destination.Deposit(destinationDeposit);
            source.TransferFunds(destination, transfer);
            Assert.AreEqual(destinationDeposit + transfer, destination.Balance);
            Assert.AreEqual(sourceDeposit - transfer, source.Balance);
        }
    }

Iteration 1: amount should not be negative,

The first (and unique) test case that Pex generates uses float.MinValue for all values, which leads to some interresting floating point issues.

this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);

...

expected: <-Infinity>

but was: <-3.40282347E+38>

at Assert.AreEqual(destinationDeposit + transfer, destination.Balance);

This little test reminds us that floats can have some weird values (infinity, minvalue, etc...).

Iteration 2: amount should not be too big,

It's time to add parameter checking to prevent negative transfer amounts. The ValidateAmount method is added in each transaction method:

        private void ValidateAmount(float amount)
        {
            if (amount < 0)
                throw new ArgumentOutOfRangeException("amount");
        }

Pex now generates 4 test cases. 3 of them pass negative values as amount to cover the argument validation code:

        [Test()]
        [ExpectedException(typeof(System.ArgumentOutOfRangeException)), ...]
        public void TransferFunds_Single_Single_Single_70408_082415_0_01()
        {
            this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);
}
...
            this.TransferFunds(float.MaxValue, float.MinValue, float.MinValue);
            this.TransferFunds(float.MaxValue, float.MaxValue, float.MinValue);

The 4-th test finds another overflow by passing all float.MaxValue.

Iteration 3: NaN handling

We add a MaximumAmount field to bound the amount of money that can be transfered.


private float maximumAmount = 1000.00F;
public float MaximumAmount
{
 get{ return maximumAmount;}
}

private void ValidateAmount(float amount)
{
if (amount <= 0)
     throw new ArgumentOutOfRangeException("amount");
    if (amount > this.MaximumAmount)
      throw new ArgumentOutOfRangeException("amount");
}

We run Pex. The test still does not fail and to our surprise it generates the following passing test: 

      this.TransferFunds(float.NaN, float.NaN, float.NaN);

Oops, NaN is a very special number. Did you know that (float.NaN == float.Nan) == false? In our case, float.NaN passes the parameter validation and assertions!

Iteration 4: The test fails!

Again, we beef up the 'amount' validation to rule out NaN:

        private void ValidateAmount(float amount)
        {
            if (float.IsNaN(amount))
                throw new ArgumentOutOfRangeException("amount");
            if (amount <= 0)
                throw new ArgumentOutOfRangeException("amount");
            if (amount > this.MaximumAmount)
                throw new ArgumentOutOfRangeException("amount");
        }

We run Pex again and, at last, we find an input to fail the test:

            this.TransferFunds(float.Epsilon, float.Epsilon, float.Epsilon);
        ...
        [test] TransferFunds_Single_Single_Single_70408_113456_0_05, AssertionException:
 expected: <2.80259693E-45>
  but was: <1.40129846E-45>

Iteration 5: Implementing TransferFunds

Now that we found an input that fails the test, we can actually add some code to TransferFunds (as in the QuickStart):

        public void TransferFunds(Account destination, float amount)
        {
            this.ValidateAmount(amount);
            destination.Deposit(amount);
            Withdraw(amount);
        }

So we run Pex again and all the tests passes. Next time, we'll take look at the implementation of the 'MinimumDeposit' feature.

posted on Sunday, April 08, 2007 2:09:48 PM UTC  #    Comments [2]
 Sunday, April 01, 2007

This is the first post about Pex and how to use it. Since Pex is a fairly large project, I'll probably stretch the content of a large number of entries. Stay tunned...

Pex gives you Parameterized Unit Tests

Data-driven tests are not something new. They exist under different forms such as MbUnit's RowTest or CombinatorialTest, VSTS's DataSource, etc... So what's so special about Pex? The major difference is that Pex finds the parameter inputs for you (and generates a unit test out of it).

Whereas the user had to (smartly) guess a set of input for the data-driven tests, Pex tries to compute the relevant inputs to those tests automatically (and may also suggest fixes).

Note: The way Pex finds the input will be covered in more details later. In a couple words, Pex performs a systematic white box analysis of the program behavior. It tries to generate a minimal test suite with maximum coverage.

Parameterized Unit Tests, what does it feel like?

Parameterized unit tests are methods with parameters. There's nothing magic about that. Pex provides a set of custom attributes so that you can author them side-by-side with classic unit tests:

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    // parameterized unit test
    // account money should never be negative, for any 'money', 'withdraw'
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
}

When Pex finds interresting data to feed the parameterized unit test, it generates a unit test method that calls the parameterized unit test. This also means that once the unit test has been generated, you do not need Pex anymore to run the repro.

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
    ...
    [TestMethod, GeneratedBy("Pex", "1.0.0.0")]
    public void TransferFunds_12345() { this.TransferFunds(12, 13); }

}

Why do I need parameterized unit tests anyway?

A parameterized unit tests generally captures more program behaviors than a single unit test which is like a micro-scenario. This will become more apparent when we start looking at some examples. 

If you were already using [RowTest] or [DataSource] as part of your testing, then you will definitely like Pex.

posted on Monday, April 02, 2007 5:52:37 AM UTC  #    Comments [2]
 Monday, March 19, 2007

Reflector.CodeMetrics swallowed the treemap coolaid ...

posted on Monday, March 19, 2007 11:01:41 AM UTC  #    Comments [0]
 Tuesday, March 13, 2007

The Pex screencast was a bit mysterious without sound and comments. I've added a 'storyboard' to help you understand it:

   http://research.microsoft.com/pex/screencast.aspx


 

posted on Tuesday, March 13, 2007 7:31:29 AM UTC  #    Comments [1]
 Thursday, March 08, 2007

Pex

I'm thrilled to present the project I joined last October: 'Pex' (for Program EXploration). Pex is a powerfull plugin for unit test frameworks that let the user write parameterized unit tests**. Pex does the hard work of computing the relevant values for those parameters, and serializing them as classic unit tests.

Here's a short screencast where we test and implement a string chunker. In the screencast, we use a parameterized unit test to express that for *any* string input and *any* chunk length, the concatenation of the chunks should be equal to the original sting.

http://research.microsoft.com/pex/screencast.aspx

More info on Pex is available at http://research.microsoft.com/pex/.

** It's actually much more than that... but let's keep that for later :)

posted on Friday, March 09, 2007 1:58:36 AM UTC  #    Comments [5]
 Saturday, February 24, 2007