# Saturday, June 30, 2007

In Pex, we added the possibility to specify the type under test of a given fixture:

public class Account {...}


[TestFixture, PexClass(typeof(Account))]
public class AccountTest {...}  

That's nice but why would it be useful... Beyond the fact that it clearly expresses the 'target' of the fixture, this kind of information can be leverage by tools like Pex.

For example, since we know that Account is the type under test, we can tune Pex to prioritize the exploration of the Account type.

Another interesting side effect is the targeted code coverage data. Instead of getting coverage information over the entire assembly, we can directly provide coverage over the type under test: the AccountTest covered xx% of Account.

Still toying around the concept, one can add a special filtering mode to the command line to execute all tests that target 'Account':

pex.exe /type-under-test:Account Bank.Tests.dll
posted on Saturday, June 30, 2007 12:23:24 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, June 02, 2007

A nice side effect of the Pex infrastructure is that it computes coverage data while exploring a parameterized test. What does this mean to the user? Well, what about this specifying that a parameterized test should reach 100% coverage**:

[PexTest]
[PexExpectedCoverage(100)] // expect 100% coverage
public void UberTest(int i)
{ ... }
** What kind of coverage are we talking about?

Pex internally tracks implicit branch coverage (which can never be covered :)). So we usually refer to basic block coverage. By default, the coverage is reported for the code that was run by the parameterized test. This is a very important difference with usual coverage tools, which give numbers for the entire assembly.

posted on Saturday, June 02, 2007 10:24:56 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, May 26, 2007

.Net 2.0 has been out for a while and it seems that 'generics' have not made it into unit test frameworks (that I know of). When I write unit tests for generics, I don't want to have to instantiate them!

For example, if I have an generic interface,

interface IFoo<T> {...}

then I'd really like to write this kind of test and let the test framework figure out an interresting instantiation (i.e. choice of T):

[Test]
public void Test<T>(IFoo<T> foo) { ... }

In the example above, System.Object can be trivially used to instantiate IFoo<T>. Of course, things get more interresting when mixing type argument, method argument and constraints :)

interface IFoo<T>
{
    R Bar<R>(T t)
     where R : IEnumerable<T>
}

In Pex, we've started to look at this problem... stay tuned.

posted on Saturday, May 26, 2007 11:09:00 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [3]
# Friday, May 25, 2007

MbUnit 2.4 is out, check it out at

http://weblogs.asp.net/astopford/archive/2007/05/24/mbunit-2-4-rtm.aspx

If you are using MbUnit parameterized tests (i.e. RowTest, CombinatorialTest) then migrating to Pex should be a piece of cake :)

posted on Friday, May 25, 2007 8:25:02 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 22, 2007

3 years after being released on CodeProject, it was about time to dust off QuickGraph and give it generics support.

posted on Sunday, April 22, 2007 1:49:21 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Thursday, April 19, 2007

With parameterized unit tests, it is not uncommon to generate a large number of exceptions. An basic exception log usually looks like this: a small message and the stack trace.

Whith this kind of output, a lot of work is still left to the user since he has, *for each frame*,  to manually open the source file and move to the line refered by the trace.

Give me the source context!

In the Pex reports we added a couple lines of code to read the source for each frame and display it in the reports (no rocket science). The cool thing is that you now can get a pretty good idea of what happened without leaving the test report. Multiply that by dozens of exceptions and you've won a loooot of time.

Here are some screenshots to illustrate this: an exception was thrown in some arcane method. The error message is not really useful (as usual).

If we expand the source and actually see the code, things become much clearer...

posted on Thursday, April 19, 2007 5:12:40 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, April 10, 2007

MbUnit supports different flavors of parameterized unit tests: RowTest, CombinatorialTest, etc... If you are already using those features, it would be very easy for you to 'pexify' them:

namespace MyTest
{
     using MbUnit.Framework;
     using Microsoft.Pex.Framework;

     [TestFixture, PexClass]
     public partial class MyTestFixture
{
[RowTest]
[Row("a", "b")]
[PexTest]

public void Test(string a, string b)
{
...
}


}
}

Isn't this nice? :)

Some little notes:

  • 'partial' is helpfull to emit the generated unit test in the same class... but not the same file. Pex also support another mode where partial is not required.
  • the Pex attributes do not 'interfere' with the MbUnit ones. Your unit tests will still run exactly the same with MbUnit.
posted on Tuesday, April 10, 2007 9:47:27 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 08, 2007

The bank sample is a small example that serves as a quickstart for NUnit. In this post, we will revisit the sample and see how existing unit tests can be refactored into Pex parameterized tests.

The original unit test:

Let's start with the initial implementation of the Account class, and the first unit test case from the sample:

namespace bank
{
  public class Account
  {
    private float balance;
    public void Deposit(float amount)
    {
      balance+=amount;
    }

    public void Withdraw(float amount)
    {
      balance-=amount;
    }

    public void TransferFunds(Account destination, float amount)
    {}

    public float Balance
    {
      get{ return balance;}
    }
  }
}
namespace bank
{
  using NUnit.Framework;

  [TestFixture]
  public class AccountTest
  {
    [Test]
    public void TransferFunds()
    {
      Account source = new Account();
      source.Deposit(200.00F);
      Account destination = new Account();
      destination.Deposit(150.00F);

      source.TransferFunds(destination, 100.00F);
      Assert.AreEqual(250.00F, destination.Balance);
      Assert.AreEqual(100.00F, source.Balance);
	
    }
  }
}

Refactoring the unit test and hooking Pex

An easy oportunity to refactor a unit test into parameterized unit tests is to extract constants as parameters, namely 200F as sourceDeposit, 150F as destinationDeposit, etc... We also add the PexClass and PexTest custom attributes to tell Pex that there are some parameterized tests: 

    using Microsoft.Pex.Framework;
    [TestFixture, PexClass]
    public partial class AccountTest
    {
        [PexTest]
        public void TransferFunds(
            float sourceDeposit,
            float destinationDeposit,
            float transfer)
        {
            Account source = new Account();
            source.Deposit(sourceDeposit);
            Account destination = new Account();
            destination.Deposit(destinationDeposit);
            source.TransferFunds(destination, transfer);
            Assert.AreEqual(destinationDeposit + transfer, destination.Balance);
            Assert.AreEqual(sourceDeposit - transfer, source.Balance);
        }
    }

Iteration 1: amount should not be negative,

The first (and unique) test case that Pex generates uses float.MinValue for all values, which leads to some interresting floating point issues.

this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);

...

expected: <-Infinity>

but was: <-3.40282347E+38>

at Assert.AreEqual(destinationDeposit + transfer, destination.Balance);

This little test reminds us that floats can have some weird values (infinity, minvalue, etc...).

Iteration 2: amount should not be too big,

It's time to add parameter checking to prevent negative transfer amounts. The ValidateAmount method is added in each transaction method:

        private void ValidateAmount(float amount)
        {
            if (amount < 0)
                throw new ArgumentOutOfRangeException("amount");
        }

Pex now generates 4 test cases. 3 of them pass negative values as amount to cover the argument validation code:

        [Test()]
        [ExpectedException(typeof(System.ArgumentOutOfRangeException)), ...]
        public void TransferFunds_Single_Single_Single_70408_082415_0_01()
        {
            this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);
}
...
            this.TransferFunds(float.MaxValue, float.MinValue, float.MinValue);
            this.TransferFunds(float.MaxValue, float.MaxValue, float.MinValue);

The 4-th test finds another overflow by passing all float.MaxValue.

Iteration 3: NaN handling

We add a MaximumAmount field to bound the amount of money that can be transfered.


private float maximumAmount = 1000.00F;
public float MaximumAmount
{
 get{ return maximumAmount;}
}

private void ValidateAmount(float amount)
{
if (amount <= 0)
     throw new ArgumentOutOfRangeException("amount");
    if (amount > this.MaximumAmount)
      throw new ArgumentOutOfRangeException("amount");
}

We run Pex. The test still does not fail and to our surprise it generates the following passing test: 

      this.TransferFunds(float.NaN, float.NaN, float.NaN);

Oops, NaN is a very special number. Did you know that (float.NaN == float.Nan) == false? In our case, float.NaN passes the parameter validation and assertions!

Iteration 4: The test fails!

Again, we beef up the 'amount' validation to rule out NaN:

        private void ValidateAmount(float amount)
        {
            if (float.IsNaN(amount))
                throw new ArgumentOutOfRangeException("amount");
            if (amount <= 0)
                throw new ArgumentOutOfRangeException("amount");
            if (amount > this.MaximumAmount)
                throw new ArgumentOutOfRangeException("amount");
        }

We run Pex again and, at last, we find an input to fail the test:

            this.TransferFunds(float.Epsilon, float.Epsilon, float.Epsilon);
        ...
        [test] TransferFunds_Single_Single_Single_70408_113456_0_05, AssertionException:
 expected: <2.80259693E-45>
  but was: <1.40129846E-45>

Iteration 5: Implementing TransferFunds

Now that we found an input that fails the test, we can actually add some code to TransferFunds (as in the QuickStart):

        public void TransferFunds(Account destination, float amount)
        {
            this.ValidateAmount(amount);
            destination.Deposit(amount);
            Withdraw(amount);
        }

So we run Pex again and all the tests passes. Next time, we'll take look at the implementation of the 'MinimumDeposit' feature.

posted on Sunday, April 08, 2007 12:09:48 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 01, 2007

This is the first post about Pex and how to use it. Since Pex is a fairly large project, I'll probably stretch the content of a large number of entries. Stay tunned...

Pex gives you Parameterized Unit Tests

Data-driven tests are not something new. They exist under different forms such as MbUnit's RowTest or CombinatorialTest, VSTS's DataSource, etc... So what's so special about Pex? The major difference is that Pex finds the parameter inputs for you (and generates a unit test out of it).

Whereas the user had to (smartly) guess a set of input for the data-driven tests, Pex tries to compute the relevant inputs to those tests automatically (and may also suggest fixes).

Note: The way Pex finds the input will be covered in more details later. In a couple words, Pex performs a systematic white box analysis of the program behavior. It tries to generate a minimal test suite with maximum coverage.

Parameterized Unit Tests, what does it feel like?

Parameterized unit tests are methods with parameters. There's nothing magic about that. Pex provides a set of custom attributes so that you can author them side-by-side with classic unit tests:

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    // parameterized unit test
    // account money should never be negative, for any 'money', 'withdraw'
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
}

When Pex finds interresting data to feed the parameterized unit test, it generates a unit test method that calls the parameterized unit test. This also means that once the unit test has been generated, you do not need Pex anymore to run the repro.

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
    ...
    [TestMethod, GeneratedBy("Pex", "1.0.0.0")]
    public void TransferFunds_12345() { this.TransferFunds(12, 13); }

}

Why do I need parameterized unit tests anyway?

A parameterized unit tests generally captures more program behaviors than a single unit test which is like a micro-scenario. This will become more apparent when we start looking at some examples. 

If you were already using [RowTest] or [DataSource] as part of your testing, then you will definitely like Pex.

posted on Sunday, April 01, 2007 3:52:37 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, March 18, 2007

Reflector.CodeMetrics swallowed the treemap coolaid ...

posted on Sunday, March 18, 2007 9:01:41 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]