# Sunday, April 08, 2007

The bank sample is a small example that serves as a quickstart for NUnit. In this post, we will revisit the sample and see how existing unit tests can be refactored into Pex parameterized tests.

The original unit test:

Let's start with the initial implementation of the Account class, and the first unit test case from the sample:

namespace bank
{
  public class Account
  {
    private float balance;
    public void Deposit(float amount)
    {
      balance+=amount;
    }

    public void Withdraw(float amount)
    {
      balance-=amount;
    }

    public void TransferFunds(Account destination, float amount)
    {}

    public float Balance
    {
      get{ return balance;}
    }
  }
}
namespace bank
{
  using NUnit.Framework;

  [TestFixture]
  public class AccountTest
  {
    [Test]
    public void TransferFunds()
    {
      Account source = new Account();
      source.Deposit(200.00F);
      Account destination = new Account();
      destination.Deposit(150.00F);

      source.TransferFunds(destination, 100.00F);
      Assert.AreEqual(250.00F, destination.Balance);
      Assert.AreEqual(100.00F, source.Balance);
	
    }
  }
}

Refactoring the unit test and hooking Pex

An easy oportunity to refactor a unit test into parameterized unit tests is to extract constants as parameters, namely 200F as sourceDeposit, 150F as destinationDeposit, etc... We also add the PexClass and PexTest custom attributes to tell Pex that there are some parameterized tests: 

    using Microsoft.Pex.Framework;
    [TestFixture, PexClass]
    public partial class AccountTest
    {
        [PexTest]
        public void TransferFunds(
            float sourceDeposit,
            float destinationDeposit,
            float transfer)
        {
            Account source = new Account();
            source.Deposit(sourceDeposit);
            Account destination = new Account();
            destination.Deposit(destinationDeposit);
            source.TransferFunds(destination, transfer);
            Assert.AreEqual(destinationDeposit + transfer, destination.Balance);
            Assert.AreEqual(sourceDeposit - transfer, source.Balance);
        }
    }

Iteration 1: amount should not be negative,

The first (and unique) test case that Pex generates uses float.MinValue for all values, which leads to some interresting floating point issues.

this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);

...

expected: <-Infinity>

but was: <-3.40282347E+38>

at Assert.AreEqual(destinationDeposit + transfer, destination.Balance);

This little test reminds us that floats can have some weird values (infinity, minvalue, etc...).

Iteration 2: amount should not be too big,

It's time to add parameter checking to prevent negative transfer amounts. The ValidateAmount method is added in each transaction method:

        private void ValidateAmount(float amount)
        {
            if (amount < 0)
                throw new ArgumentOutOfRangeException("amount");
        }

Pex now generates 4 test cases. 3 of them pass negative values as amount to cover the argument validation code:

        [Test()]
        [ExpectedException(typeof(System.ArgumentOutOfRangeException)), ...]
        public void TransferFunds_Single_Single_Single_70408_082415_0_01()
        {
            this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);
}
...
            this.TransferFunds(float.MaxValue, float.MinValue, float.MinValue);
            this.TransferFunds(float.MaxValue, float.MaxValue, float.MinValue);

The 4-th test finds another overflow by passing all float.MaxValue.

Iteration 3: NaN handling

We add a MaximumAmount field to bound the amount of money that can be transfered.


private float maximumAmount = 1000.00F;
public float MaximumAmount
{
 get{ return maximumAmount;}
}

private void ValidateAmount(float amount)
{
if (amount <= 0)
     throw new ArgumentOutOfRangeException("amount");
    if (amount > this.MaximumAmount)
      throw new ArgumentOutOfRangeException("amount");
}

We run Pex. The test still does not fail and to our surprise it generates the following passing test: 

      this.TransferFunds(float.NaN, float.NaN, float.NaN);

Oops, NaN is a very special number. Did you know that (float.NaN == float.Nan) == false? In our case, float.NaN passes the parameter validation and assertions!

Iteration 4: The test fails!

Again, we beef up the 'amount' validation to rule out NaN:

        private void ValidateAmount(float amount)
        {
            if (float.IsNaN(amount))
                throw new ArgumentOutOfRangeException("amount");
            if (amount <= 0)
                throw new ArgumentOutOfRangeException("amount");
            if (amount > this.MaximumAmount)
                throw new ArgumentOutOfRangeException("amount");
        }

We run Pex again and, at last, we find an input to fail the test:

            this.TransferFunds(float.Epsilon, float.Epsilon, float.Epsilon);
        ...
        [test] TransferFunds_Single_Single_Single_70408_113456_0_05, AssertionException:
 expected: <2.80259693E-45>
  but was: <1.40129846E-45>

Iteration 5: Implementing TransferFunds

Now that we found an input that fails the test, we can actually add some code to TransferFunds (as in the QuickStart):

        public void TransferFunds(Account destination, float amount)
        {
            this.ValidateAmount(amount);
            destination.Deposit(amount);
            Withdraw(amount);
        }

So we run Pex again and all the tests passes. Next time, we'll take look at the implementation of the 'MinimumDeposit' feature.

posted on Sunday, April 08, 2007 12:09:48 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 01, 2007

This is the first post about Pex and how to use it. Since Pex is a fairly large project, I'll probably stretch the content of a large number of entries. Stay tunned...

Pex gives you Parameterized Unit Tests

Data-driven tests are not something new. They exist under different forms such as MbUnit's RowTest or CombinatorialTest, VSTS's DataSource, etc... So what's so special about Pex? The major difference is that Pex finds the parameter inputs for you (and generates a unit test out of it).

Whereas the user had to (smartly) guess a set of input for the data-driven tests, Pex tries to compute the relevant inputs to those tests automatically (and may also suggest fixes).

Note: The way Pex finds the input will be covered in more details later. In a couple words, Pex performs a systematic white box analysis of the program behavior. It tries to generate a minimal test suite with maximum coverage.

Parameterized Unit Tests, what does it feel like?

Parameterized unit tests are methods with parameters. There's nothing magic about that. Pex provides a set of custom attributes so that you can author them side-by-side with classic unit tests:

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    // parameterized unit test
    // account money should never be negative, for any 'money', 'withdraw'
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
}

When Pex finds interresting data to feed the parameterized unit test, it generates a unit test method that calls the parameterized unit test. This also means that once the unit test has been generated, you do not need Pex anymore to run the repro.

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
    ...
    [TestMethod, GeneratedBy("Pex", "1.0.0.0")]
    public void TransferFunds_12345() { this.TransferFunds(12, 13); }

}

Why do I need parameterized unit tests anyway?

A parameterized unit tests generally captures more program behaviors than a single unit test which is like a micro-scenario. This will become more apparent when we start looking at some examples. 

If you were already using [RowTest] or [DataSource] as part of your testing, then you will definitely like Pex.

posted on Sunday, April 01, 2007 3:52:37 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, March 18, 2007

Reflector.CodeMetrics swallowed the treemap coolaid ...

posted on Sunday, March 18, 2007 9:01:41 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Monday, March 12, 2007

The Pex screencast was a bit mysterious without sound and comments. I've added a 'storyboard' to help you understand it:

   http://research.microsoft.com/pex/screencast.aspx


 

posted on Monday, March 12, 2007 5:31:29 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Thursday, March 08, 2007

Pex

I'm thrilled to present the project I joined last October: 'Pex' (for Program EXploration). Pex is a powerfull plugin for unit test frameworks that let the user write parameterized unit tests**. Pex does the hard work of computing the relevant values for those parameters, and serializing them as classic unit tests.

Here's a short screencast where we test and implement a string chunker. In the screencast, we use a parameterized unit test to express that for *any* string input and *any* chunk length, the concatenation of the chunks should be equal to the original sting.

http://research.microsoft.com/pex/screencast.aspx

More info on Pex is available at http://research.microsoft.com/pex/.

** It's actually much more than that... but let's keep that for later :)

posted on Thursday, March 08, 2007 9:58:36 AM (Pacific Standard Time, UTC-08:00)  #    Comments [5]
# Saturday, February 24, 2007
posted on Saturday, February 24, 2007 1:09:02 AM (Pacific Standard Time, UTC-08:00)  #    Comments [2]
# Monday, February 19, 2007

All my Reflector addins have a new home:

http://www.codeplex.com/reflectoraddins

Please use the codeplex issue tracking to log bugs.

 

posted on Monday, February 19, 2007 8:04:29 PM (Pacific Standard Time, UTC-08:00)  #    Comments [2]
# Saturday, November 18, 2006

Getting this dusty blog back to life....

posted on Saturday, November 18, 2006 6:08:28 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Monday, September 11, 2006

GLEE, a graph layout engine developed in Microsoft Research. It is now available for downloading at http://research.microsoft.com/research/downloads/download.aspx?FUID=c927728f-8872-4826-80ee-ecb842d10371.

 

posted on Monday, September 11, 2006 5:13:07 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Saturday, September 09, 2006

After 2 years in the CLR, I'm moving job (and building) to Microsoft Research. I will be working on Parametrized Unit Testing.

 

posted on Saturday, September 09, 2006 11:21:39 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [3]