# Sunday, April 22, 2007

3 years after being released on CodeProject, it was about time to dust off QuickGraph and give it generics support.

posted on Sunday, April 22, 2007 1:49:21 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Thursday, April 19, 2007

With parameterized unit tests, it is not uncommon to generate a large number of exceptions. An basic exception log usually looks like this: a small message and the stack trace.

Whith this kind of output, a lot of work is still left to the user since he has, *for each frame*,  to manually open the source file and move to the line refered by the trace.

Give me the source context!

In the Pex reports we added a couple lines of code to read the source for each frame and display it in the reports (no rocket science). The cool thing is that you now can get a pretty good idea of what happened without leaving the test report. Multiply that by dozens of exceptions and you've won a loooot of time.

Here are some screenshots to illustrate this: an exception was thrown in some arcane method. The error message is not really useful (as usual).

If we expand the source and actually see the code, things become much clearer...

posted on Thursday, April 19, 2007 5:12:40 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, April 10, 2007

MbUnit supports different flavors of parameterized unit tests: RowTest, CombinatorialTest, etc... If you are already using those features, it would be very easy for you to 'pexify' them:

namespace MyTest
{
     using MbUnit.Framework;
     using Microsoft.Pex.Framework;

     [TestFixture, PexClass]
     public partial class MyTestFixture
{
[RowTest]
[Row("a", "b")]
[PexTest]

public void Test(string a, string b)
{
...
}


}
}

Isn't this nice? :)

Some little notes:

  • 'partial' is helpfull to emit the generated unit test in the same class... but not the same file. Pex also support another mode where partial is not required.
  • the Pex attributes do not 'interfere' with the MbUnit ones. Your unit tests will still run exactly the same with MbUnit.
posted on Tuesday, April 10, 2007 9:47:27 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 08, 2007

The bank sample is a small example that serves as a quickstart for NUnit. In this post, we will revisit the sample and see how existing unit tests can be refactored into Pex parameterized tests.

The original unit test:

Let's start with the initial implementation of the Account class, and the first unit test case from the sample:

namespace bank
{
  public class Account
  {
    private float balance;
    public void Deposit(float amount)
    {
      balance+=amount;
    }

    public void Withdraw(float amount)
    {
      balance-=amount;
    }

    public void TransferFunds(Account destination, float amount)
    {}

    public float Balance
    {
      get{ return balance;}
    }
  }
}
namespace bank
{
  using NUnit.Framework;

  [TestFixture]
  public class AccountTest
  {
    [Test]
    public void TransferFunds()
    {
      Account source = new Account();
      source.Deposit(200.00F);
      Account destination = new Account();
      destination.Deposit(150.00F);

      source.TransferFunds(destination, 100.00F);
      Assert.AreEqual(250.00F, destination.Balance);
      Assert.AreEqual(100.00F, source.Balance);
	
    }
  }
}

Refactoring the unit test and hooking Pex

An easy oportunity to refactor a unit test into parameterized unit tests is to extract constants as parameters, namely 200F as sourceDeposit, 150F as destinationDeposit, etc... We also add the PexClass and PexTest custom attributes to tell Pex that there are some parameterized tests: 

    using Microsoft.Pex.Framework;
    [TestFixture, PexClass]
    public partial class AccountTest
    {
        [PexTest]
        public void TransferFunds(
            float sourceDeposit,
            float destinationDeposit,
            float transfer)
        {
            Account source = new Account();
            source.Deposit(sourceDeposit);
            Account destination = new Account();
            destination.Deposit(destinationDeposit);
            source.TransferFunds(destination, transfer);
            Assert.AreEqual(destinationDeposit + transfer, destination.Balance);
            Assert.AreEqual(sourceDeposit - transfer, source.Balance);
        }
    }

Iteration 1: amount should not be negative,

The first (and unique) test case that Pex generates uses float.MinValue for all values, which leads to some interresting floating point issues.

this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);

...

expected: <-Infinity>

but was: <-3.40282347E+38>

at Assert.AreEqual(destinationDeposit + transfer, destination.Balance);

This little test reminds us that floats can have some weird values (infinity, minvalue, etc...).

Iteration 2: amount should not be too big,

It's time to add parameter checking to prevent negative transfer amounts. The ValidateAmount method is added in each transaction method:

        private void ValidateAmount(float amount)
        {
            if (amount < 0)
                throw new ArgumentOutOfRangeException("amount");
        }

Pex now generates 4 test cases. 3 of them pass negative values as amount to cover the argument validation code:

        [Test()]
        [ExpectedException(typeof(System.ArgumentOutOfRangeException)), ...]
        public void TransferFunds_Single_Single_Single_70408_082415_0_01()
        {
            this.TransferFunds(float.MinValue, float.MinValue, float.MinValue);
}
...
            this.TransferFunds(float.MaxValue, float.MinValue, float.MinValue);
            this.TransferFunds(float.MaxValue, float.MaxValue, float.MinValue);

The 4-th test finds another overflow by passing all float.MaxValue.

Iteration 3: NaN handling

We add a MaximumAmount field to bound the amount of money that can be transfered.


private float maximumAmount = 1000.00F;
public float MaximumAmount
{
 get{ return maximumAmount;}
}

private void ValidateAmount(float amount)
{
if (amount <= 0)
     throw new ArgumentOutOfRangeException("amount");
    if (amount > this.MaximumAmount)
      throw new ArgumentOutOfRangeException("amount");
}

We run Pex. The test still does not fail and to our surprise it generates the following passing test: 

      this.TransferFunds(float.NaN, float.NaN, float.NaN);

Oops, NaN is a very special number. Did you know that (float.NaN == float.Nan) == false? In our case, float.NaN passes the parameter validation and assertions!

Iteration 4: The test fails!

Again, we beef up the 'amount' validation to rule out NaN:

        private void ValidateAmount(float amount)
        {
            if (float.IsNaN(amount))
                throw new ArgumentOutOfRangeException("amount");
            if (amount <= 0)
                throw new ArgumentOutOfRangeException("amount");
            if (amount > this.MaximumAmount)
                throw new ArgumentOutOfRangeException("amount");
        }

We run Pex again and, at last, we find an input to fail the test:

            this.TransferFunds(float.Epsilon, float.Epsilon, float.Epsilon);
        ...
        [test] TransferFunds_Single_Single_Single_70408_113456_0_05, AssertionException:
 expected: <2.80259693E-45>
  but was: <1.40129846E-45>

Iteration 5: Implementing TransferFunds

Now that we found an input that fails the test, we can actually add some code to TransferFunds (as in the QuickStart):

        public void TransferFunds(Account destination, float amount)
        {
            this.ValidateAmount(amount);
            destination.Deposit(amount);
            Withdraw(amount);
        }

So we run Pex again and all the tests passes. Next time, we'll take look at the implementation of the 'MinimumDeposit' feature.

posted on Sunday, April 08, 2007 12:09:48 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 01, 2007

This is the first post about Pex and how to use it. Since Pex is a fairly large project, I'll probably stretch the content of a large number of entries. Stay tunned...

Pex gives you Parameterized Unit Tests

Data-driven tests are not something new. They exist under different forms such as MbUnit's RowTest or CombinatorialTest, VSTS's DataSource, etc... So what's so special about Pex? The major difference is that Pex finds the parameter inputs for you (and generates a unit test out of it).

Whereas the user had to (smartly) guess a set of input for the data-driven tests, Pex tries to compute the relevant inputs to those tests automatically (and may also suggest fixes).

Note: The way Pex finds the input will be covered in more details later. In a couple words, Pex performs a systematic white box analysis of the program behavior. It tries to generate a minimal test suite with maximum coverage.

Parameterized Unit Tests, what does it feel like?

Parameterized unit tests are methods with parameters. There's nothing magic about that. Pex provides a set of custom attributes so that you can author them side-by-side with classic unit tests:

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    // parameterized unit test
    // account money should never be negative, for any 'money', 'withdraw'
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
}

When Pex finds interresting data to feed the parameterized unit test, it generates a unit test method that calls the parameterized unit test. This also means that once the unit test has been generated, you do not need Pex anymore to run the repro.

[TestClass, PexClass] // VSTS fixture containing parameterized tests
public class AccountTest
{
    [PexTest]
    public void TransferFunds(int money, int withdraw) {...}
    ...
    [TestMethod, GeneratedBy("Pex", "1.0.0.0")]
    public void TransferFunds_12345() { this.TransferFunds(12, 13); }

}

Why do I need parameterized unit tests anyway?

A parameterized unit tests generally captures more program behaviors than a single unit test which is like a micro-scenario. This will become more apparent when we start looking at some examples. 

If you were already using [RowTest] or [DataSource] as part of your testing, then you will definitely like Pex.

posted on Sunday, April 01, 2007 3:52:37 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]