# Thursday, July 03, 2008

Brian Keller dropped by our offices to record a movie on Pex. A good old white board session to explain how pex works:



posted on Thursday, July 03, 2008 11:21:54 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Monday, June 23, 2008

We've added a little filter to Pex that flags potential testability issues in your source code (at least in the context of unit testing). The idea is simple: if your code logic branches over API that depends on the environment (file system, UI, network, time, etc...), you're not doing 'pure' unit testing anymore and it could be flagged as a testability issue.

Why do we do this anyway?

The problem is that Pex is very sensitive to those issues: it cannot control the environment. Let's take a look at a simple example where we write a custom stream class with a testability issue in one of the constructor:


The constructor checks that the file exists. In order to pass that point in the program, you would actually need to create or find an actual file in the file system. You're talking to the file system, this is already integration testing.

Parameterized drama

Let's be optimistic and write a parameterized unit test that reads a file:


Pex comes back with a single test, passing null as fileName (i.e. default(string)). Since Pex does not know how the file system works, it cannot create a valid file name, and that's where the story ends.


Help is on the way

To help you diagnose these issues, we've added specialized logging flags. Pex shows hints in the issue notification area:


Clicking on that event will give you additional important data: which API was called and the stacktrace at the time of the call :)


posted on Monday, June 23, 2008 11:08:33 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Wednesday, June 04, 2008

Ever wonder what was happening inside the ResourceReader (where do those resources come from anyway!).... Check out Nikolai's post on using Pex to test this (complicated) class...

posted on Wednesday, June 04, 2008 7:17:06 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, May 24, 2008

If you're interrested on reading more about Pex, here are some online document that should get you satisfied:

The tutorial contains hands-on labs and exercises to understand how Pex generates new test cases. Highly recommend if you're interrested on Pex.

Happy reading!

posted on Saturday, May 24, 2008 9:00:01 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Thursday, May 22, 2008

Nikolai broke the news; we've just released Pex 0.5... get it while it's hot!

We're eager to hear some feedback, so don't hesitate to tell us what you think about it.

posted on Thursday, May 22, 2008 9:30:00 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [11]
# Thursday, May 08, 2008
posted on Thursday, May 08, 2008 8:40:27 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Wednesday, February 27, 2008

This is one problem for which we don't have an elegant solution yet:

what is the best way to craft a name for a generated test?

Let's see an example; given the following parameterized unit test,

[PexMethod] void Test(int i) {
    if (i == 123) throw ArgumentException();

Pex would generate 2 tests: i = 0, i = 123. So it seems doable to infer test names such as

[TestMethod] void Test0() { this.Test(0); }
[TestMethod, EE(typeof(ArgumentException))]
void Test123ThrowsArgumentException() { this.Test(123); }

So what's so difficult about it? Well, most PUT's aren't that simple and as the size of the generated parameter increases, the methods might increase as well (strings getting bigger). Here's a list of potential problems:

  • the method should stay relatively small (less than 80 chars),
  • the parameters might be objects or classes, which do look well in a string format,
  • generated strings might be huge and contain weird unicode characters,
  • the tests might involve mock choices which again do not render well to strings,
  • the more parameters the more cryptic things become

Our approach: Timestamps

To the light of all those problems, we've taken the shortcut route in Pex by simply using combination of the parameterized unit test method signature and the timestamp when the test is generated:

[TestMethod] void TestInt32_20080224_124301_35() { ... }

This is ugly, how can I change that?

We've added an extensibility point to support custom test naming scheme. After registering your 'namer', you will get opportunity to craft a test name following your favorite code standard. You will have the generated test, output, exception, etc... at your disposition to make an intelligent choice there.

Off course, you're also welcome to drop a comment on this post to suggest a better scheme.

posted on Wednesday, February 27, 2008 12:18:46 AM (Pacific Standard Time, UTC-08:00)  #    Comments [2]
# Saturday, February 23, 2008

So how do you write negative test cases with Pex? Here's a nice solution I was working on. I'm wondering

Traditional ExpectedExceptions style

Let's test the constructor a type Foo that takes a reference argument. We expect to throw a ArgumentNullException when null is passed. Therefore, we could write

void Test() {
     new Foo(null);

or using xUnit style assertions,

void Test() {
     Assert.Throws(delegate { new Foo(null); });

Pex ExpectedExcetions style

Let's refactor our first test and push 'null' as a test parameter. We add an assertion after the constructor to make sure a null parameter never succeeds.

void Test(object input) {
    new Foo(input);
    Assert.IsNotNul(input); // we should never get here

The interesting part is that this test will not only test for the null value but also for the passing values as well. Moreover, there might be more checks over the input which might trigger other exceptions. In that case, additional asserts could be added -- or even better, centralized in an invariant method.

So, what do you think?

posted on Saturday, February 23, 2008 1:40:50 AM (Pacific Standard Time, UTC-08:00)  #    Comments [2]
# Saturday, February 16, 2008

Looks like I've just made it in for the Seattle ALT.NET 'un'-conference :)

posted on Saturday, February 16, 2008 3:02:01 PM (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Friday, February 15, 2008

Some may wonder if there's any relation between my nickname 'Peli' and 'Pex': there's none! Pex just stands for "Program Exploration".

In fact, Pex was started long before I joined the project by my colleague Nikolai Tillmann. Nikolai started the MUTT project (read this), which is the ancestor of Pex. He's the mastermind behind the IL rewriter, symbolic engine, and well a large part of Pex :) Hopefully, I'll convince him to start a blog :).

posted on Friday, February 15, 2008 10:43:55 AM (Pacific Standard Time, UTC-08:00)  #    Comments [0]