# Saturday, July 14, 2007

Pex can analyze regular expressions*** and generate strings that matches them automatically!

What does it mean? Well, maybe, somewhere deep in your code your are validating some string with a regex (for example a url). In order to test the validation code, one needs to craft inputs that does not match (easy) and matches (harder) the regex.

Let Pex do it:

So what if Pex could be smart enough to understand a regex and craft inputs accordingly? In the example below, it would be very hard for a random generate to generate a string that matches the regex.

[PexTest]
public void Url([PexAssumeIsNotNull]string s)
{
    if (Regex.IsMatch(s, "(?<Protocol>\w+):\/\/(?<Domain>[\w@][\w.:@]+)\/?[\w\.?=%&=\-@/$,]*"))
        throw new PexCoverThisException(); // random won't find this
}

"" would be a failing match and foo://foo.com a good match. (To generate the correct match, my brain simulated the regex automaton and estimated one possible path). Interestingly, Pex generates....

Ā://Āā

Pretty ugly... but correct! This reminds us that while regex are used to validate input, what they'll let through is sometimes scary.

Compiled Regex + Pex = Love

The great part about supporting the regular expressions is that it comes for free (almost) since Regex can be compiled to IL in .Net. When the BCL generates the regex IL code, it effectily builds the automaton... which can be analyzed by Pex!!!

Refresher: Pex works analyzing the MSIL being executed.

Hey but not all Regex are compiled!

That's true. Compiling the regex is optional so Pex needs to do a little bit of 'plumbing' to make sure all regular expressions are compiled. This is simply done by substituting the real .ctor of the Regex class with a customized version that compiles the regex. I'll talk about substitutions deeper in the future.

*** Of course, the bigger the regex is, the harder it is going to be for Pex to craft a successful match.

posted on Saturday, July 14, 2007 2:49:53 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Sunday, July 08, 2007

I finally posted a project that I had been developing last year in my daily commutes: Toad (www.codeplex.com/toad).

Toad is an (experimental) compiler infrastructure to build languages top of .Net, without knowing much about IL. Mostly it was a way for me to learn first hand all the little subtleties of writing multiple language on top of .Net.

What does Toad give you?

  • A dynamic console that supports multiple languages,
  • An Expression/Statement tree that can be translated back to IL,
  • Built-in debugging capabilities,
  • A set of Visitors on top this AST that implement various optimization and language constructs (this is the fun part),
  • There's a lot of stuff, I used to take the bus twice a day. :)
  • Toad does not use the DLR maybe in the future.

What is it good for?

  • Toy around with IL and try to build your micro language.
  • Don't use it for anything serious.

Good For Nothing Language

 The 'Good for nothing' language is a toy language that Joel Pobar used to illustrate managed IL compilers. With Gfn, you can write simple program such as:

var i = 123;
print i;
for j = i to 10 do print i+j; end;

The parsing stuff

I took Joel's example and wrote a grammar for the Gold parser. It looks like something like this that is then compiled by Gold:

<Program> ::= <BlockStatement>

<BlockStatement> ::= <BlockStatement> <Statement> ';'
            | <Statement> ';'

<Statement> ::= 'var' Identifier '=' <Expression>
    | Identifier '=' <Expression>
    | 'for' Identifier '=' <Expression> 'to' <Expression> 'do' <BlockStatement> 'end'
    | 'read_int' Identifier
    | 'print' <Expression>
    | 'return' <Expression>

At the end of the process, you get a visitor for the code AST. This is the boiler plate code to start using Toad to build the compiler:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected virtual object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
     throw new NotSupportedException("RuleStatementforIdentifierEqtodoend");
}

Implementing the 'for' loop

Let's take a look at the implementation of the for loop. We need to:

  • extract the from, to expressions,
  • declare a variable for the index,
  • build the predicate i < to etc...,
  • emit symbols for debugging,

The code below shows the full implementation of that statement:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected override object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
    // from, to
    Expression from = (Expression)this.VisitNode(node[3]);
    Expression to = (Expression)this.VisitNode(node[5]);

    // x = from
    VariableDeclarationStatement init = Stm.Var(node[1].Data, from);

    // body
    this.Context.PushScope();
    this.Context.PushVariable(init.Variable);

    Unit body = (Unit)this.VisitNode(node[7]);

    // i < to;
    Expression ilto = Expr.LessThanOrEqual(Expr.Var(init), to);
    ilto.Symbol = FromNode(node[1]);
    // ++i
    Statement inc = Stm.Expr(Expr.PrePlusPlus(Expr.Var(init)));

    this.Context.PopScope();

    ForStatement forstm = Stm.For(init, ilto, inc);
    forstm.Body = Stm.FromUnit(body);

    return forstm;
}

Hooking up the visitor in Toad

The last step is to package our visitor as a 'language' and return the generated statements or expression to Toad. We can spin up the interactive console, load Gfn and try it out:

image

Nothing really mind blowing here. Let's turn on live IL debugging to see what happening under the hood. This mode emits the IL source of the method, adds a bunch of 'nop' instructions to force the debugger to step on each IL instruction:

image

You can see in the generated code how the expressions got translated into IL instructions. 

posted on Sunday, July 08, 2007 3:23:32 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Thursday, July 05, 2007

Pex has a couple samples that get bundled in our installer. We have them in our main solution so that we get the benefits of Refactoring (that they still compiled :)). This approach has one little problem: the sample projects are referencing the Pex projects... which are not shipped! If packaged as is, they'll never compile on the user's machine.

ProjectReference -> Reference

We solved this problem by implementing an MSBuild task that 'patches' .csproj project files by replacing ProjectReference nodes with Reference:

<ProjectReference Include="..\..\..\Pex\Framework\Microsoft.Pex.Framework.csproj">
  ...
</ProjectReference>

becomes

<Reference Include="Microsoft.Pex.Framework">...</Reference>

There's a bit of coding involved to it (resolving the assembly name, selecting the appropriate projects, etc...) but that's basically it :)

posted on Thursday, July 05, 2007 9:02:38 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, July 03, 2007

One of the problem with Pex is that.... it's yet another dependency in your test project. Pex has it's own attributes which makes it very difficult to 'strip it out' of the test source. Many teams won't allow to check-in assemblies, they won't install external components on the build machine and just forget about touching the source code!

So how do you strip Pex?

In "Condition" lies the answer

An elegant solution uses a bit of Reflection, CodeDom and the MSBuild Condition attribute: generate Attribute stubs (shadows) and bind them to the project.

Pex modifies the project file and uses MSBuild conditions to conditionally include files and assembly references in the generated test project.

  • A boolean property PexShadows to controls the shadowing state: true if shadowed, false or missing otherwize.
<Project DefaultTargets="Build" ...>
  <PropertyGroup>
    ...
    <PexShadows>true</PexShadows>
  </PropertyGroup>
  • a conditional reference to Microsoft.Pex.Framework:
    <Reference 
        Include="Microsoft.Pex.Framework" 
        Condition="$(PexShadows) != 'true'" />

when the property PexShadows evalutes to true, the Microsoft.Pex.Framework assembly is not referenced anymore.

  • a file containing all the custom attributes 'stubs' (for all attributes in the Microsoft.Pex.Framework) is generated (automatically of course :)). Pex also dumps the source of several other helper classes. Each generated file is added to the test project conditionally:
    <Compile Include="Properties\PexAttributes.cs" Condition="$(PexShadows) == 'true'">
      <AutoGen>True</AutoGen>
      <DependentUpon>AssemblyInfo.cs</DependentUpon>
    </Compile>

That's it! Flip the value of PexShadows to bind/unbing your test project from Pex.

So what does Visual Studio says ?

Visual Studio is totally fooled by the maneuver. It shows the conditional files and references as if nothing happened. Add a menu item in the project context menu to switch it on and off and we are good to go :)

posted on Tuesday, July 03, 2007 10:28:12 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, June 30, 2007

In Pex, we added the possibility to specify the type under test of a given fixture:

public class Account {...}


[TestFixture, PexClass(typeof(Account))]
public class AccountTest {...}  

That's nice but why would it be useful... Beyond the fact that it clearly expresses the 'target' of the fixture, this kind of information can be leverage by tools like Pex.

For example, since we know that Account is the type under test, we can tune Pex to prioritize the exploration of the Account type.

Another interesting side effect is the targeted code coverage data. Instead of getting coverage information over the entire assembly, we can directly provide coverage over the type under test: the AccountTest covered xx% of Account.

Still toying around the concept, one can add a special filtering mode to the command line to execute all tests that target 'Account':

pex.exe /type-under-test:Account Bank.Tests.dll
posted on Saturday, June 30, 2007 12:23:24 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, June 02, 2007

A nice side effect of the Pex infrastructure is that it computes coverage data while exploring a parameterized test. What does this mean to the user? Well, what about this specifying that a parameterized test should reach 100% coverage**:

[PexTest]
[PexExpectedCoverage(100)] // expect 100% coverage
public void UberTest(int i)
{ ... }
** What kind of coverage are we talking about?

Pex internally tracks implicit branch coverage (which can never be covered :)). So we usually refer to basic block coverage. By default, the coverage is reported for the code that was run by the parameterized test. This is a very important difference with usual coverage tools, which give numbers for the entire assembly.

posted on Saturday, June 02, 2007 10:24:56 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Saturday, May 26, 2007

.Net 2.0 has been out for a while and it seems that 'generics' have not made it into unit test frameworks (that I know of). When I write unit tests for generics, I don't want to have to instantiate them!

For example, if I have an generic interface,

interface IFoo<T> {...}

then I'd really like to write this kind of test and let the test framework figure out an interresting instantiation (i.e. choice of T):

[Test]
public void Test<T>(IFoo<T> foo) { ... }

In the example above, System.Object can be trivially used to instantiate IFoo<T>. Of course, things get more interresting when mixing type argument, method argument and constraints :)

interface IFoo<T>
{
    R Bar<R>(T t)
     where R : IEnumerable<T>
}

In Pex, we've started to look at this problem... stay tuned.

posted on Saturday, May 26, 2007 11:09:00 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [3]
# Friday, May 25, 2007

MbUnit 2.4 is out, check it out at

http://weblogs.asp.net/astopford/archive/2007/05/24/mbunit-2-4-rtm.aspx

If you are using MbUnit parameterized tests (i.e. RowTest, CombinatorialTest) then migrating to Pex should be a piece of cake :)

posted on Friday, May 25, 2007 8:25:02 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Sunday, April 22, 2007

3 years after being released on CodeProject, it was about time to dust off QuickGraph and give it generics support.

posted on Sunday, April 22, 2007 1:49:21 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Thursday, April 19, 2007

With parameterized unit tests, it is not uncommon to generate a large number of exceptions. An basic exception log usually looks like this: a small message and the stack trace.

Whith this kind of output, a lot of work is still left to the user since he has, *for each frame*,  to manually open the source file and move to the line refered by the trace.

Give me the source context!

In the Pex reports we added a couple lines of code to read the source for each frame and display it in the reports (no rocket science). The cool thing is that you now can get a pretty good idea of what happened without leaving the test report. Multiply that by dozens of exceptions and you've won a loooot of time.

Here are some screenshots to illustrate this: an exception was thrown in some arcane method. The error message is not really useful (as usual).

If we expand the source and actually see the code, things become much clearer...

posted on Thursday, April 19, 2007 5:12:40 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]