In a previous, we were looking at partial trust the lack of support for it. In this post, I'll show the key 'fixes' that we did to make Pex 'partial trust' aware.

Simulating Partial Trust

The easiest way to run under partial trust is to run your .net application from the network. However, in the context of a test framework, this would not work since many required permissions would not be granted (reflection, i/o, etc...). So we need a new AppDomain whose security policy considers the test framework assemblies as fully trusted.

• Get a new AppDomain:

string trust = "LocalIntranet";
AppDomain domain = AppDomain.CreateAppDomain(trust);

• Load the named permission set

PermissionSet permission = GetNamedPermissionSet(trust);

• Create the code group structure that associate the partial trust permission to any code

UnionCodeGroup code= new UnionCodeGroup(
    new AllMembershipCondition(),
    new PolicyStatement(permission, PolicyStatementAttribute.Nothing)); 

• give full trust to each test framework assembly:

StrongName strongName = CreateStrongName(typeof(TestFixtureAttribute).Assembly);
PermissionSet fullTrust = new PermissionSet(PermissionState.Unrestricted);
UnionCodeGroup fullTrustCode = new UnionCodeGroup(
    new StrongNameMembershipCondition(strongName.PublicKey, strongName.Name, strongName.Version), 
    new PolicyStatement(fullTrust, PolicyStatementAttribute.Nothing));
code.AddChild(fullTrustCode);

• Assign the policy to the AppDomain

PolicyLevel policy = PolicyLevel.CreateAppDomainLevel();
policy.RootCodeGroup = code;
domain.SetAppDomainPolicy(policy); 

This is basically it (the rest of the details are left as an exercise :)).

Let them call you

Make sure to add the AllowPartiallyTrustedCallers to the test framework assembly otherwize users won't be allowed to call into it...

A twist...

Pex is bit invasive when it comes to partial trust. Pex rewrites the IL at runtime and turns all method bodies into... unsafe code (that is unverifiable). At this point, any will not run because of the SkipVerification permission.

No problemo, just add it to the permissionset:

permission.AddPermission(
    new SecurityPermission(SecurityPermissionFlag.SkipVerification)
    ); 

Writing code that will run in partial trust is tricky... and writing unit tests for it turns out to be even trickier. Does your favorite unit test framework support partial trust?

After looking on NUnit, MbUnit or VSTT, it did not look like so (if I overlooked this feature, don't hesitate to point it out to me). No magic command line flag or custom attribute to get the work done.

Even more interesting, the test framework assemblies don't have the AllowPartiallyTrustedCallersAttribute... so you can't even use their assertion classes in partial trust as it will trigger a SecurityException... Good stuff.

A common requirement for unit test framework is the ability to test internal types.

That's easy! use InternalsVisibleToAttribute

With .Net 2.0 and up, this is a fairly easy task thanks to the InternalsVisibleToAttribute: add it to the product assembly to give 'visibility rights' to the test assembly.

// in assembly Foo
internal class Foo {}
// giving visibility rights to the Foo.Tests assembly
[assembly:InternalsVisibleTo("Foo.Tests")]

On the test assembly side, this works because unit test are 'closed' methods which do not expose any internal types.

[Test]
public void FooTest() {
    Foo foo = new Foo(); // we're using the internal type Foo 
                         // but it's hidden in the unit test
}

What about parameterized tests? Make them internal as well

If one of the parameters of the test is internal, the test method will have to be internal as well in order to compile:

[PexTest]
internal void FooTest(Foo foo) {
   ... 
}

Not pretty but still gets the job done. Pex will generate public unit test methods that invoke the internal parameterized test method, and we'll be happy:

[Test]
public void FooTest_12345() {
    this.FooTest(null);
}

What about MbUnit RowTest?

This issue was never faced by MbUnit RowTest because it only accepts intrinsic types such as int, long, etc... Those types are obviously public :)

Finally found the time and motivation to upgrade the Reflection.Emit language to Reflector 5.0. Available now at http://www.codeplex.com/reflectoraddins .

Pex can analyze regular expressions*** and generate strings that matches them automatically!

What does it mean? Well, maybe, somewhere deep in your code your are validating some string with a regex (for example a url). In order to test the validation code, one needs to craft inputs that does not match (easy) and matches (harder) the regex.

Let Pex do it:

So what if Pex could be smart enough to understand a regex and craft inputs accordingly? In the example below, it would be very hard for a random generate to generate a string that matches the regex.

[PexTest]
public void Url([PexAssumeIsNotNull]string s)
{
    if (Regex.IsMatch(s, "(?<Protocol>\w+):\/\/(?<Domain>[\w@][\w.:@]+)\/?[\w\.?=%&=\-@/$,]*"))
        throw new PexCoverThisException(); // random won't find this
}

"" would be a failing match and foo://foo.com a good match. (To generate the correct match, my brain simulated the regex automaton and estimated one possible path). Interestingly, Pex generates....

Ā://Āā

Pretty ugly... but correct! This reminds us that while regex are used to validate input, what they'll let through is sometimes scary.

Compiled Regex + Pex = Love

The great part about supporting the regular expressions is that it comes for free (almost) since Regex can be compiled to IL in .Net. When the BCL generates the regex IL code, it effectily builds the automaton... which can be analyzed by Pex!!!

Refresher: Pex works analyzing the MSIL being executed.

Hey but not all Regex are compiled!

That's true. Compiling the regex is optional so Pex needs to do a little bit of 'plumbing' to make sure all regular expressions are compiled. This is simply done by substituting the real .ctor of the Regex class with a customized version that compiles the regex. I'll talk about substitutions deeper in the future.

*** Of course, the bigger the regex is, the harder it is going to be for Pex to craft a successful match.

I finally posted a project that I had been developing last year in my daily commutes: Toad (www.codeplex.com/toad).

Toad is an (experimental) compiler infrastructure to build languages top of .Net, without knowing much about IL. Mostly it was a way for me to learn first hand all the little subtleties of writing multiple language on top of .Net.

What does Toad give you?

• A dynamic console that supports multiple languages,
• An Expression/Statement tree that can be translated back to IL,
• Built-in debugging capabilities,
• A set of Visitors on top this AST that implement various optimization and language constructs (this is the fun part),
• There's a lot of stuff, I used to take the bus twice a day. :)
• Toad does not use the DLR maybe in the future.

What is it good for?

• Toy around with IL and try to build your micro language.
• Don't use it for anything serious.

Good For Nothing Language

 The 'Good for nothing' language is a toy language that Joel Pobar used to illustrate managed IL compilers. With Gfn, you can write simple program such as:

var i = 123;
print i;
for j = i to 10 do print i+j; end;

The parsing stuff

<Program> ::= <BlockStatement>

<BlockStatement> ::= <BlockStatement> <Statement> ';'
            | <Statement> ';'

<Statement> ::= 'var' Identifier '=' <Expression>
    | Identifier '=' <Expression>
    | 'for' Identifier '=' <Expression> 'to' <Expression> 'do' <BlockStatement> 'end'
    | 'read_int' Identifier
    | 'print' <Expression>
    | 'return' <Expression>

At the end of the process, you get a visitor for the code AST. This is the boiler plate code to start using Toad to build the compiler:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected virtual object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
     throw new NotSupportedException("RuleStatementforIdentifierEqtodoend");
}

Implementing the 'for' loop

Let's take a look at the implementation of the for loop. We need to:

• extract the from, to expressions,
• declare a variable for the index,
• build the predicate i < to etc...,
• emit symbols for debugging,

The code below shows the full implementation of that statement:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected override object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
    // from, to
    Expression from = (Expression)this.VisitNode(node[3]);
    Expression to = (Expression)this.VisitNode(node[5]);

    // x = from
    VariableDeclarationStatement init = Stm.Var(node[1].Data, from);

    // body
    this.Context.PushScope();
    this.Context.PushVariable(init.Variable);

    Unit body = (Unit)this.VisitNode(node[7]);

    // i < to;
    Expression ilto = Expr.LessThanOrEqual(Expr.Var(init), to);
    ilto.Symbol = FromNode(node[1]);
    // ++i
    Statement inc = Stm.Expr(Expr.PrePlusPlus(Expr.Var(init)));

    this.Context.PopScope();

    ForStatement forstm = Stm.For(init, ilto, inc);
    forstm.Body = Stm.FromUnit(body);

    return forstm;
}

Hooking up the visitor in Toad

The last step is to package our visitor as a 'language' and return the generated statements or expression to Toad. We can spin up the interactive console, load Gfn and try it out:

Nothing really mind blowing here. Let's turn on live IL debugging to see what happening under the hood. This mode emits the IL source of the method, adds a bunch of 'nop' instructions to force the debugger to step on each IL instruction:

You can see in the generated code how the expressions got translated into IL instructions. 

Pex has a couple samples that get bundled in our installer. We have them in our main solution so that we get the benefits of Refactoring (that they still compiled :)). This approach has one little problem: the sample projects are referencing the Pex projects... which are not shipped! If packaged as is, they'll never compile on the user's machine.

ProjectReference -> Reference

We solved this problem by implementing an MSBuild task that 'patches' .csproj project files by replacing ProjectReference nodes with Reference:

<ProjectReference Include="..\..\..\Pex\Framework\Microsoft.Pex.Framework.csproj">
  ...
</ProjectReference>

becomes

<Reference Include="Microsoft.Pex.Framework">...</Reference>

There's a bit of coding involved to it (resolving the assembly name, selecting the appropriate projects, etc...) but that's basically it :)

One of the problem with Pex is that.... it's yet another dependency in your test project. Pex has it's own attributes which makes it very difficult to 'strip it out' of the test source. Many teams won't allow to check-in assemblies, they won't install external components on the build machine and just forget about touching the source code!

So how do you strip Pex?

In "Condition" lies the answer

An elegant solution uses a bit of Reflection, CodeDom and the MSBuild Condition attribute: generate Attribute stubs (shadows) and bind them to the project.

Pex modifies the project file and uses MSBuild conditions to conditionally include files and assembly references in the generated test project.

• A boolean property PexShadows to controls the shadowing state: true if shadowed, false or missing otherwize.

<Project DefaultTargets="Build" ...>
  <PropertyGroup>
    ...
    <PexShadows>true</PexShadows>
  </PropertyGroup>

• a conditional reference to Microsoft.Pex.Framework:

    <Reference 
        Include="Microsoft.Pex.Framework" 
        Condition="$(PexShadows) != 'true'" />

when the property PexShadows evalutes to true, the Microsoft.Pex.Framework assembly is not referenced anymore.

• a file containing all the custom attributes 'stubs' (for all attributes in the Microsoft.Pex.Framework) is generated (automatically of course :)). Pex also dumps the source of several other helper classes. Each generated file is added to the test project conditionally:

    <Compile Include="Properties\PexAttributes.cs" Condition="$(PexShadows) == 'true'">
      <AutoGen>True</AutoGen>
      <DependentUpon>AssemblyInfo.cs</DependentUpon>
    </Compile>

That's it! Flip the value of PexShadows to bind/unbing your test project from Pex.

So what does Visual Studio says ?

Visual Studio is totally fooled by the maneuver. It shows the conditional files and references as if nothing happened. Add a menu item in the project context menu to switch it on and off and we are good to go :)

In Pex, we added the possibility to specify the type under test of a given fixture:

public class Account {...}
 

[TestFixture, PexClass(typeof(Account))]
public class AccountTest {...}  

That's nice but why would it be useful... Beyond the fact that it clearly expresses the 'target' of the fixture, this kind of information can be leverage by tools like Pex.

For example, since we know that Account is the type under test, we can tune Pex to prioritize the exploration of the Account type.

Another interesting side effect is the targeted code coverage data. Instead of getting coverage information over the entire assembly, we can directly provide coverage over the type under test: the AccountTest covered xx% of Account.

Still toying around the concept, one can add a special filtering mode to the command line to execute all tests that target 'Account':

pex.exe /type-under-test:Account Bank.Tests.dll

A nice side effect of the Pex infrastructure is that it computes coverage data while exploring a parameterized test. What does this mean to the user? Well, what about this specifying that a parameterized test should reach 100% coverage**:

[PexTest]
[PexExpectedCoverage(100)] // expect 100% coverage
public void UberTest(int i)
{ ... }

** What kind of coverage are we talking about?

Pex internally tracks implicit branch coverage (which can never be covered :)). So we usually refer to basic block coverage. By default, the coverage is reported for the code that was run by the parameterized test. This is a very important difference with usual coverage tools, which give numbers for the entire assembly.