# Wednesday, October 10, 2007

When someone is writing a book that contains code snippets, the question of (automatically) keeping those in sync quickly becomes very imporant. There's already lots of different solutions to this problem (every author has probably it's own), here's yet another one for C# that we've developed to author the Pex documentation.

Goals

A couple things that we wanted to acheive with this tool:

  • snippets are always compilable and run as expected,
  • snippets can be full classes, methods or even partial statements
  • simple :)

'#region' based solution

This solution uses the #region directive to define a snippet. The region describe contains the snippet name, which will be used to dump it into a file. For example, given this piece of C#,

...
#region snippet StackExamplePart3
stack.Push(new object);
#endregion
...

Our parse will extract the code in the region and write it to StackExamplePart3.tex, which gets pulled in our LaTeX scripts.

\begin{verbatim}
stack.Push(new object);
\end{verbatim}

That's it?

Yes, you can author snippets that stay compilable and up to date:

  • we can author all the snippets in Visual Studio and we are sure they always compile
  • it's very easy to parse the #region's (left as exercise ;))
  • #region are very flexible in terms what they contain so we can have snippets containing partial methods, statement, etc...
  • the scheme aslo supports nested regions which is usefull when one explains an example line by line, and integrate the entire sample at the end. For example, DeclaringUnitTest is a 'sub'-snippet of UnitTest:
#region snippet UnitTest
#region snippet DeclaringUnitTest
[TestMethod]
void Test(int i)
#endregion
{
   
}
#endregion
  • we can integrate our snippets in unit tests and verify they work as expected
  • the tool can be integrated into the build process as a post-command build

 

posted on Wednesday, October 10, 2007 11:41:32 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [2]
# Saturday, September 29, 2007

In Pex, we have our own version of all (and more) collection classes from the BCL. This duplication may sound stupid but there's a very good reason behind it: since Pex instruments code (i.e. rewrites IL), the BCL collections might be instrumented as well... leading to a poorer performance (instrumented code is slower for many reasons). Therefore, to avoid this situation, we ended up using our own implementation which never gets instrumented.

While using our collections, we added 2 features that we like a lot: readonly interfaces and small collections.

Readonly interfaces

Readonly interfaces are handy to 'safely' expose collections, once you start using them you get hooked:

  • ICountable<T>, a readonly enumerable with a Count.
interface ICountable<T> : IEnumerable<T>
{
    int Count {get;}
}
  • ICopyable<T>, a countable that can be copied around,
interface ICopyable<T> : ICountable<T>
{
    void CopyTo(T[] array, int index);
}
  • IIndexable<T>, a indexed collection
interface IIndexable<T> :  ICopyable<T>
{
    T this[int index] { get; }
}

We also have a couple more interfaces for dictionary style collections but you get the idea.

Small Collections

Small collections are optimized collections with 0, 1 or 2 elements. Depending on your scenario, these can improve the performance of your application while lowering the pressure on the GC (create less objects -> less stuff to garbage collect). If you create a lot of objects which each have collections, ask yourself: are you lazy allocating the collections? what should be the initial size of the collection? do you expect more than one element on average?

posted on Saturday, September 29, 2007 11:33:15 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Sunday, September 23, 2007

xUnit, the new variation on the 'unit test framework' theme comes with support for data driven tests: 'Theories' (funny name by btw). Pex is a plugin for test frameworks, so we've added support for xUnit as well.

[PexClass] // xUnit does not have fixture attributes
public class MyTests
{
    [Theory, DataViaXXXX] // xUnit theories
    [PexTest] // let pex help you 
    public void Test(int a, ....)
    {}
}

posted on Sunday, September 23, 2007 9:26:32 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Thursday, September 13, 2007

Z3 is the high performance theorem prover that Pex uses to solve constraint systems (and find bugs). You can now play with it too! 

http://research.microsoft.com/projects/z3/

Z3 is written in C++, but it has a .Net API to make you happy.

posted on Thursday, September 13, 2007 5:01:41 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Friday, September 07, 2007

In a previous, we were looking at partial trust the lack of support for it. In this post, I'll show the key 'fixes' that we did to make Pex 'partial trust' aware.

Simulating Partial Trust

The easiest way to run under partial trust is to run your .net application from the network. However, in the context of a test framework, this would not work since many required permissions would not be granted (reflection, i/o, etc...). So we need a new AppDomain whose security policy considers the test framework assemblies as fully trusted.

  • Get a new AppDomain:
string trust = "LocalIntranet";
AppDomain domain = AppDomain.CreateAppDomain(trust);
  • Load the named permission set
PermissionSet permission = GetNamedPermissionSet(trust);
  • Create the code group structure that associate the partial trust permission to any code
UnionCodeGroup code= new UnionCodeGroup(
    new AllMembershipCondition(),
    new PolicyStatement(permission, PolicyStatementAttribute.Nothing));
  • give full trust to each test framework assembly:
StrongName strongName = CreateStrongName(typeof(TestFixtureAttribute).Assembly);
PermissionSet fullTrust = new PermissionSet(PermissionState.Unrestricted);
UnionCodeGroup fullTrustCode = new UnionCodeGroup(
new StrongNameMembershipCondition(strongName.PublicKey, strongName.Name, strongName.Version),
new PolicyStatement(fullTrust, PolicyStatementAttribute.Nothing));
code.AddChild(fullTrustCode);
  • Assign the policy to the AppDomain
PolicyLevel policy = PolicyLevel.CreateAppDomainLevel();
policy.RootCodeGroup = code;
domain.SetAppDomainPolicy(policy);

This is basically it (the rest of the details are left as an exercise :)).

Let them call you

Make sure to add the AllowPartiallyTrustedCallers to the test framework assembly otherwize users won't be allowed to call into it...

A twist...

Pex is bit invasive when it comes to partial trust. Pex rewrites the IL at runtime and turns all method bodies into... unsafe code (that is unverifiable). At this point, any will not run because of the SkipVerification permission.

No problemo, just add it to the permissionset:

permission.AddPermission(
    new SecurityPermission(SecurityPermissionFlag.SkipVerification)
    );

 

posted on Friday, September 07, 2007 11:39:22 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Tuesday, September 04, 2007

Writing code that will run in partial trust is tricky... and writing unit tests for it turns out to be even trickier. Does your favorite unit test framework support partial trust?

After looking on NUnit, MbUnit or VSTT, it did not look like so (if I overlooked this feature, don't hesitate to point it out to me). No magic command line flag or custom attribute to get the work done.

Even more interesting, the test framework assemblies don't have the AllowPartiallyTrustedCallersAttribute... so you can't even use their assertion classes in partial trust as it will trigger a SecurityException... Good stuff.

posted on Tuesday, September 04, 2007 10:01:05 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Thursday, August 23, 2007

A common requirement for unit test framework is the ability to test internal types.

That's easy! use InternalsVisibleToAttribute

With .Net 2.0 and up, this is a fairly easy task thanks to the InternalsVisibleToAttribute: add it to the product assembly to give 'visibility rights' to the test assembly.

// in assembly Foo
internal class Foo {}
// giving visibility rights to the Foo.Tests assembly
[assembly:InternalsVisibleTo("Foo.Tests")]

On the test assembly side, this works because unit test are 'closed' methods which do not expose any internal types.


[Test]
public void FooTest() {
    Foo foo = new Foo(); // we're using the internal type Foo
                         // but it's hidden in the unit test
}

What about parameterized tests? Make them internal as well

If one of the parameters of the test is internal, the test method will have to be internal as well in order to compile:

[PexTest]
internal void FooTest(Foo foo) {
   ...
}

Not pretty but still gets the job done. Pex will generate public unit test methods that invoke the internal parameterized test method, and we'll be happy:

[Test]
public void FooTest_12345() {
    this.FooTest(null);
}

What about MbUnit RowTest?

This issue was never faced by MbUnit RowTest because it only accepts intrinsic types such as int, long, etc... Those types are obviously public :)

posted on Thursday, August 23, 2007 10:24:08 AM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Wednesday, August 22, 2007

Finally found the time and motivation to upgrade the Reflection.Emit language to Reflector 5.0. Available now at http://www.codeplex.com/reflectoraddins .

image

posted on Wednesday, August 22, 2007 5:26:55 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Saturday, July 14, 2007

Pex can analyze regular expressions*** and generate strings that matches them automatically!

What does it mean? Well, maybe, somewhere deep in your code your are validating some string with a regex (for example a url). In order to test the validation code, one needs to craft inputs that does not match (easy) and matches (harder) the regex.

Let Pex do it:

So what if Pex could be smart enough to understand a regex and craft inputs accordingly? In the example below, it would be very hard for a random generate to generate a string that matches the regex.

[PexTest]
public void Url([PexAssumeIsNotNull]string s)
{
    if (Regex.IsMatch(s, "(?<Protocol>\w+):\/\/(?<Domain>[\w@][\w.:@]+)\/?[\w\.?=%&=\-@/$,]*"))
        throw new PexCoverThisException(); // random won't find this
}

"" would be a failing match and foo://foo.com a good match. (To generate the correct match, my brain simulated the regex automaton and estimated one possible path). Interestingly, Pex generates....

Ā://Āā

Pretty ugly... but correct! This reminds us that while regex are used to validate input, what they'll let through is sometimes scary.

Compiled Regex + Pex = Love

The great part about supporting the regular expressions is that it comes for free (almost) since Regex can be compiled to IL in .Net. When the BCL generates the regex IL code, it effectily builds the automaton... which can be analyzed by Pex!!!

Refresher: Pex works analyzing the MSIL being executed.

Hey but not all Regex are compiled!

That's true. Compiling the regex is optional so Pex needs to do a little bit of 'plumbing' to make sure all regular expressions are compiled. This is simply done by substituting the real .ctor of the Regex class with a customized version that compiles the regex. I'll talk about substitutions deeper in the future.

*** Of course, the bigger the regex is, the harder it is going to be for Pex to craft a successful match.

posted on Saturday, July 14, 2007 2:49:53 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [1]
# Sunday, July 08, 2007

I finally posted a project that I had been developing last year in my daily commutes: Toad (www.codeplex.com/toad).

Toad is an (experimental) compiler infrastructure to build languages top of .Net, without knowing much about IL. Mostly it was a way for me to learn first hand all the little subtleties of writing multiple language on top of .Net.

What does Toad give you?

  • A dynamic console that supports multiple languages,
  • An Expression/Statement tree that can be translated back to IL,
  • Built-in debugging capabilities,
  • A set of Visitors on top this AST that implement various optimization and language constructs (this is the fun part),
  • There's a lot of stuff, I used to take the bus twice a day. :)
  • Toad does not use the DLR maybe in the future.

What is it good for?

  • Toy around with IL and try to build your micro language.
  • Don't use it for anything serious.

Good For Nothing Language

 The 'Good for nothing' language is a toy language that Joel Pobar used to illustrate managed IL compilers. With Gfn, you can write simple program such as:

var i = 123;
print i;
for j = i to 10 do print i+j; end;

The parsing stuff

I took Joel's example and wrote a grammar for the Gold parser. It looks like something like this that is then compiled by Gold:

<Program> ::= <BlockStatement>

<BlockStatement> ::= <BlockStatement> <Statement> ';'
            | <Statement> ';'

<Statement> ::= 'var' Identifier '=' <Expression>
    | Identifier '=' <Expression>
    | 'for' Identifier '=' <Expression> 'to' <Expression> 'do' <BlockStatement> 'end'
    | 'read_int' Identifier
    | 'print' <Expression>
    | 'return' <Expression>

At the end of the process, you get a visitor for the code AST. This is the boiler plate code to start using Toad to build the compiler:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected virtual object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
     throw new NotSupportedException("RuleStatementforIdentifierEqtodoend");
}

Implementing the 'for' loop

Let's take a look at the implementation of the for loop. We need to:

  • extract the from, to expressions,
  • declare a variable for the index,
  • build the predicate i < to etc...,
  • emit symbols for debugging,

The code below shows the full implementation of that statement:

// <Statement> ::= for Identifier '=' <Expression> to <Expression> do <BlockStatement> end
protected override object VisitRuleStatementforIdentifierEqtodoend(SyntaxNode node)
{
    // from, to
    Expression from = (Expression)this.VisitNode(node[3]);
    Expression to = (Expression)this.VisitNode(node[5]);

    // x = from
    VariableDeclarationStatement init = Stm.Var(node[1].Data, from);

    // body
    this.Context.PushScope();
    this.Context.PushVariable(init.Variable);

    Unit body = (Unit)this.VisitNode(node[7]);

    // i < to;
    Expression ilto = Expr.LessThanOrEqual(Expr.Var(init), to);
    ilto.Symbol = FromNode(node[1]);
    // ++i
    Statement inc = Stm.Expr(Expr.PrePlusPlus(Expr.Var(init)));

    this.Context.PopScope();

    ForStatement forstm = Stm.For(init, ilto, inc);
    forstm.Body = Stm.FromUnit(body);

    return forstm;
}

Hooking up the visitor in Toad

The last step is to package our visitor as a 'language' and return the generated statements or expression to Toad. We can spin up the interactive console, load Gfn and try it out:

image

Nothing really mind blowing here. Let's turn on live IL debugging to see what happening under the hood. This mode emits the IL source of the method, adds a bunch of 'nop' instructions to force the debugger to step on each IL instruction:

image

You can see in the generated code how the expressions got translated into IL instructions. 

posted on Sunday, July 08, 2007 3:23:32 PM (Pacific Daylight Time, UTC-07:00)  #    Comments [0]