This post is just a small remainder of the meta-physical questions I'm asking myself, specially because testing is basically trying to solve an unsolvable problem (Halting theorem)

1. What is a bug ? (I like this one)
2. Are there family of bugs ? (see Mutation Testing) Can we detect them statically (i.e. using Metadata only)
3. Is there a way to get an a-priori estimation on the location of the bugs ?
4. Can we apply Risk Managment technology to source code ? Automatically ?
5. How can you effectively test a database ? By effectively I mean: not restricted to transactions, not database reconstruction for each test,
6. How can you effectively test a GUI ? By effectively I mean, without having to actually record and program user interfaction,
7. Should you test the code that does the testing ? If yes, should you test the code that tests code that does the testing... and so on,
8. You are facing an untested application, you don't have the time to test all the code. Where should you start ?

 

I have attended yesterday to a presentation about SharePoint by Patrick Tissegem and Inge De Neef (?could not find her blog?). Inge, sorry for the question :). Patrick was presenting through a VPN and doing live stuff on this SharePoint site (Inge also did live manipuation). It did not crash, amazing. This was my first contact with SharePoint and it looked like a terrific product for creating intranet, but it also appeared to me that it could quickly become a mess if you start tweaking it!

I also met a lot of fellow belgian bloggers. Since I'm not good with names, I must have forgotten many but here are the people I saw and spoke to: Jan Tiellens, Tom Mertens (MSDN content manager I think), Thomas Delrue (who will be doing an internship at Microsft as SDE/T this summer), and I forgot the other... Make yourself heard :)

There was also a surprise from the MSDN team before the presentation where Tom said:

"We have two .Net gurus in the room[...] I would like you to give a warm applause to Jonathan de Halleux and Thomas Delrue[...]".

I must say I was not easy with that, but he then said:

"Microsoft has a gift for both of them[...] bottles of champagne[...]"

Then I was much more relax :) :) :)

That's another graph-based Reflector Add-In. It creates and displays the dependency graph between the assemblies that are loaded in Reflector.

ps: Thanks to Lutz who wrote the AssemblyGrapher application to make it a Reflector add-in.

I have received an interresting post from Kent Boogaart on the CodeProject article that pointed out a major limitation of Composite Unit Testing:

I'm a little unsure how this tests all an interface's members. Take ICollection, for example. Your CollectionTest implementation would receive an instance of ICollection manufactured via an appropriate factory.

Say you wanted to test that ICollection.Count was returning the correct value. How would you test this without knowledge of the factory method called?

Thanks,
Kent

There he has a point. You cannot test ICollection.Count using Composite Unit Testing as it is now. You could test it through the IList test interface, but that's not the point.

A way of overcoming the limitation would be the ability for the user to give a "gold" instance along with the test instance. The gold instance would be mock of the interface as it "should" be and could be used to do the ICollection.Count test. For each factory method, the framework would look for it's mock counter part, if found, it is feeded to the test method. Let's illustrate that on ICollection:

The gold instance class (mock):

// the mock
public class CollectionMock : ICollection
{
    private int count=0;
    ...
    public int Count
    { 
        get{ return count;}
        set{this.count=value;}
    }
    ...
}

The tested instance factory

public class ArrayListFactory
{
    // provider for empty arraylist
    [Factory] // new attribute, tells the framework it is a factory method
    public ArrayList Empty
    {
        get
        {
            return new ArrayList();
        }
    }
    [MockFactory] // tells the framework it is a mock method
    public MockCollection EmptyMock // name is important must be "Method"+Mock
    {
        get
        {
            MockCollection col = new MockCollection();
            return col;
        }
    }
    // 1 provider + mock
    [Factory]
    public ArrayList One
    {
        get
        {
            return Empty.Add(null);
        }
    }
    [MockFactory]
    public MockCollection OneMock // name is important must be "Method"+Mock
    {
        get
        {
            MockCollection col = new MockCollection(); 
            col.Count=1;
            return col;
        }
    }
}

The test fixture:

[TypeFixtue(typeof(ICollection)]
[ProviderFactory(typeof(ArrayListFactory),typeof(ICollection))] 
public class CollectionTest
{
    [Test]
    public void CountTest(ICollection tested, ICollection mock)
    {
        Assert.AreEqual(mock.Count,tested.Count,"ICollection.Count");
    }
}

The framework would detect if a mock is available and automatically feed it to the method. Any comments ?

I have been facing a silly bug related to AppDomain, so stupid that it is worth mentionned in a blog.

The symptoms

Load an assembly in a separate AppDomain, create an instance of a type. Things work correctly. Then unload the domain, and recreate the type. Somehow, strange bugs (Custom Attributes not detected, the debugger messed up) arise.

How-to recreate it:

Assume that we have the Assembly MbUnit.Core.dll, MbUnit.Framework.dll (that references Core) and Test.dll that references both. Core defines an abstract Custom Attribute TestPatternFixtureAttribute,

public abstract TestFixturePatternAttribute :Attribute{...}

that is inherited in Framework as TestFixtureAttribute

public class TestFixtureAttribute : TestFixturePatternAttribute{...}

and TestFixtureAttribute is used in Test to tag fixture classes

[TestFixture]
public class MyTest{...}

At last, Core contains RemoteTestTree a serializable class that takes care of loading assemblies, extracting fixtures (class taged with TestPatternFixtureAttribute) and launching tests.

1. Create a separate AppDomain using AppDomain.CreateDomain,
2. Create an instance of RemoteTestTree using AppDomain.CreateInstanceFromAndUnWrap
3. Load Test.dll in the domain and find fixtures. At this point, everything works and we found the fixtures.
4. Unload the separate AppDomain using AppDomain.UnLoad,
5. Re-Create an instance of RemoteTestTree (at this point, weird things happen because the debugger seems to tell: RemoteTestTree is not initialized)
6. Load Test.dll in the domain and find fixtures. This time, no fixture is found because the framework seems to tell that TestFixtureAttribute does not inherit from TestPatternAttribute.

The fix

In point 2,5, I have been using AppDomain.CreateInstanceFromAndUnWrap, this is the error. I should have been using AppDomain.CreateInstanceAndUnWrap. The difference is subtle, 4 letters.

The explanation

I'm still unsure of the explanation.... so I'm waiting for someone to come up with a relevant explanation :)

With the kind help of Lutz Roeder, I have recompiled the IL Grapher into a Reflector Add-in...

In a near future, MbUnit will support storing of the test results in a SQL database (SQL server supported). The database structure is done and the BLL/DAL is finished. Here's a snapshot of the database schema.

 

I have added the possibility to sort fixture by importance (or severity): Critical, Severe, Default, NoOneReallyCaresAbout.

[TestFixture]
[Importance(TestImportance.Critical)]
public SomeFixture
{...}

A number of new assertions classes have been added to MbUnit since the latest post on this topic.  The new helper classes involve Arrays, collection, compiler, serialization, web...

ArrayAssert

This helper class contains method to compare arrays:

byte[] expected = ...;
byte[] actual = ...;
ArrayAssert.AreEqual(expected,actual);

The method compares the rank, the length and makes element-wize comparaison.

ColAssert

This class provides several methods to compare two ICollection instance:

ICollection expected = ...;
ICollection actual = ...;

ColAssert.IsSynchronized(actual);
ColAssert.AreCountEqual(expected,actual);
ColAssert.AreEqual(expected,actual);

The class also provides method to test the collection count, syncroot, synchronization, etc...

SerialAssert

This class contains various methods to test the "serializability" of objects.

SerialAssert.IsXmlSerializable(typeof(MyClass));

WebAssert

This class contains assertions on the properties of web control and web pages.

CompilerAssert

This class contains assertions to check that snippets are compilable:

String source = ...; // C# code to compile
// verify that source compiles
CompilerAssert.Compiles(CompilerAssert.CSharpCompiler, source);

What about your assertions ?

There is also a CodeSmith template that can let you build "strongly-typed" assertion classes out of existing types. See in the templates directory.

I just came back from Microsoft, Redmond where I attented interviews for SDE/T in the CLR team. (Interviews at Microsoft is a unique experience on its own). It looks like it worked because I'm moving in October to Redmond. :) I would like to thank Michael Corning, Harry Robinson and Holly Barbacovi for their support on this adventure.

Me and Michael Corning at Building 44 in Redmond.

Don't be fooled by the bad quality of the photo, it was pooring rain (the famous Redmond weather).