Update: Andrew Kazyrevich does the final performance analysis on his blog.
Update II: Stubs v0.12.40430.3 has Partial Stubs support.
Stubs is a lightweight stub framework that is entirely based on delegates. We designed it so that it brings as little overhead as possible, and as a side effect, is very effective with Pex. In this post, we’ll see how Stubs ‘look’ with respect to other frameworks.
Andrew Kazyrevich started a project, mocking-framework-compare, back in February that compared Moq, Rhino, NMock2 and Isolator. This is project compares various ‘mocking scenarios’ across all those frameworks. I added Stubs to the mix today (go check it out).
What’s a stub in Stubs anyway?
Before we dig in, let’s recap how stubs look like when you use Stubs. Stubs uses delegate to ‘hook’ behavior to interface members. For every stubbed interface and class, code is generated at a compile time: One stub class per interface and class. For each stubbed method, the stub class contains a field. The field has a delegate type matching the stubbed method. As a user, you can simply attach delegates (or lambdas) to the delegate fields to assign*any* behavior to each member. If no user delegate is set, Stubs calls into a fallback behavior, i.e. throw or return the default value.
Let’s take one of the examples of mocking-framework-compare. The little snippet below ‘stubs’ the TouchIron() method (which is implemented explicitly) by attaching a lambda to the TouchIron field.
SIHand is the stub type of the IHand interface, it was generated at compile time by the Stubs framework. Its implementation looks more or less like this:
The TouchIron methods really shows the main idea behind stubs: delegates are all we need to move behavior around.
Stubs: relying on the language rather than an API
One of the differences of Stubs with other mock frameworks is that Stubs does not have any API: delegates, lambdas, closures are all features of the programming language, while assertions are part of the test framework. This is quite obvious when we compare the BrainTest using Moq and using Stubs. In this test, we need to ensure that the brain coordinates the hand and the mouth so that when a hot iron is touched by the hand, the mouth yells.
- Moq: we set up the hand to throw a burn exception when touched with a hot iron and verify that the mouth yelled.
Moq, as Rhino or Isolate, make smart use of expression trees and/or Reflection.Emit to define the mock behavior. The expectation and behavior are set through nice and fluent APIs which really make the developer’s life’s easier.
- Stubs: same scenario, we attach a lambda that throws if iron is hot and use a local to verify that Yell is called (this local is pushed to the heap by the compiler).
Stubs does not have any API. It relies on lambdas (to define new behaviors), closures (to track side effects) which are given for free by the C# 3.0 language.
Does performance matter when it comes to mocking? It’s not really the question we are trying to answer here. We’re just looking at the benchmark results from the mocking-framework-compare project. When looking at the results, you’ll notice that Stubs may be 100x to 1000x faster than other frameworks. This is no surprise since stubs boil down to a virtual method call, while other frameworks do much more work (in fact we expected worse numbers).
Data units of msec resolution = 0.394937 usec
Moq : 100 repeats: 37.471 +- 12% msec
Rhino : 100 repeats: 38.030 +- 7% msec
NMock2 : 100 repeats: 24.035 +- 4% msec
Stubs : 100 repeats: 0.115 +- 8% msec
Moq : 100 repeats: 86.913 +- 7% msec
Rhino : 100 repeats: 61.142 +- 6% msec
NMock2 : 100 repeats: 27.378 +- 6% msec
Stubs : 100 repeats: 0.071 +- 6% msec
Moq : 100 repeats: 82.434 +- 6% msec
Rhino : 100 repeats: 47.471 +- 5% msec
NMock2 : 100 repeats: 11.334 +- 10% msec
Stubs : 100 repeats: 0.042 +- 15% msec
Mocking method arguments.
Moq : 100 repeats: 142.668 +- 4% msec
Rhino : 100 repeats: 45.118 +- 5% msec
NMock2 : 100 repeats: 22.344 +- 7% msec
Stubs : 100 repeats: 0.078 +- 4% msec
Moq : 100 repeats: 117.581 +- 5% msec
Rhino : 100 repeats: 58.827 +- 6% msec
Stubs : 100 repeats: 0.054 +- 6% msec
Moq : 100 repeats: 92.482 +- 4% msec
Rhino : 100 repeats: 40.921 +- 3% msec
Stubs : 100 repeats: 0.493 +- 18% msec
Press any key to continue . . .
There are many areas where the Stubs fall short or simply don’t support it. Currently, Stubs are only emitted for C# and partial mocks will be supported in the next version.
Getting started with Stubs
Stubs comes with the Pex installer. If you’re interested on using it, check out the getting started page on the Stubs project page where you can also download the full Stubs primer.