Wednesday, February 27, 2008

This is one problem for which we don't have an elegant solution yet:

what is the best way to craft a name for a generated test?

Let's see an example; given the following parameterized unit test,

[PexMethod] void Test(int i) {
    if (i == 123) throw ArgumentException();
}

Pex would generate 2 tests: i = 0, i = 123. So it seems doable to infer test names such as

[TestMethod] void Test0() { this.Test(0); }
[TestMethod, EE(typeof(ArgumentException))]
void Test123ThrowsArgumentException() { this.Test(123); }

So what's so difficult about it? Well, most PUT's aren't that simple and as the size of the generated parameter increases, the methods might increase as well (strings getting bigger). Here's a list of potential problems:

  • the method should stay relatively small (less than 80 chars),
  • the parameters might be objects or classes, which do look well in a string format,
  • generated strings might be huge and contain weird unicode characters,
  • the tests might involve mock choices which again do not render well to strings,
  • the more parameters the more cryptic things become

Our approach: Timestamps

To the light of all those problems, we've taken the shortcut route in Pex by simply using combination of the parameterized unit test method signature and the timestamp when the test is generated:

[TestMethod] void TestInt32_20080224_124301_35() { ... }

This is ugly, how can I change that?

We've added an extensibility point to support custom test naming scheme. After registering your 'namer', you will get opportunity to craft a test name following your favorite code standard. You will have the generated test, output, exception, etc... at your disposition to make an intelligent choice there.

Off course, you're also welcome to drop a comment on this post to suggest a better scheme.

Wednesday, February 27, 2008 8:07:24 PM UTC
Why do you need to 'name' the test?

The test name above is "Test"

What you are talking about is the Test Run, and that's a different type of object.

A test run should be the test name plus attributes such that the combination of the keys is unique.
Nick
Wednesday, February 27, 2008 11:25:45 PM UTC
We need to give a name to each unit test that we generate: 'Test' is the name of the parameterized unit test.

Ultimately, we need to create a method name that is unique so that the generated test suite compiles :)
Comments are closed.