Monday, June 21, 2004

The Database Populator Framework  follows the same idea of DbMonster (or at least, was partially inspired from it). The purpose of the framework is to provide a flexible (and smart) set of random data generators to populate your database.

Why ?

If you plan to do database testing, the solution are not much: use transaction, create/kill database after each unit test, clean up after each test or use the new technique using Enterprise Services.

The objective of DPF is not to test the database but provide "food" for your unit test. By generating data randomly (satisfying the constraint), you do not need to clean up and you will avoid test clashing on each other. Moreover, you can also use the DPF to test your database under load.

Random but no dumb

In order to generate data for testing database, you need to take into accounts the constraints on the columns and between the tables. By providing a DataSet representing you database, the framework analyses it and create a set of generators. The data generated satisfies all the constraint. Of course, each row generator can be customized for your own needs, but in general, the information contained in the dataset is enough.

The ideas

The idea are rather simple:

  • Each DataColumn has its corresponding IDataGenerator instance,
  • Each DataTable has its corresponding ITablePopulator instance,
  • Each UniqueConstraint has its corresponding IUniqueValidator instance which takes care of ensuring the all unique constraint are validated,
  • Each ForeignKeyConstraint has its corresponding IForeignKeyConstraint instance which will fetch valid foreign key values

The generation of a new row is made inside a ITablePopulator as follows:

  1. Create a row,
  2. Call each IDataGenerator and fill the row,
  3. For each foreign key, get relevant values,
  4. Verify unique constraints, if violated go to 1,
  5. return row

The first bits

The first bits of the framework is located in the MbUnit.Framework.Data namespace. Althout the framework still needs work, the first example is running and is quite promising.

I have built a simple database containing User - Order - Product - OrderProduct:

this.dataSet=new DataSet();

DataTable users=dataSet.Tables.Add("Users");
DataColumn userID = users.Columns.Add("UserID",typeof(int));
DataColumn userName=Users.Columns.Add("UserName",typeof(string));
DataColumn userName.AllowDBNull=false;

DataTable orders=dataSet.Tables.Add("Orders");
DataColumn orderID=orders.Columns.Add("OrderID",typeof(int));
DataColumn orderDate = orders.Columns.Add("OrderDate",typeof(DateTime));
DataColumn oUserID = orders.Columns.Add("UserID",typeof(int));

DataTable products=dataSet.Tables.Add("Products");
DataColumn productID=products.Columns.Add("ProductID",typeof(int));
DataColumn productName = products.Columns.Add("ProductName",typeof(string));
DataColumn productPrice = products.Columns.Add("ProductPrice",typeof(decimal));

DataTable orderProducts=dataSet.Tables.Add("OrderProducts");
DataColumn opOrderID=orderProducts.Columns.Add("OrderID",typeof(int));
DataColumn opProductID=orderProducts.Columns.Add("ProductID",typeof(int));
DataColumn quantity=orderProducts.Columns.Add("Quantity",typeof(int));
// pks
users.Constraints.Add("PK_Users",userID,true);
orders.Constraints.Add("PK_Orders",orderID,true);
products.Constraints.Add("PK_Products",productID,true);
orderProducts.Constraints.Add("PK_OrderProducts",
new DataColumn[]{ opOrderID, opProductID}
,true);
// fks
orders.Constraints.Add("FK_Orders_Users",userID,oUserID);
orderProducts.Constraints.Add("FK_OrderProducts_Orders",orderID,opOrderID);
orderProducts.Constraints.Add("FK_OrderProducts_Products",productID,opProductID);

This database is easily populated using the DPF:

// creating populator
IDatabasePopulator pop=new DatabasePopulator();
// anaylising internal structure
this.pop.Populate(this.db.DataSet);

After that, you can customize the behavior of every data generator. Once this is done, you are ready to "feed" your database:

//getting the users table populator
ITablePopulator userPop = pop.Tables[users];
// adding new row
users.Rows.Add( userPop.Generate() );
posted on Monday, June 21, 2004 1:03:00 PM UTC  #    Comments [8]
Tracked by:
"business grant money" (online) [Trackback]
"criminal lawyer nj" (online) [Trackback]
"new york lawyer" (online) [Trackback]
Monday, June 06, 2005 5:54:47 PM UTC
Peli's Blog
Monday, June 06, 2005 5:54:54 PM UTC
Peli's Blog
Monday, June 06, 2005 5:54:55 PM UTC
overflow
Monday, June 06, 2005 5:54:55 PM UTC
This looks interesting. I had never heard of DbMonster. A long time ago, I used a perl script called nonsense, for one half of this problem (getting the random - but controlled - test data). I recently ported it to C#, and wrote a small program to fill a database with an obscene amount of data. I would have liked to have had something a little bit more intelligent than hard coded SqlCommands doing inserts.
<br>
<br>I am going to spend a few days with this and <a title="MbUnit, Generating Unit Testing and Model Based Testing Framework for .NET Framework" href="http://mbunit.tigris.org" target="_blank">MbUnit</a>, which I just took my first serious look at (and like). I have been using NUnit for quite a while, and it looks like transitioning from NUnit to <a title="MbUnit, Generating Unit Testing and Model Based Testing Framework for .NET Framework" href="http://mbunit.tigris.org" target="_blank">MbUnit</a> is a no brainer. Thanks!
Chris Bilson
Monday, June 06, 2005 5:54:56 PM UTC
Hi Chris,
<br>
<br>Actually, <a title="MbUnit, Generating Unit Testing and Model Based Testing Framework for .NET Framework" href="http://mbunit.tigris.org" target="_blank">MbUnit</a> can load NUnit fixture without any recompilation. It does not handle well TestFixuteSetUp or TestFixtureTearDown.
<br>
<br>On the database smart data generation, I will be working on it next in order to prepare the workshop I'm presenting on september 9. I plan to &quot;feed&quot; Northwind with data.
<br>
<br>The way I see the &quot;data&quot; generator, it should be used in conjunction with a business layer (that already have). So it is not the job of the generator to actually do the inserts, he's job is only to generate new data, compatible with the db schema and state.
<br>
<br>:)
Jonathan de Halleux
Monday, June 06, 2005 5:54:56 PM UTC
I have the same need to generate a large amount of data, and before starting to write a program from scratch i did some googoling around. The project you're talking about seems very interesting. If i can help i'll be very glad.
<br>I can program with C#, VB.NET and T-SQL without problems.
Davide Mauri
Monday, June 06, 2005 5:54:57 PM UTC
The framework is operational. Make sure you have a look at the following entries
<br>
<br><a target="_new" href="http://blog.dotnetwiki.org/archive/2004/09/06/908.aspx">http://blog.dotnetwiki.org/archive/2004/09/06/908.aspx</a>
<br><a target="_new" href="http://blog.dotnetwiki.org/archive/2004/09/10/941.aspx">http://blog.dotnetwiki.org/archive/2004/09/10/941.aspx</a>
<br>
<br>You will get the latest bits of the framework with <a title="TestDriven.NET" href="http://www.testdriven.net" target="_blank">TestDriven.NET</a>
Jonathan de Halleux
Monday, June 06, 2005 5:54:57 PM UTC
While learning to write unit tests for my class library code i've noticed that I repeatedly have to write setup code that involves hard coding data. This seems ok until you realize this is now static data and should always pass given no other changes. I b
Test it till it hurts
Comments are closed.