First attempts with Gallio

February 14, 2009 at 2:21 PMAndre Loker

From time to time I like to check my development workflow and tools to see whether I’m still up to date. This time I thought it might be a nice idea to give Gallio a shot. If you don’t now Gallio: it is a platform to uniformly run unit tests from whatever unit test framework (MbUnit, NUnit, MSTest etc.) you use. At least that’s how I understand Gallio’s mission. It also integrates into a bunch of other tools which I use (CruiseControl.NET, NAnt, NCover, ReSharper et al).

So basically it sounds like a cool idea. On top of it, version 3 of MbUnit – my favourite unit testing framework – is available only bundled with Gallio, that's why decided to give it try. Until I only used MbUnit 2.4.

And here’s a list of twitter-esque thoughts about Gallio that came up during my first 60 minutes of use.

  • Installation is no problem
  • Migrating from MbUnit 2.4 to MbUnit 3 didn’t take much effort, at least for the small project I chose for my Gallio evaluation. Mainly I had to replace the RowTest attribute with a conventional Test attribute, because row tests have been consolidated into the normal tests. But you can also use the MbUnit.Compatibility Assembly and don’t change the existing tests at all. Big plus!
  • The documentation is rather incomplete. There are still a lot of TODOs. Especially a migration guide for MbUnit 2 users would be really useful.
  • Integration into Testdriven.NET works well, no problems here…
  • … although the test runner feels a little less snappy than MbUnit 2.4, startup time is slightly higher. The 190 unit tests in the small test project took about 5,2 seconds using MbUnit 2.4 and 7 seconds using MbUnit. I’ve yet to test how this scales with more tests.
  • ReSharper integration seems to be incomplete and instable. If I use MbUnit 2.4 tests, not all tests are run. Only about 30 of my tests appeared in the unit test browser. Using MbUnit 3, all tests were found, but occasionally ReSharper through exceptions when I selected failed tests.
  • Using Testdriven.NET’s “Test with”=>”Gallio Icarus” context menu action fires up Icarus (the Gallio test running GUI) but doesn’t load the assembly of the project that I want to test.
  • Running tests in Icarus is unbearably slow. The tests took more than 40 seconds to run (compare that to the 7 seconds using Testdriven.NET or even 5 seconds using MbUnit 2.4 + Testdriven.NET).
    I guess this is due to the fact that the result window is refreshed after every single unit tests. Thus, the performance even degrades with every additional unit tests. I wonder what happened if I were to run tests of my bigger projects.
  • The one real showstopper however is the fact that suddenly unit tests begin to fail that have been perfectly valid before. This happens when I use Rhino.Mocks to create a partial mock and expect for an ArgumentNullException to be thrown in the ctor of the partially mocked type. Luckily this is easily reproducible:

Class under test:

   1: public abstract class SomeClass {
   2:   // Throws ArgumentNullException if value is null
   3:   protected SomeClass(string value) {
   4:     if (value == null) {
   5:       throw new ArgumentNullException("value");
   6:     }
   7:   }
   8: }

The unit test:

   1: [Test, ExpectedArgumentNullException]
   2: public void CtorShouldThrowIfArgumentIsNull() {
   3:   var mocks = new MockRepository();
   4:   mocks.PartialMock<SomeClass>(default(string));
   5: }

Results when run under MbUnit 2.4


Results when run under MbUnit 3


A System.ArgumentNullException is expected, but instead a System.ArgumentNullException has been thrown. Possibly it’s a framework version conflict, I don’t know. Anyway, a bug report is on its way (uodate 02/20/2009: the bug report has been closed as won’t fix).

My personal conclusion for now

I really appreciate the idea behind Gallio to unify unit testing. Different projects use different unit testing frameworks. Especially in connection with automated builds it could be useful to have one test runner that supports them all. And of course, I really like that they have chosen MbUnit as their primary unit testing framework, because I think MbUnit rocks and is superior to the rather basic NUnit. I haven’t looked much into the version 3 API of MbUnit, but I think there are some nifty new features that can be really useful in daily life - I’d really like to use them.

But no matter I sympathise with this project, it has some major flaws: lack of documentation, a sluggish GUI, perfectly valid tests that fail, integration issues. All those things sum converge into a situation that makes Gallio not really usable for me as it is now.

Nevertheless Gallio is very promising. I will definitively come back to it from time to time to see how it improves. To the Gallio developers: you’re doing the right thing, keep it up!

BTW: The Gallio people have established a merchandising shop, which I think is intended to support the project moneywise. I think it’s a cool way to support a project and still get something back. But please don’t forget to put the major part of you energy into the project itself, not the merchandising!

Posted in: Tools

Tags: , ,

Pingbacks and trackbacks (3)+