Debugging speed of Gallio fixtures – reloaded

March 25, 2009 at 10:19 PMAndre Loker

Last week I mentioned the slow debugging of Gallio unit tests. After having published that blog post I had a conversation with Jeff Brown about this issue. After showing him in a video how huge the difference in speed between debugging an MbUnit 2.4 and an MbUnit 3 test was he pointed out that this might have to do with the fact that MbUnit 2.4 uses the multi threaded apartment state (MTA) per default, while MbUnit 3 uses single threaded apartment state (STA).

I tested this difference by debugging the test fixture shown below:

   1: [TestFixture]
   2: public class CalculatorTests {
   3:  
   4:   [Test, ApartmentState(ApartmentState.STA)]
   5:   public void Adding2And3ShouldReturn5_STA() {
   6:     TheTest();
   7:   }
   8:  
   9:   [Test, ApartmentState(ApartmentState.MTA)]
  10:   public void Adding2And3ShouldReturn5_MTA() {
  11:     TheTest();
  12:   }
  13:  
  14:   void TheTest() {
  15:     var x = 2;
  16:     var y = 3;
  17:     var expected = x + y;
  18:     var sut = new Calculator();
  19:     var actual = sut.Add(x, y);
  20:     Assert.AreEqual(expected, actual);
  21:     MessageBox.Show("END OF TEST!");
  22:   }
  23: }

And indeed: the STA unit test ran sluggish as reported while the MTA based unit test could be debugged at virtually the same speed as I am used to from MbUnit 2.4 tests.

I reported that back to Jeff and he proposed to change the default thread apartment state to MTA with the option to override this default using an assembly scoped attribute (for unit tests that require STA).

Thumbs up to Jeff!

Update:

Jeff reported that after some further research the solution to the problem might not be as simple as this and he needs to think it over. Anyway, I’m glad to see that this issue is being worked on.

Posted in: Tools

Tags: ,

Slow debugging of MbUnit tests with Gallio and TD.NET

March 18, 2009 at 12:04 PMAndre Loker

No, this is not another rant about the troubles I’m having migrating to Gallio, so don’t be alarmed :-) In fact, the Gallio people have proven to be really bothered about the issues I have (see previous post and especially the comments). Thanks a lot for your commitment!

Sometimes I need to debug unit tests to see why they fail. TestDriven.NET has a nice feature that allows just that.

image

This has always worked perfectly smooth with MbUnit 2.4 when using Gallio (3.0.6.727) with MbUnit the execution during debugging was very sluggish. This problem has been reported in the mailing list.

One workaround was posted on the thread mentioned above: disable the “property evaluation and other implicit function calls” option in the Visual Studio debugging options.

image

Doing this reverted the debugging experience to normal speed. The only drawback is that you need to “manually” refresh the values of properties by clicking on the two green arrows on the right side.

image

This is not the most convenient thing in the world, but I hope that it’s just a temporary state of affairs that the Gallio folks are able to fix soon.

Posted in: Tools

Tags: , ,

Using Gallio with NCover

March 17, 2009 at 11:39 PMAndre Loker

I’m really keen to use MbUnit 3, but I still have some obstacles with Gallio to overcome. I use NCover to check the coverage created by the unit tests in the assembly that is being tested – and only in that assembly. The whole procedure is part of an automatic build process executed with NAnt.

NAnt + NCover + MbUnit 2.4 = Happy Go Lucky!

For the combo I used before (using MbUnit 2.4) my build files contained something similar to this:

   1: <property name="unittest.cmdline" value='/ap:"${build.dir}" /rf:"${testresults.dir}" /rnf:"mbunit-${target.name}" /rt:Xml "${target.path}"'/>
   2: <property name="unittest.runner" value="${tools.dir}\MbUnit\mbunit.cons.exe"/>
   3:  
   4: <exec
   5:   program="${tools.dir}\NCover\ncover.console.exe"
   6:   workingdir="${build.dir}"
   7:   commandline='"${unittest.runner}" ${unittest.cmdline} //a "${coverage.assemblies}" //w "${build.dir}" //reg //x "${testresults.dir}\coverage-${target.name}.xml" //ea CoverageExcludeAttribute'
   8:   if="${ncover.supported}"        
   9: />

While this might look scary, it boils down to calling this:

ncover.console gallio.host.exe SomeAssembly.Tests.dll //a SomeAssembly

(plus a number of switches that I omitted for simplicity). This works perfectly and generates a nice coverage file for SomeAssembly based on the tests in SomeAssembly.Tests.dll – just what I want.

NAnt + NCover + MbUnit 2.4 Gallio = Does Not Compute

My first train of thought was: hey, with Gallio being more or less the successor of MbUnit, moving to Gallio shouldn’t be so hard, just replace MbUnit.Cons.exe with Gallio.Echo.exe, fiddle with the arguments and you’re done. So I changed the build file to do something along these lines:

ncover.console gallio.echo.exe SomeAssembly.Tests.dll //a SomeAssembly

And indeed, the tests were run and a nice shiny coverage file was generated – which however was empty. Erm? What now? I don’t know exactly what causes this behaviour, it seems that Gallio runs the tests in some separate app domain or so which prevents NCover from instrumenting it.

Second chance: let Gallio do it!

I remembered that there was a special switch to set the “runner” used by Gallio. It defaults to “IsolatedAppDomain”, but there’s also a “NCover” runner available. Someone gave me the hint to simply use this runner.

image

Maybe I only need to tell Gallio to use NCover and we’re happy? So I changed my build file to do something like:

gallio.echo.exe SomeAssembly.Tests.dll /r:NCover

And indeed, now it runs the tests and creates a non-empty coverage file. Sweet! A bit slowish, but running at least. The problem is – and you might have predicted it – that the coverage is gathered of all loaded assembly, including the assemblies of MbUnit and Gallio. So even for a trivial assembly being tested I get a coverage file of 5 MB. I couldn’t find a way to pass arguments regarding NCover. Argh! The problem is known by the way, but yet to be fixed.

Update: this problem is fixed in 3.0.6 builds. There you can pass arguments to NCover using the NCoverArguments and NCoverCoverageFile options.

At last: taming the beast

Believe me if tell you that at that point I was just annoyed. As of now all that Gallio gave me is headaches and frustration. This article was initially meant to reflect this.

Up to the point when I reconsidered the problem of the empty log file and remembered that there was a “Local” runner listed in the Gallio.Echo.exe help. Maybe this would force Gallio to run the tests in a way such that NCover can do its magic? Thus, I tried this (note the /r switch):

ncover.console gallio.echo.exe SomeAssembly.Tests.dll /r:Local //a SomeAssembly

Let’s see what we get:

  • Tests are found and executed? Check.
  • The coverage file is non-empty? Check.
  • The coverage file only contains information about “SomeAssembly”? Check.

Could it be? Finally, after several disappointments I am finally able to integrate Gallio in my automatic builds! Well, I have not integrated the Gallio reports fully in my CruiseControl.NET web dashboard, but I’ll keep that for another day. As for today, I’m happy with what I achieved.

Update:

Gallio also provides a NAnt task which you can use instead of <exec>’ing. Starting from version 3.0.6 you can also pass arguments to NCover (as described above). I tested the following task in NAnt and it worked smoothly:

   1: <gallio result-property="testrunner.exit-code" 
   2:         application-base-directory="${build.dir}"
   3:         runner-type="NCover" 
   4:         failonerror="false" 
   5:         report-name-format="gallio-${target.name}" 
   6:         report-types="xml" 
   7:         report-directory="${testresults.dir}">
   8:   <runner-property value="NCoverArguments='//w ${build.dir} //q //ea CoverageExcludeAttribute //a ${coverage.assemblies}'" />
   9:   <runner-property value="NCoverCoverageFile='${testresults.dir}\coverage-${target.name}.xml'" />
  10:   <assemblies>
  11:     <include name="${target.path}" />
  12:   </assemblies>
  13: </gallio>
  14:  
  15: <fail if="${testrunner.exit-code != '0'}" >The return code should have been 0!</fail>

Thanks to Bruno Wouters and Jeff Brown for the hint!

 

Conclusion

To condense the story into two sentences:

  1. If you want to use Gallio with NCover be sure to choose the “Local” runner by setting the /r:Local option.
  2. Don’t use the NCover runner if you need any control over NCover options. Update: or use version 3.0.6+

To the Gallio people

I hope this article isn’t utterly stupid or useless. I looked on the web but did not came across anything useful regarding NCover and Gallio. Also, I couldn’t find any hint about it in the documentation. I hope this article helps people that like me would really like to use Gallio but struggle with issues.

Update: thanks for your help!

Posted in: Tools

Tags: , , ,

First attempts with Gallio

February 14, 2009 at 2:21 PMAndre Loker

From time to time I like to check my development workflow and tools to see whether I’m still up to date. This time I thought it might be a nice idea to give Gallio a shot. If you don’t now Gallio: it is a platform to uniformly run unit tests from whatever unit test framework (MbUnit, NUnit, MSTest etc.) you use. At least that’s how I understand Gallio’s mission. It also integrates into a bunch of other tools which I use (CruiseControl.NET, NAnt, NCover, ReSharper et al).

So basically it sounds like a cool idea. On top of it, version 3 of MbUnit – my favourite unit testing framework – is available only bundled with Gallio, that's why decided to give it try. Until I only used MbUnit 2.4.

And here’s a list of twitter-esque thoughts about Gallio that came up during my first 60 minutes of use.

  • Installation is no problem
  • Migrating from MbUnit 2.4 to MbUnit 3 didn’t take much effort, at least for the small project I chose for my Gallio evaluation. Mainly I had to replace the RowTest attribute with a conventional Test attribute, because row tests have been consolidated into the normal tests. But you can also use the MbUnit.Compatibility Assembly and don’t change the existing tests at all. Big plus!
  • The documentation is rather incomplete. There are still a lot of TODOs. Especially a migration guide for MbUnit 2 users would be really useful.
  • Integration into Testdriven.NET works well, no problems here…
  • … although the test runner feels a little less snappy than MbUnit 2.4, startup time is slightly higher. The 190 unit tests in the small test project took about 5,2 seconds using MbUnit 2.4 and 7 seconds using MbUnit. I’ve yet to test how this scales with more tests.
  • ReSharper integration seems to be incomplete and instable. If I use MbUnit 2.4 tests, not all tests are run. Only about 30 of my tests appeared in the unit test browser. Using MbUnit 3, all tests were found, but occasionally ReSharper through exceptions when I selected failed tests.
  • Using Testdriven.NET’s “Test with”=>”Gallio Icarus” context menu action fires up Icarus (the Gallio test running GUI) but doesn’t load the assembly of the project that I want to test.
  • Running tests in Icarus is unbearably slow. The tests took more than 40 seconds to run (compare that to the 7 seconds using Testdriven.NET or even 5 seconds using MbUnit 2.4 + Testdriven.NET).
    image
    I guess this is due to the fact that the result window is refreshed after every single unit tests. Thus, the performance even degrades with every additional unit tests. I wonder what happened if I were to run tests of my bigger projects.
  • The one real showstopper however is the fact that suddenly unit tests begin to fail that have been perfectly valid before. This happens when I use Rhino.Mocks to create a partial mock and expect for an ArgumentNullException to be thrown in the ctor of the partially mocked type. Luckily this is easily reproducible:

Class under test:

   1: public abstract class SomeClass {
   2:   // Throws ArgumentNullException if value is null
   3:   protected SomeClass(string value) {
   4:     if (value == null) {
   5:       throw new ArgumentNullException("value");
   6:     }
   7:   }
   8: }

The unit test:

   1: [Test, ExpectedArgumentNullException]
   2: public void CtorShouldThrowIfArgumentIsNull() {
   3:   var mocks = new MockRepository();
   4:   mocks.PartialMock<SomeClass>(default(string));
   5: }

Results when run under MbUnit 2.4

image

Results when run under MbUnit 3

image

A System.ArgumentNullException is expected, but instead a System.ArgumentNullException has been thrown. Possibly it’s a framework version conflict, I don’t know. Anyway, a bug report is on its way (uodate 02/20/2009: the bug report has been closed as won’t fix).

My personal conclusion for now

I really appreciate the idea behind Gallio to unify unit testing. Different projects use different unit testing frameworks. Especially in connection with automated builds it could be useful to have one test runner that supports them all. And of course, I really like that they have chosen MbUnit as their primary unit testing framework, because I think MbUnit rocks and is superior to the rather basic NUnit. I haven’t looked much into the version 3 API of MbUnit, but I think there are some nifty new features that can be really useful in daily life - I’d really like to use them.

But no matter I sympathise with this project, it has some major flaws: lack of documentation, a sluggish GUI, perfectly valid tests that fail, integration issues. All those things sum converge into a situation that makes Gallio not really usable for me as it is now.

Nevertheless Gallio is very promising. I will definitively come back to it from time to time to see how it improves. To the Gallio developers: you’re doing the right thing, keep it up!

BTW: The Gallio people have established a merchandising shop, which I think is intended to support the project moneywise. I think it’s a cool way to support a project and still get something back. But please don’t forget to put the major part of you energy into the project itself, not the merchandising!

Posted in: Tools

Tags: , ,

Kleine Toolkunde (I)

February 11, 2008 at 11:58 PMAndre Loker

Es mag ja durchaus Entwickler geben, die mit Notepad und csc.exe ihre Erfüllung finden. Ich gehöre definitiv nicht dazu und bin bekennender IDE Liebhaber. Neben den Standard-Features die Visual Studio so bietet, gibt es da draußen noch eine ganze Reihe kostenloser und -pflichtiger Add-Ins, die dem Entwickler das Leben noch einfacher machen. Oder sagen wir: weniger schwer.

Hier also eine kleine Liste von must-have Add-Ins für Visual Studio:

  • Resharper – ohne geht nicht, sorry. Wenn man sich einmal in das Tool eingearbeitet hat, ist ein Arbeiten ohne diesen treuen Begleiter eine Qual. Viel sagen muss man zu diesem vielseitigen Tool wohl nicht.
  • VisualSVN – eine vernünftige Subversion-Einbindung. Anders als das relativ instabile AnkhSVN macht sich VisualSVN das Leben viel leichter und verrichtet seine Arbeitet sozusagen huckepack auf dem Rücken der bekannten SVN Shell-Extension TortoiseSVN. Das Add-In kostet rund 37€, ist aber jeden Cent wert, weil es eine sehr löbliche Eigenschaft hat: man merkt nicht, dass es da ist. Stattdessen macht es einfach das, was es soll, ohne den Nutzer zu stören. Empfehlenswert ist auch das Repository-Verwaltungstool VisualSVN Server, welches gratis (!) zu downloaden ist.
  • VersioningControlledBuild – Versionsnummern im Griff. Wenn man nicht gerade einen Buildserver verwendet, ist dieses Add-In sehr nützlich, da es bei jedem Buildvorgang die Versionsnummern der Assemblies erhöht. Deutliche Versionsnummern machen die Kommunikation bei der Fehlersuche erheblich einfacher!
  • GhostDoc – XML-Dokumentation von Geisterhand. Sein wir ehrlich - es ist schon manchmal lästig, Code zu kommentieren. GhostDoc nimmt dem Programmierer insofern schon ein großen Teil der Arbeit ab, indem es XML-Dokumentations-Gründgerüste erstellt und so gut wie möglich probiert auszufüllen. Dabei schaut GhostDoc nach dem Namen des entsprechenden Members und nach bekannten Mustern (z.B. Eventhandlern). Man sollte nur vor lauter Automatismus nicht vergessen, diejenigen Informationen nachzutragen, die eben nicht automatisch abgeleitet werden können…
  • TestDriven.NET - Unit Tests aus der IDE heraus starten. So wird man beim TDD nicht immer dadurch gestört, zwischen Testrunner GUI und IDE wechseln zu müssen. TestDriven.NET unterstützt unter anderem MbUnit, mein persönlicher Favorit unter den Unit Test Frameworks.
  • Web Deployment Project - ein Tool von Microsoft, um Webseiten (Web Sites) und Webanwendungen (Web Applications) vorzukompilieren und zu deployen. Unter anderem erlaubt das Add-In diverse Varianten, um mehrere (oder alle) Pages und Controls zu einer einzelnen Assembly zu kompilieren. Dabei können auch Control-Bibliotheken generiert werden, wodurch die enthaltenen User Controls in anderen Projekten wie Custom Controls verwendet werden können. Code re-use, ik hör dir trapsen!

Posted in: Tools

Tags: , , , , , , ,