Andre Loker .NET wisdom

I’m really keen to use MbUnit 3, but I still have some obstacles with Gallio to overcome. I use NCover to check the coverage created by the unit tests in the assembly that is being tested – and only in that assembly. The whole procedure is part of an automatic build process executed with NAnt.

NAnt + NCover + MbUnit 2.4 = Happy Go Lucky!

For the combo I used before (using MbUnit 2.4) my build files contained something similar to this:

   1: <property name="unittest.cmdline" value='/ap:"${build.dir}" /rf:"${testresults.dir}" /rnf:"mbunit-${target.name}" /rt:Xml "${target.path}"'/>

   2: <property name="unittest.runner" value="${tools.dir}\MbUnit\mbunit.cons.exe"/>

   3:  

   4: <exec

   5:   program="${tools.dir}\NCover\ncover.console.exe"

   6:   workingdir="${build.dir}"

   7:   commandline='"${unittest.runner}" ${unittest.cmdline} //a "${coverage.assemblies}" //w "${build.dir}" //reg //x "${testresults.dir}\coverage-${target.name}.xml" //ea CoverageExcludeAttribute'

   8:   if="${ncover.supported}"        

   9: />

While this might look scary, it boils down to calling this:

ncover.console gallio.host.exe SomeAssembly.Tests.dll //a SomeAssembly

(plus a number of switches that I omitted for simplicity). This works perfectly and generates a nice coverage file for SomeAssembly based on the tests in SomeAssembly.Tests.dll – just what I want.

NAnt + NCover + MbUnit 2.4 Gallio = Does Not Compute

My first train of thought was: hey, with Gallio being more or less the successor of MbUnit, moving to Gallio shouldn’t be so hard, just replace MbUnit.Cons.exe with Gallio.Echo.exe, fiddle with the arguments and you’re done. So I changed the build file to do something along these lines:

ncover.console gallio.echo.exe SomeAssembly.Tests.dll //a SomeAssembly

And indeed, the tests were run and a nice shiny coverage file was generated – which however was empty. Erm? What now? I don’t know exactly what causes this behaviour, it seems that Gallio runs the tests in some separate app domain or so which prevents NCover from instrumenting it.

Second chance: let Gallio do it!

I remembered that there was a special switch to set the “runner” used by Gallio. It defaults to “IsolatedAppDomain”, but there’s also a “NCover” runner available. Someone gave me the hint to simply use this runner.

Maybe I only need to tell Gallio to use NCover and we’re happy? So I changed my build file to do something like:

gallio.echo.exe SomeAssembly.Tests.dll /r:NCover

And indeed, now it runs the tests and creates a non-empty coverage file. Sweet! A bit slowish, but running at least. The problem is – and you might have predicted it – that the coverage is gathered of all loaded assembly, including the assemblies of MbUnit and Gallio. So even for a trivial assembly being tested I get a coverage file of 5 MB. I couldn’t find a way to pass arguments regarding NCover. Argh! The problem is known by the way, but yet to be fixed.

Update: this problem is fixed in 3.0.6 builds. There you can pass arguments to NCover using the NCoverArguments and NCoverCoverageFile options.

At last: taming the beast

Believe me if tell you that at that point I was just annoyed. As of now all that Gallio gave me is headaches and frustration. This article was initially meant to reflect this.

Up to the point when I reconsidered the problem of the empty log file and remembered that there was a “Local” runner listed in the Gallio.Echo.exe help. Maybe this would force Gallio to run the tests in a way such that NCover can do its magic? Thus, I tried this (note the /r switch):

ncover.console gallio.echo.exe SomeAssembly.Tests.dll /r:Local //a SomeAssembly

Let’s see what we get:

• Tests are found and executed? Check.
• The coverage file is non-empty? Check.
• The coverage file only contains information about “SomeAssembly”? Check.

Could it be? Finally, after several disappointments I am finally able to integrate Gallio in my automatic builds! Well, I have not integrated the Gallio reports fully in my CruiseControl.NET web dashboard, but I’ll keep that for another day. As for today, I’m happy with what I achieved.

Update:

Gallio also provides a NAnt task which you can use instead of <exec>’ing. Starting from version 3.0.6 you can also pass arguments to NCover (as described above). I tested the following task in NAnt and it worked smoothly:

   1: <gallio result-property="testrunner.exit-code" 

   2:         application-base-directory="${build.dir}"

   3:         runner-type="NCover" 

   4:         failonerror="false" 

   5:         report-name-format="gallio-${target.name}" 

   6:         report-types="xml" 

   7:         report-directory="${testresults.dir}">

   8:   <runner-property value="NCoverArguments='//w ${build.dir} //q //ea CoverageExcludeAttribute //a ${coverage.assemblies}'" />

   9:   <runner-property value="NCoverCoverageFile='${testresults.dir}\coverage-${target.name}.xml'" />

  10:   <assemblies>

  11:     <include name="${target.path}" />

  12:   </assemblies>

  13: </gallio>

  14:  

  15: <fail if="${testrunner.exit-code != '0'}" >The return code should have been 0!</fail>

Thanks to Bruno Wouters and Jeff Brown for the hint!

Conclusion

To condense the story into two sentences:

1. If you want to use Gallio with NCover be sure to choose the “Local” runner by setting the /r:Local option.
2. Don’t use the NCover runner if you need any control over NCover options. Update: or use version 3.0.6+

To the Gallio people

I hope this article isn’t utterly stupid or useless. I looked on the web but did not came across anything useful regarding NCover and Gallio. Also, I couldn’t find any hint about it in the documentation. I hope this article helps people that like me would really like to use Gallio but struggle with issues.

Update: thanks for your help!