Using Gallio with NCover

March 17, 2009 at 11:39 PMAndre Loker

I’m really keen to use MbUnit 3, but I still have some obstacles with Gallio to overcome. I use NCover to check the coverage created by the unit tests in the assembly that is being tested – and only in that assembly. The whole procedure is part of an automatic build process executed with NAnt.

NAnt + NCover + MbUnit 2.4 = Happy Go Lucky!

For the combo I used before (using MbUnit 2.4) my build files contained something similar to this:

   1: <property name="unittest.cmdline" value='/ap:"${build.dir}" /rf:"${testresults.dir}" /rnf:"mbunit-${target.name}" /rt:Xml "${target.path}"'/>
   2: <property name="unittest.runner" value="${tools.dir}\MbUnit\mbunit.cons.exe"/>
   3:  
   4: <exec
   5:   program="${tools.dir}\NCover\ncover.console.exe"
   6:   workingdir="${build.dir}"
   7:   commandline='"${unittest.runner}" ${unittest.cmdline} //a "${coverage.assemblies}" //w "${build.dir}" //reg //x "${testresults.dir}\coverage-${target.name}.xml" //ea CoverageExcludeAttribute'
   8:   if="${ncover.supported}"        
   9: />

While this might look scary, it boils down to calling this:

ncover.console gallio.host.exe SomeAssembly.Tests.dll //a SomeAssembly

(plus a number of switches that I omitted for simplicity). This works perfectly and generates a nice coverage file for SomeAssembly based on the tests in SomeAssembly.Tests.dll – just what I want.

NAnt + NCover + MbUnit 2.4 Gallio = Does Not Compute

My first train of thought was: hey, with Gallio being more or less the successor of MbUnit, moving to Gallio shouldn’t be so hard, just replace MbUnit.Cons.exe with Gallio.Echo.exe, fiddle with the arguments and you’re done. So I changed the build file to do something along these lines:

ncover.console gallio.echo.exe SomeAssembly.Tests.dll //a SomeAssembly

And indeed, the tests were run and a nice shiny coverage file was generated – which however was empty. Erm? What now? I don’t know exactly what causes this behaviour, it seems that Gallio runs the tests in some separate app domain or so which prevents NCover from instrumenting it.

Second chance: let Gallio do it!

I remembered that there was a special switch to set the “runner” used by Gallio. It defaults to “IsolatedAppDomain”, but there’s also a “NCover” runner available. Someone gave me the hint to simply use this runner.

image

Maybe I only need to tell Gallio to use NCover and we’re happy? So I changed my build file to do something like:

gallio.echo.exe SomeAssembly.Tests.dll /r:NCover

And indeed, now it runs the tests and creates a non-empty coverage file. Sweet! A bit slowish, but running at least. The problem is – and you might have predicted it – that the coverage is gathered of all loaded assembly, including the assemblies of MbUnit and Gallio. So even for a trivial assembly being tested I get a coverage file of 5 MB. I couldn’t find a way to pass arguments regarding NCover. Argh! The problem is known by the way, but yet to be fixed.

Update: this problem is fixed in 3.0.6 builds. There you can pass arguments to NCover using the NCoverArguments and NCoverCoverageFile options.

At last: taming the beast

Believe me if tell you that at that point I was just annoyed. As of now all that Gallio gave me is headaches and frustration. This article was initially meant to reflect this.

Up to the point when I reconsidered the problem of the empty log file and remembered that there was a “Local” runner listed in the Gallio.Echo.exe help. Maybe this would force Gallio to run the tests in a way such that NCover can do its magic? Thus, I tried this (note the /r switch):

ncover.console gallio.echo.exe SomeAssembly.Tests.dll /r:Local //a SomeAssembly

Let’s see what we get:

  • Tests are found and executed? Check.
  • The coverage file is non-empty? Check.
  • The coverage file only contains information about “SomeAssembly”? Check.

Could it be? Finally, after several disappointments I am finally able to integrate Gallio in my automatic builds! Well, I have not integrated the Gallio reports fully in my CruiseControl.NET web dashboard, but I’ll keep that for another day. As for today, I’m happy with what I achieved.

Update:

Gallio also provides a NAnt task which you can use instead of <exec>’ing. Starting from version 3.0.6 you can also pass arguments to NCover (as described above). I tested the following task in NAnt and it worked smoothly:

   1: <gallio result-property="testrunner.exit-code" 
   2:         application-base-directory="${build.dir}"
   3:         runner-type="NCover" 
   4:         failonerror="false" 
   5:         report-name-format="gallio-${target.name}" 
   6:         report-types="xml" 
   7:         report-directory="${testresults.dir}">
   8:   <runner-property value="NCoverArguments='//w ${build.dir} //q //ea CoverageExcludeAttribute //a ${coverage.assemblies}'" />
   9:   <runner-property value="NCoverCoverageFile='${testresults.dir}\coverage-${target.name}.xml'" />
  10:   <assemblies>
  11:     <include name="${target.path}" />
  12:   </assemblies>
  13: </gallio>
  14:  
  15: <fail if="${testrunner.exit-code != '0'}" >The return code should have been 0!</fail>

Thanks to Bruno Wouters and Jeff Brown for the hint!

 

Conclusion

To condense the story into two sentences:

  1. If you want to use Gallio with NCover be sure to choose the “Local” runner by setting the /r:Local option.
  2. Don’t use the NCover runner if you need any control over NCover options. Update: or use version 3.0.6+

To the Gallio people

I hope this article isn’t utterly stupid or useless. I looked on the web but did not came across anything useful regarding NCover and Gallio. Also, I couldn’t find any hint about it in the documentation. I hope this article helps people that like me would really like to use Gallio but struggle with issues.

Update: thanks for your help!

Posted in: Tools

Tags: , , ,

Comments (14) -

Bruno Wouters
Belgium Bruno Wouters says:

Hi Andre,

I also use Gallio and NCover to run tests. Below you can see the NAnt file I use. The runner-extension can be removed if you don’t use TeamCity. I like this kind of setup better than invoking command line applications. The runner-property will be available from 3.0.6. You can download fresh builds from http://ccnet.gallio.org/Distributables/. Not sure when the stable will be released.


<loadtasks assembly="${ToolsPath}\Gallio\bin\Gallio.NAntTasks.dll" />
  <target name="test" depends="build">
    <echo message="Started testing ${AssemblyToTest}"/>
    <gallio result-property="exitCode" runner-type="${RunnerType}" failonerror="false" report-name-format="Tests" report-types="html" report-directory="${ReportsPath}">
        <runner-property value="NCoverArguments='//l ${WorkingDirectory}\Coverage.log //ea CoverageExcludeAttribute //a ${TestedAssemblyName}'" />
        <runner-property value="NCoverCoverageFile='${WorkingDirectory}\Coverage.xml'" />
      <runner-extension value="TeamCityExtension,Gallio.TeamCityIntegration" />
      <assemblies>  
          <include name="${AssemblyToTest}" />
      </assemblies>
    </gallio>
    <fail if="${exitCode != '0'}" >One or more tests failed. Please check the log for more details</fail>    
  </target>

  <loadtasks assembly="${ToolsPath}\NCoverExplorer.Extras\bin\NCoverExplorer.NAntTasks.dll" />
  <target name="gencoverreport" depends="test">
    <echo message="Generating coverage report for ${TestAssemblyDll}"/>
    <ncoverexplorer
      program="${ToolsPath}\NCoverExplorer\NCoverExplorer.Console.exe"
      outputDir="${ReportsPath}"
      satisfactoryCoverage="90"
      minimumCoverage="90"
      reportType="4"
      htmlReportName="CoverageSummary.html"
      failMinimum="True"
      showExcluded="False">
      <fileset>
          <include name="${WorkingDirectory}\Coverage.xml" />
      </fileset>
      <moduleThresholds>
          <moduleThreshold moduleName="${TestedAssemblyDll}" satisfactoryCoverage="${SatisfactoryCoverage}" />
      </moduleThresholds>
     </ncoverexplorer>
  </target>


Greets,

Bruno Wouters
Belgium Bruno Wouters says:

I forgot to replace the runnertype variable, runner type should be NCover.
${RunnerType} = NCover

other variables will probably explain them self.

Thanks for your feedback. I used the <exec> approach to switch between unit testing frameworks more easily, but I'll have a look at the 3.0.6 builds and the gallio-tasks.

Using Gallio with NCover has been a bit tricky.  On the one hand, you can pass the runner="NCover" argument to the NAnt task, however prior to v3.0.6 you could not then also pass in command-line arguments for NCover.

Now you can... The syntax looks like this:

<gallio runner-type="NCover">
  <assemblies>
    <include name="MyTestAssembly.dll" />
  </assemblies>
  <runner-property value="NCoverArguments='//a MyTestAssembly;MyCode'" />
</gallio>


However beginning with v3.0.6

The point about documentation is well-taken.  We do need some assistance writing out more docs.  However, many people seem to find these blog posts very useful.  Thanks for writing it!

There's actually a bunch of NAnt / MSBuild docs in there.  It's just a bit hard to find at the moment.  So I've just updated http://www.gallio.org/Docs.aspx to provide some helpful shortcuts and tips for these common questions!  Smile

P.S.  v3.0.6 will probably be released tomorrow assuming I don't hit any show-stoppers.  You can also grab a recent build from http://ccnet.gallio.org/Distributables/

Oops, left a dangling sentence there...

Beginning with v3.0.6 we support passing arguments to NCover, as shown.

You can also turn it around and call echo from NCover

japikse.blogspot.com/.../...ver3-quick-review.html

That's what I'm doing, isn't it?

You need to use the //pm option with gallio.echo.exe. This will provide the coverage results on the tests and not gallio. Smile

Joe Feser
NCover

The full command line would be:

ncover.console.exe gallio.echo.exe //pm gallio.host.exe SomeAssembly.Tests.dll

//pm gallio.host.exe tells ncover to profile the gallio.host process.
SomeAssembly.Tests.dll is a parameter for gallio.

Hope this helps.

Joe

As in
ncover.console gallio.echo.exe SomeAssembly.Tests.dll //a SomeAssembly //pm gallio.echo.exe?

This executes the tests but then does not continue as if it is waiting for gallio to finish or so.

I also tried
ncover.console gallio.echo.exe SomeAssembly.Tests.dll //a SomeAssembly //pm gallio.host.exe
But this also freezes.

What advantage does //pm have over //a in this context? With Gallio's /r:Local option I can use //a and it works.

I tried
NCover.Console.exe Gallio.Echo.exe //pm Gallio.Host.exe SomeAssembly.Tests.dll
as described in a forum post (http://www.ncover.com/forum/show_topic/910)

But this also provides coverage for the Gallio code.

Andre,

//pm is an NCover 2/3 feature and not 1.5.8 so it would not work for you.

You should really look at adding your article to the documentation of Gallio.  I downloaded the code from subversion and they have a documentation section that you could add what you have learned.

Curious do you know how to achieve the same results using MsBuild.  I started out writing our build scripts using Nant but have since moved to MsBuild and have not yet implemented Gallio but will be doing so shortly.  Any pointers?

I was having problems with your /r:local solution.  Gallio was not loading the correct app.config.  Here is the response I got from the Gallio people...

"That said, your problem is in the way you are using NCover.  You can either
use /r:IsolatedAppDomain and wrap Gallio.Echo.exe with a call to
NCover.Console.exe, or you can use /r:NCover and get built-in support for
NCover."


"Don't use /r:Local though.  Basically this prevents Gallio from creating a
new AppDomain which means your app.config settings will not be available. "

Switching to /r:IsolatedAppDomain fixed the problem.

Sean

Pingbacks and trackbacks (4)+