You can run your NBench performance tests using NUnit and ReSharper.
All the logic to run performance tests in NBench is actually embedded in the core library and for this reason an external runner is not strictly required. You can easily integrate your unit testing framework and use your favorite tools to run your performance tests.
In this post, I show you how you can run NBench tests with NUnit and ReSharper.
Full source code is available here
How to create your NBench performance tests
You can create performance tests using attributes as usual.
The only difference is to inherit from a new base class.
The PerformanceTestSuite base class
The abstract class PerformanceTestStuite<T> defines a parametric test in NUnit that run a single performance test (benchmark). The TestCaseSource attribute point to a method that returns all the performance tests so this single NUnit test basically runs all the performance tests defined in the type T.
The Benchmarks method use an instance of the ReflectionDiscovery class to find all the benchmarks and then return a TestCaseData for each benchmark found.
The discovery class takes an IBenchmarkOutput instance that is used by the benchmark.Finish() method to notify the result of the performance test. For each benchmark NBench tells us all the assertion results and we can use those information to run a NUnit assert and print useful information to the console.
Note: the reason why I had to use generics is because NUnit requires the TestCaseSource method to be static and in a static method I don’t have a way to get type information.
Running the tests in ReSharper
This is how running the tests looks like in the ReSharper test runner.
For a failing test, you get a clear error message.
For a passing test, you still get information about the actual results of your performance tests.
Conclusions
I think this is great! You can add this base class into your own project and very easily get your team to start writing automatic performance tests. Of course, the biggest problem is to define what to test but it’s great that now we have some tools that helps us to do it so easily. The learning curve is so slow.
NBench vision is to become a complete, user-friendly and fully extensible framework for automatic performance testing in .NET and more importantly, to make automatic performance testing a regular software development practice. I think this framework is a very good step forward in this direction.
NBench is an open source project. If you like the vision, you can help. Contribution is very welcome. The github page of the project is https://github.com/petabridge/NBench.
Previous posts
- NBench Performance Testing – Code Throughput
- NBench Testing – Memory Allocations
- NBench Testing – Garbage Collection
Any chance of xUnit support too?
It would be most useful to have an NBench.NUnit and NBench.xUnit NuGet package with this built into it.
You can easily adapt the super class to use xUnit as well. I can create a blog post about it. I created this class so it’s not part of NBench yet and there are many ways it can be improved. For example at the moment is picking measurements as well even if they probably should not. It’s a kind of an hack to integrate NBench with a unit test framework to proof that it is possible today. The idea of having separate package for NUnit or xUnit is not too bad, you can suggest it in the github project for consideration. Thanks
Fyi, the source code link points at “TestingGarbageCollection”, not at “TestingWithNunit” (https://github.com/angellaa/dotnetalgorithms/tree/master/NBench/TestingWithNUnit )
Thanks a lot. I fixed it.
[…] we take a look at NBench Performance Testing – NUnit and ReSharper Integration we can see how to extend our test capabilities using NUnit to run our extensions. i.e. with NBench […]
Is it possible to get the full report to output to the Resharper result window? (i.e. markdown report)
This gives me what I want for now :
var discovery = new ReflectionDiscovery(new CompositeBenchmarkOutput(consoleOutput));
[…] Obviously we run a build/continuous integration server, so we want to integrate these tests into our builds. Now it may be that you can use the NBench runner within your build, but (as mentioned in a previous post) there’s already a way to achieve this integration with the likes on NUnit (see NBench Performance Testing – NUnit and ReSharper Integration). […]
[…] a previous post) there’s already a way to achieve this integration with the likes on NUnit (see NBench Performance Testing – NUnit and ReSharper Integration […]