NBench Performance Testing – Code Throughput

A couple of weeks ago, Aaron Stannard (known for his work on akka.net) introduced a new automated .NET performance framework called NBench.

This little framework captured my attention immediately because I am not aware of frameworks that offer a unit testing like experience for performance testing.

If you know how to write a unit test you can learn NBench very quickly!

In this post, I am showing how to use NBench to measure the throughput for any user defined code.

Full source code is available here

Creating a performance test suite simply consists of creating a class.

You can have a setup method (decorated with the PerfSetup attribute) and a cleanup method (decorated with the PerfCleanup attribute) and every public method decorated with the PerfBenchmark attribute is a performance test. The life cycle is the same of Setup/Test/TearDown in unit tests.

As an example, let’s try to measure the throughput of adding a key value pair into a Dictionary.

There are two possible approaches.

First Approach

The first approach to create a throughput test is to use the RunMode Throughput and use the CounterThroughputAssertion attribute.

NBench is going to call the test method multiple times, the test method execute the code under test and increments a NBench counter we created to keep track of the number of times we execute the Add operation. The CounterThroughputAssertion attribute assert that the we expect to be able to perform 20 millions Add operations per second on average. The test will fail otherwise.

Under the covers NBench executes 10 test iterations to collect multiple throughput values and calculate the average throughput. You can change this value using the NumberOfIterations property in the PerfBenchmark attribute.

Second Approch

The second approach consist of using the RunMode Iterations instead.

In this case, NBench will call the test method only once and it is up to you to call the Add operation multiple times in the body of the method. It seems to be a good practice to stop the loop when you reach your expected minimum throughput. All the rest of the code works in the same way. NBench still execute 10 test iterations to calculate the average throughput to be used in the assertion.

Running the tests

You can use the run.bat file to execute the tests. The bat file simply call the NBench runner against the assembly that contains your tests.

You can see from the output that the two tests passed and that the actual average throughput in both cases was around 21 millions of operations per second.

It’s is interesting to notice that the performance in the second approach is higher. This is because in the first approach NBench call the method multiple times using reflection while in the second approach the test method was called only once per test iteration. I tend to prefer the second approach because it provides a more accurate result even if it is a bit more verbose.

NBench runner creates a markdown file for each test where you can find detailed performance information.

You can find information for every test iteration, the average, the min and max throughput and the standard deviation.

NBench offers the ability to execute assertions on memory usage and garbage collections and soon on windows performance counters. This would expand the capabilities of NBench significantly. I’ll keep an eye on NBench and we will explore other features in future posts. I think NBench is a very interesting framework and I hope to see it growing. I am looking forward to see ReSharper support and the ability to execute time based assertions.