January 11

NBench Performance Testing – Code Throughput


A couple of weeks ago, Aaron Stannard (known for his work on akka.net) introduced a new automated .NET performance framework called NBench.

This little framework captured my attention immediately because I am not aware of frameworks that offer a unit testing like experience for performance testing.

If you know how to write a unit test you can learn NBench very quickly!

In this post, I am showing how to use NBench to measure the throughput for any user defined code.

Full source code is available here

Creating a performance test suite simply consists of creating a class.

You can have a setup method (decorated with the PerfSetup attribute) and a cleanup method (decorated with the PerfCleanup attribute) and every public method decorated with the PerfBenchmark attribute is a performance test. The life cycle is the same of Setup/Test/TearDown in unit tests.

As an example, let’s try to measure the throughput of adding a key value pair into a Dictionary.

There are two possible approaches.

First Approach

The first approach to create a throughput test is to use the RunMode Throughput and use the CounterThroughputAssertion attribute.

Dictionary Throughput Tests using NBench

NBench is going to call the test method multiple times, the test method execute the code under test and increments a NBench counter we created to keep track of the number of times we execute the Add operation. The CounterThroughputAssertion attribute assert that the we expect to be able to perform 20 millions Add operations per second on average. The test will fail otherwise.

Under the covers NBench executes 10 test iterations to collect multiple throughput values and calculate the average throughput. You can change this value using the NumberOfIterations property in the PerfBenchmark attribute.

Second Approch

The second approach consist of using the RunMode Iterations instead.

Measuring Dictionary Add Throughput using NBench

In this case, NBench will call the test method only once and it is up to you to call the Add operation multiple times in the body of the method. It seems to be a good practice to stop the loop when you reach your expected minimum throughput. All the rest of the code works in the same way. NBench still execute 10 test iterations to calculate the average throughput to be used in the assertion.

Running the tests

You can use the run.bat file to execute the tests. The bat file simply call the NBench runner against the assembly that contains your tests.

How to run the NBench test runner

You can see from the output that the two tests passed and that the actual average throughput in both cases was around 21 millions of operations per second.

Message from a passed NBench test Message from a passed NBench test

It’s is interesting to notice that the performance in the second approach is higher. This is because in the first approach NBench call the method multiple times using reflection while in the second approach the test method was called only once per test iteration. I tend to prefer the second approach because it provides a more accurate result even if it is a bit more verbose.

NBench runner creates a markdown file for each test where you can find detailed performance information.

Markdown results from NBench tests

You can find information for every test iteration, the average, the min and max throughput and the standard deviation.

Statistics from NBench test run

NBench offers the ability to execute assertions on memory usage and garbage collections and soon on windows performance counters. This would expand the capabilities of NBench significantly. I’ll keep an eye on NBench and we will explore other features in future posts. I think NBench is a very interesting framework and I hope to see it growing. I am looking forward to see ReSharper support and the ability to execute time based assertions.


You may also like

  • https://uploads.disquscdn.com/images/3fafc5b8f9f4fec7396721b6b09501d779cad9d2e3023e528c662d5feb3f61bc.png Hey Andrea,
    The approach RunMode Throughput has been failed on my end.
    Error msg show:

    * [FAIL] Expected [Counter] AddCounterAAAA to must be greater than 20,000,000.00 operations; actual value was 0.00 operations.

    NBench.NBenchException: Error occurred during $TestingCodeThroughput.DictionaryThroughputTests+AddThroughput_ThroughputMode RUN. —> System.OutOfMemoryException: Array dimensions exceeded supported range.
    at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
    at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
    at TestingCodeThroughput.DictionaryThroughputTests.AddThroughput_ThroughputMode(BenchmarkContext context) in d:tempdotnetalgorithms-masterNBenchTestingCodeThroughputTestingCodeThroughputDictionaryThroughputTests.cs:line 28
    at NBench.Sdk.ReflectionBenchmarkInvoker.c__DisplayClass10_0.b__0(BenchmarkContext ctx)
    at NBench.Sdk.Benchmark.WarmUp()
    — End of inner exception stack trace —

    Do you know why is that ?

    • Looks like is an OutOfMemory exception. Running this sample code requires a lot of memory so make sure you have enough memory available when you run it (2GB at least I would say) or reduce the AcceptableMinAddThroughput value. I tried to run it again and it works on my machine.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

    Subscribe to our newsletter now!

    Get instant access to the Master C# 9 webinar to learn all the new exciting C# 9 features quickly.
    Get regular videos and news on C# and .NET and offers on products and services to master C#.

    We will collect, use and protect your data in accordance with our Privacy Policy.