We are on a greenfield project and we already knew the possible cause of it 🙂
We implemented long-running operations as described in the Microsoft REST API guideline and we were aware that we didn't implement a retention policy for our API. Put it simply, we were indefinitely holding the results in memory for the entire duration of our application.
I started to spike a quick solution to it and with my surprise, the change didn't make much of a difference.
I violated the first rule of performance improvement: measure first!
The importance of measuring
In this blog post I shared more about the importance of defining performance goals and measuring before even starting to attempt a fix.
With the memory profiler, I identified that the majority of the memory was held inside MsBuild assemblies and with the performance profiler I identified the hot method causing the memory leak. The bottleneck was a method that was loading a MsBuild Project in memory and it was called a lot of times. We basically hit this issue on the msbuild repository.
My solution was building a cache of projects and correctly unloading projects when needed.
THAT was a massive 20X improvement in memory consuption and I am still working on it.
What's the lesson?
Never make assumptions when it comes to performance improvements and finding memory leaks. Use tools that can measure performance and can help to identify the real issue and avoid wasting time in optimizing a less significant part of the code.
Do you have an interesting experience to share on performance improvement?
Leave a comment below.