MsTest and NUnit are the two most popular unit testing frameworks in the .NET ecosystem. I always get questions from my students asking about which should be used and why. Let’s talk about the advantages and disadvantages of each solution.
Parallelization
Parallelization is one of the most critical components when it comes to picking a unit testing library. A unit testing library that has poor parallelization is automatically at a massive disadvantage because it will slow down your continuous integration pipeline.
Parallelization with MsTest x NUnit
Parallelization at the Class aka Fixture level with NUnit is just fine. The problem arises when you want to parallelize at the method aka test method level.
How to parallelize methods with NUnit
The problem is that class level fields are not natively thread safe. As a result, NUnit expects the end-user to make sure that our code is thread-safe and will not handle that for us.
Here is what you have to do in order to have your NUnit test methods run in parallel. This article also does an excellent job at explaining.
The key here is that you cannot reuse the SetUp and Teardown from NUnit and have to write your own.
The SetUp needs to be part of the TestScope of the IDisposable. The Teardown needs to happen in the Dispose().
This may be acceptable to you.
The question is do you want to do the extra work to manage thread-safety of your tests?
Execution speed
MsTest is a native unit testing library that comes with Visual Studio from Microsoft. NUnit is an extra Nuget package that needs to be installed on top and interact with the APIs of Visual Studio. Nunit likely doesn’t have direct integration into the native APIs like MsTest does.
As a result, from my experience, MsTest is faster. I don’t have hard data on this, but from my experience and intuition, it only makes sense that MsTest is faster with its tight integration with Visual Studio.
Which one is more future-proof?
Can Microsoft break NUnit’s integration with Visual Studio or can NUnit break Visual Studio? Which organization has the control here? It’s obviously Microsoft and MsTest. There’s no guarantee that their future changes will not break something about NUnit.
MsTest is less likely to be impacted by future changes to software.
Should you use MsTest or NUnit?
The parity between the APIs is so similar that it’s negligible. The main difference is the ability of MsTest to execute in parallel at the method level. Also, the tight integration of MsTest with Visual Studio provides advantages in both speed and robustness when compared to NUnit. As a result, I recommend MsTest.
NUnit, MSTest, and XUnit all comes with my VS2019.
Its better to use mstestv2 due to one Main reason – full integration with azure test plan/test parameters
FWIW NUnit 3.13 recently introduced the ‘FixtureLifeCycle’ attribute to enable users to specify if they wish for a new test fixture per test case (which seems to be the behaviour that some people expect).
But even before that addition, the parallelisation approach outlined above is not the only way to address method level parallelisation in NUnit. Another approach is using ThreadStatic and/or AsyncLocal properties/fields. As an aside, if you’re doing Selenium tests you can look into using a framework such as Atata to give you parallel execution (and a bunch of other great features) out of the box.
The remark that MSTest is a native unit testing library that comes with Visual Studio is a misrepresentation. MsTest v2 (the current version of mstest) ships as a Nuget package, just like NUnit (and the NUnit 3 test adapter). There is no difference in that respect. Also Visual Studio ships with project templates for NUnit/MSTest/xUnit projects for .NET core/.NET 5, so there is no extra work involved when choosing NUnit over MSTest.
In terms of ‘direct integration into the native apis’, it is important to understand that MStest v2 relies on the VSTest platform to actually run tests and report results. VSTest is the platform which powers the test window in visual studio, vstest.console.exe, and the VSTest tasks in TFS/Azure Devops. MSTest does not have any special ‘direct integration’, it is relying on the APIs exposed by vstest.
As the article points out, Microsoft has a vested interest in ensuring that MSTest and VSTest play nice together. One drawback I’ve experienced with using NUnit is that some of the more ‘advanced’ features of NUnit have some integration issues with VSTest and the test explorer window in Visual Studio, and NUnit is still waiting for Visual Studio to address them. One such issue is that modifying the display name for NUnit tests can cause ‘duplicate’ tests to show up in test explorer and test information (such as the source location) is lost so double clicking on a test in test explorer wont take you to the test in source. There are workarounds to the issue such as disabling Live Unit Testing, but there’s other tradeoffs. If you are interested to read more about the issues, they are all currently tracked in a project in the NUnit3-vs-adapter repo (https://github.com/nunit/nunit3-vs-adapter/projects/2). Also, some concepts in NUnit do not have an equivalent in the object model defined by VSTest. One such example is test outcomes. NUnit defines more test outcomes than are available in VSTest, so the nunit 3 test adapter doesn’t have a direct match to map some of those extra test outcomes to (I can’t remember specifics off the top of my head but I think VSTest doesn’t define ignored or skipped?).
Some benefits I enjoy with NUnit (and have been unable to replicate with MSTest) include:
– Very flexible customisation of test names (https://docs.nunit.org/articles/nunit/technical-notes/nunit-internals/specs/Generation-of-Test-Names-Spec.html?q=test%20name)
– Strongly typed traits/properties (https://docs.nunit.org/articles/nunit/writing-tests/attributes/property.html#custom-property-attributes)
– Using TestCaseData in data driven tests to dynamically add test properties/categories based on test data (https://docs.nunit.org/articles/nunit/writing-tests/TestCaseData.html?q=testcasedata). This alone saves me so much time in larger test suites to create self organising test categories.
– Creating custom test attribute by subclassing the [Test], [TestCase], [TestCaseSource] attributes to provide extended functionality such as ensuring certain properties have defined values for tests.
– Much greater control over test execution via the NUnit extension points (https://docs.nunit.org/articles/nunit/extending-nunit/Framework-Extensibility.html). For example it is easy to create an attribute which can run some logic before a test and conditionally skip the test (and provide a meaningful message).
– The NUnit provides a wide range of vstest runsettings options (https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#vs-test-runsettings-configuration). Sometimes I enable the ‘DumpXmlTestDiscovery’ and process the XML because it provides rich metadata about the tests including the test properties and traits.
Benefits I’ve found for MSTest include access to additional TFS related parameters when running tests using the ‘test plan’ and ‘test run’ selector in TFS (I think this may be what PiePie is referring to above?). The parameters I’m referring to can be seen here: https://github.com/microsoft/testfx/blob/master/src/Adapter/MSTest.CoreAdapter/Constants.cs#L117-L131
There is an open issue (issue 537) on the nunit3-vs-adapter repo for NUnit to add this behaviour.
If my memory serves me correctly the behaviour for parameterised tests is different between MSTest and NUnit. If I have a single parameterised test in MSTest with 2 data iterations, I will only have a single test in VSTest which runs twice (with different data). So it will run both of the data iterations regardless of whether I only wanted to run one of them. With NUnit it would be 2 different tests which I can run independently (and even in parallel). I much prefer NUnit’s behaviour but YMMV.
I also find the official documentation for NUnit to be more detailed than MSTest.
I don’t have any comments on assertions between the two frameworks because I use FluentAssertions.
SS, thanks so much for this awesome addition! Would you be willing to provide this in a document format, covering one feature at a time, with code examples that we can add to our article and give you full credit of course?