//Adroll script

What is parallelization?

Parallelization is the process of running tests simultaneously on different environments. Such as different browsers or operating systems. It is very useful because it can dramatically decrease the test execution time.

I’ll show you three of the most popular test parallelization frameworks you can use in your C# test automation framework:

  1. NUnit
  2. MSTest
  3. SpecFlow

Parallelization with NUnit

NUnit is one of the most popular testing frameworks for .NET languages. It is an open-source, easy to understand, and has user-friendly attributes.

How to set up parallelization in NUnit

The first thing you need to do is to define the parallel scope of the project. Secondly the level of parallelism, i.e. the number of tests that can be ran in parallel, in your AssemblyInfo.cs class. It should look like this:

//Docs on NUnit parallelization: https://github.com/nunit/docs/wiki/Framework-Parallel-Test-Execution
[assembly: Parallelizable(ParallelScope.Fixtures)]
//Set this value to the Maximum amount of VMs that you have in Sauce Labs
[assembly: LevelOfParallelism(100)]
view raw AssemblyInfo.cs hosted with ❤ by GitHub

Consequently, add the [TestFixture] attribute to the test class. And finally, use the attribute [Parallelizable] in the test fixture.

With NUnit, the parallelization will be done at the class level. The class attributes will look like this:

[TestFixture]
[Parallelizable]
public class Tests
{
// test methods and other class content
}
view raw ParallelNUnit.cs hosted with ❤ by GitHub

This attribute means that the test and all its descendants may be run in parallel with others at the same level.

You can read more about the scopes of test parallelization in NUnit in their official documentation

Advantages of using NUnit for parallelization

The greatest advantage in using NUnit for test parallelization is that is allows cross-platform and cross-browser testing. For instance, here’s a quick example of defining various configurations:

public static IEnumerable LatestConfigurations
{
get
{
//chrome on Mac
yield return new TestFixtureData("Chrome", "latest", "macOS 10.13");
yield return new TestFixtureData("Chrome", "latest-1", "macOS 10.13");
yield return new TestFixtureData("Chrome", "latest-2", "macOS 10.13");
//chrome on Windows(#1 platform as of 2019)
yield return new TestFixtureData("Chrome", "latest", "Windows 10");
yield return new TestFixtureData("Chrome", "latest-1", "Windows 10");
yield return new TestFixtureData("Chrome", "latest-2", "Windows 10");
//chrome on Windows 7(#3 platform as of 2019)
yield return new TestFixtureData("Chrome", "latest", "Windows 7");
yield return new TestFixtureData("Chrome", "latest-1", "Windows 7");
yield return new TestFixtureData("Chrome", "latest-2", "Windows 7");
//safari
yield return new TestFixtureData("Safari", "latest", "macOS 10.14");
yield return new TestFixtureData("Safari", "latest", "macOS 10.12");
//firefox
yield return new TestFixtureData("Firefox", "latest", "macOS 10.13");
yield return new TestFixtureData("Firefox", "latest-1", "macOS 10.13");
yield return new TestFixtureData("Firefox", "latest-2", "macOS 10.13");
//edge
yield return new TestFixtureData("MicrosoftEdge", "latest", "Windows 10");
yield return new TestFixtureData("MicrosoftEdge", "latest-1", "Windows 10");
yield return new TestFixtureData("MicrosoftEdge", "latest-2", "Windows 10");
//IE
yield return new TestFixtureData("Internet Explorer", "latest", "Windows 10");
yield return new TestFixtureData("Internet Explorer", "latest", "Windows 7");
}
}

And then how to use them use them in the test class:

[TestFixture]
[TestFixtureSource(typeof(CrossBrowserData),
nameof(CrossBrowserData.LatestConfigurations))]
[Parallelizable]
public class Tests
{
// test methods and other class content
}

This will result in tests being run in parallel on the different platform and browser configurations.

I also have a sample project here you can check out. 

NUnit limitations

One of the main limitations of NUnit is that it does not run test methods in parallel, only test classes. Methods inside a class will be run one after another. To parallelize with NUnit at the test method level, you can set up your tests like this. However, I warn you, it’s a bit of extra work. You might not find worth the time:

[TestFixture]
[Parallelizable(ParallelScope.All)]
class ParallelAtMethodsWithSelenium
{
private sealed class TestScope : IDisposable
{
public IWebDriver Driver { get; }
private string SauceUserName { get; }
private string SauceAccessKey { get; }
private Dictionary<string, object> SauceOptions { get; }
public TestScope()
{
var chromeOptions = new ChromeOptions
{
BrowserVersion = "latest",
PlatformName = "Windows 10",
UseSpecCompliantProtocol = true
};
//TODO please supply your Sauce Labs user name in an environment variable
SauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
//TODO please supply your own Sauce Labs access Key in an environment variable
SauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY", EnvironmentVariableTarget.User);
SauceOptions = new Dictionary<string, object>
{
["username"] = SauceUserName,
["accessKey"] = SauceAccessKey
};
SauceOptions.Add("name", TestContext.CurrentContext.Test.Name);
chromeOptions.AddAdditionalCapability("sauce:options", SauceOptions, true);
Driver = new RemoteWebDriver(new Uri("https://ondemand.saucelabs.com/wd/hub"),
chromeOptions.ToCapabilities(), TimeSpan.FromSeconds(600));
}
public void Dispose()
{
//clean-up code goes here
if (Driver == null)
return;
var passed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Passed;
((IJavaScriptExecutor)Driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed"));
Driver.Quit();
}
}
[Test]
public void Test1()
{
using (var scope = new TestScope())
{
scope.Driver.Navigate().GoToUrl("https://www.saucedemo.com");
Assert.AreEqual("Swag Labs", scope.Driver.Title);
}
}
[Test]
public void Test2()
{
using (var scope = new TestScope())
{
scope.Driver.Navigate().GoToUrl("https://www.saucedemo.com");
Assert.AreEqual("Swag Labs", scope.Driver.Title);
}
}
}

Parallelization with MSTest

MSTest is Visual Studio’s built-in test library. Its greatest advantage is that it allows us to run tests in parallel at the method level.

How to set up parallelization in MSTest

Setting up MSTest for running test in parallel is very simple and straightforward. Just like we did for NUnit, we need to add the Parallelize property in the AssemplyInfo.cs:

[assembly: Parallelize(Workers = 100, Scope = ExecutionScope.MethodLevel)]

The above property instructs our code to run up to 100 test methods in parallel. In addition, there’s no need to do anything special in the test classes. Why? Because the parallelization is already set up and the test methods will run in parallel:

[TestClass]
public class ParallelSeleniumMethods
{
[TestMethod]
public void SeleniumTest1()
{
SimpleTest();
}
[TestMethod]
public void SeleniumTest2()
{
SimpleTest();
}
[TestMethod]
public void SeleniumTest3()
{
SimpleTest();
}
[TestMethod]
public void SeleniumTest4()
{
SimpleTest();
}
[TestMethod]
public void SeleniumTest5()
{
SimpleTest();
}
}
view raw MSTestClass.cs hosted with ❤ by GitHub

Advantages of using MSTest for parallelization

The biggest advantage of MSTest is that it allows parallelization at the method level. As opposed to the other frameworks which only allow parallelization at the class level. So, for example, If you have 100 test methods in 5 classes, MSTest will let you run 100 tests in parallel. However, with NUnit, you can only run the 5 tests in parallel, one from each test class at the time.

It’s also very easy to configure – just one line of code and you’re good to go.

MSTest limitations

As opposed to NUnit, MsTest can run at the method level. However not at the class level. This makes cross-browser or cross-platform testing more difficult to set up.

If you’re not sure what to choose between NUnit and MSTest, you can check out this article


Parallelization with SpecFlow

For the BDD fans out there, SpecFlow is a great C# framework. Therefore it allows writing the test scenarios in plain English. It uses the Gherkin syntax (Given – When – Then), and then defining the step definitions in C#. 

How to set up parallelization in SpecFlow

SpecFlow is more complex than MSTest and NUnit when it comes to parallelization. Most importantly, you need to do is to set up a Hooks class that should look like this:

[Binding]
public class Hooks
{
public IWebDriver Driver { get; }
private readonly ScenarioContext _scenarioContext;
public Hooks(IWebDriver driver, ScenarioContext scenarioContext)
{
Driver = driver;
_scenarioContext = scenarioContext;
}
[AfterScenario]
public void Teardown()
{
if (Driver == null){ return; }
var passed = _scenarioContext.ScenarioExecutionStatus == ScenarioExecutionStatus.OK;
((IJavaScriptExecutor)Driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed"));
Driver.Quit();
}
}
view raw Hooks.cs hosted with ❤ by GitHub

Next, you need to configure a DriverSetup class. Here, you have the [BeforeScenario] method, where you initialize the driver and register it as an instance.

[Binding]
public class DriverSetup
{
private IObjectContainer _objectContainer;
public IWebDriver Driver;
public DriverSetup(IObjectContainer objectContainer)
{
_objectContainer = objectContainer;
}
[BeforeScenario]
public void BeforeScenario()
{
//TODO please supply your Sauce Labs user name in an environment variable
var sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
//TODO please supply your own Sauce Labs access Key in an environment variable
var sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY", EnvironmentVariableTarget.User);
var sauceOptions = new Dictionary<string, object>
{
["username"] = sauceUserName,
["accessKey"] = sauceAccessKey
};
var chromeOptions = new ChromeOptions
{
BrowserVersion = "latest",
PlatformName = "Windows 10"
};
chromeOptions.AddAdditionalOption("sauce:options", sauceOptions);
Driver = new RemoteWebDriver(new Uri("https://ondemand.saucelabs.com/wd/hub"),
chromeOptions.ToCapabilities(), TimeSpan.FromSeconds(30));
_objectContainer.RegisterInstanceAs(Driver);
}
}
view raw DriverSetup.cs hosted with ❤ by GitHub

Lastly, in the step definition class, you need to set up the parallelization attribute:

[assembly: Parallelize(Workers = 100, Scope = ExecutionScope.ClassLevel)] 

The parallelization is done through MSTest, so this part is done exactly the same way. This means that you can also add this configuration to the AssembleInfo class instead of the step definition.

With the above setting, 100 tests can be run in parallel. However, with SpecFlow, parallelization is done at the class level. In other words, meaning that only features files can run in parallel.

You can check out this Git repository for a project sample with SpecFlow parallel tests.

Advantages of using SpecFlow for parallelization

If your project is already a BDD project that uses SpecFlow, the advantage is that running feature files in parallel saves a lot of execution time.

SpecFlow limitations

In conclusion, the biggest disadvantage, I would say, is that it is so difficult to configure.

Apart from the complex configuration, SpecFlow parallelization only allows feature files to run in parallel, and not individual tests.

What do you think?

Which testing framework do you prefer and why?

0
Tweet
Share
Share