In this article we tackle all of the common challenges encountered when trying to do continuous integration with Azure DevOps aka VSTS aka TFS. Some critical topics that are tackled here are:
- How to work with Azure DevOps environment variables
- How to create a build pipeline
By the way, if you aren’t aware, TFS, VSTS and Azure DevOps are all technically the same solution. Over several years, Microsoft did a lot of rebranding that created the confusion between all of these products. Hence it went from TFS to VSTS to Azure DevOps relatively quickly. Causing lots of confusion. The latest version is called Azure DevOps. I hope that it stays with this name for the foreseeable future 🙏
How to set Sauce Labs environment variables in Azure DevOps with .NET Core?
These are instructions if you are working with .NET Core
1. Create environment variables in Azure DevOps
Create the environment variables in ADO and assign values to those variables.
Here’s an example
My recommendation is that you name the variables to something similar that I have above. Do not name your ADO variables the same as your Environment variables as that will cause you issues when you are trying to read them. So don’t name your ADO variable SAUCE_USER_NAME for example
2. Make sure you are reading values from environment variables in your code
For this to work, you need to be reading values into environment variables exactly like this:
var sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME");
var sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY");3. Configure your YAML to read values from VSTS aka Aure DevOps and set them into system environment variables
Your YAML needs this piece of code to be able to set environment variables of the Azure DevOps box
#need to use the vso tasks so that the env variables persist trhough tasks in ADO
- powershell: |
Write-Host "Our Sauce Username in ADO is=> $($env:SAUCE_USER)";
Write-Host "Our Sauce Access Key in ADO is=> $($env:SAUCE_KEY)";
Write-Host ("##vso[task.setvariable variable=SAUCE_USERNAME]$($env:SAUCE_USER)")
Write-Host ("##vso[task.setvariable variable=SAUCE_ACCESS_KEY]$($env:SAUCE_KEY)")Learn more about VSTS aka Azure DevOps logging commands.
Here is the full YAML file that works.
Extra resources
An excellent blog about environment variables in ADO
How to set Sauce Labs environment variables in Azure DevOps with .NET Framework?
There might be a few ways to do this, but I found that using a Powershell script works fine for me.
Using a Powershell script
- First you need to create some environment variables in your Azure DevOps UI that you want to use for values. This is an example of a variable that I would like to set on the test agent for my automation scripts. For example sauce.userName. I will use the value of this variable(sauce.userName) and have a Powershell script set it in my System Environment Variables of the test agent when my automation is running. That way, the value of this variable isn’t exposed to the public.
2. Next, you will want to create a Powershell script that you attach to your solution. Here’s my solution layout.
Don’t forget to make sure to copy your Powershell script to output directory on build in your .csproj
Here is what you want in your Powershell script.
Param(
[string]$sauceUserName,
[string]$sauceAccessKey,
[string]$rdcVodQaNativeAppApiKey,
[string]$rdcSauceDemoIosRdcApiKey
)
Write-Output "sauce.userName value from ADO was passed as a Argument in the ADO Task called $env:SAUCE_USERNAME " +
"to sauceUserName variable in the Posh. This is the value found=>$sauceUserName"
Write-Output "sauce.accessKey that was passed in from Azure DevOps=>$sauceAccessKey"
Write-Output "sauce.rdc.VodQaNativeAppApiKey stored in Azure DevOps=>$rdcVodQaNativeAppApiKey"
Write-Output "sauce.rdc.SauceDemoIosRdcApiKey stored in Azure DevOps=>$rdcSauceDemoIosRdcApiKey"
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_ACCESS_KEY", "$sauceAccessKey", "User")
[Environment]::SetEnvironmentVariable("VODQC_RDC_API_KEY", "$rdcVodQaNativeAppApiKey", "User")
[Environment]::SetEnvironmentVariable("SAUCE_DEMO_IOS_RDC_API_KEY", "$rdcSauceDemoIosRdcApiKey", "User")3. Create a Powershell task in your Pipeline
The Powershell script will read the values that are passed in from the Azure DevOps and then run those values through SetEnvironmentVariable() in the .ps1
When this task runs, I’m having it output the values to the Azure DevOps logs so that I can make sure the right values are being read. Here’s an example
How to read environment variables in Azure DevOps?
If you are reading an environment variable in your code like this
var sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY");Make sure that you do NOT set the EnvironmentVariableTarget.User as your 2nd parameter as Azure DevOps will not be able to read that variable.
Staged test execution in Azure DevOps
What if you want to run different suites in your Azure pipeline so that you have a gated release. For example, you should be trying to run unit tests first, followed by integration tests, followed by acceptance tests. In that case, you need the ability to filter.
By default, Azure DevOps will attempt to run all of your tests in your solution. Regardless of whether you are using NUnit or MsTest or something weird like XUnit (although I never tried with this and wouldn’t)
Here is an example YAML file with the same exact configuration as above
steps:
- task: [email protected]
displayName: 'Run Best Practices Tests without performance'Filtering tests using Azure DevOps
In order to be able to have a staged test execution in our continuous integration pipeline, we need to be able to filter our tests. I have 100s of tests in this repo but I would only like to run the tests that have [Category(“BestPractices”)] from Nunit.
Furthermore, I would like to exclude all of the performance tests, even though they are also categorized as “BestPractices”. As a result, the configuration of our Azure DevOps task looks like this.
And here is what the same exact thing looks like in a .yml
steps:
- task: [email protected]
displayName: 'Run Best Practices Tests without performance'
inputs:
testFiltercriteria: 'TestCategory=BestPractices&TestCategory!=Performance'
runInParallel: true
testRunTitle: 'Best Practices Tests'More on the filtering capabilities in Azure DevOps aka VSTS aka TFS
Why DevOps?
value of dev-ops
Adding continuous integration
- Select Builds > Edit > Triggers. Under Continuous integration, select on the name of your repository.
- Toggle on the checkbox labeled Enable continuous integration.
- You can select CI for both merges and Pull Requests – https://prnt.sc/kb0g6n
How to create scheduled builds
- Picking up from the section above, click the Add button under Scheduled section
- Update the time and the day of when you want to run your builds
- Save & queue
How to set up your test .dll paths
Docs from MS on file matching patterns
The hardest time I had was how to configure my paths for my automation .DLL files. It’s not intuitive to know what is the working directory of your code base.
Here are some examples of what’s worked for me:
Executable Paths
| "C:\Source\Github\dot-net-sauce\SauceExamples\Web.Tests\bin\Debug\Web.Tests.dll" | '**\$(BuildConfiguration)\**\Web.Tests.dll' |
|---|---|
| "C:\Source\Github\dot-net-sauce\SauceExamples\SeleniumNunit\bin\Debug\SeleniumNunit.dll" | '**\$(BuildConfiguration)\**\SeleniumNunit.dll' |
How to pass parameters to test code from a build or release pipeline?
A: Use a runsettings file to pass values as parameters to your test code. For example, in a release that contains several stages, you can pass the appropriate app URL to each the test tasks in each one. The runsettings file and matching parameters must be specified in the Visual Studio Test task.
How to add a status badge to your Github repo
The instructions on this are really good from Microsoft and you can follow them here in the Get the status badge section
Finally, you want to have a powershell step in your YAML that executes this Powershell script and passes in the values from the variables that you set in the Azure DevOps UI.
Below is what my YAML step looks like and I’m basically setting the SAUCE_USERNAME and SAUCE_ACCCESSKEY variables.
In order for the YAML to understand where these variables come from, you need to convert sauce.userName to SAUCE_USERNAME. That’s the Azure convention. Read more about working with variables.
Basically the value stored in sauce.userName is passed in as a variable called $env:SAUCE_USERNAME . It’s the same exact variable, but when you convert it from the ADO UI to YAML, that’s the convention, I know, it’s weird…
sauce.AccessKey = $env:SAUCE_ACCESSKEY
word.a.b.c = WORD_A_B_C
- Building open source projects with Azure DevOps
- Azure DevOps for .NET teams
- Get started with DevOps on Azure
- VS for Mac
- VS 2017 Productivity updates
All about YAML
Working YAML file for UI Automation
Possibly a useful YAML for test automation
I found this YAML snippet for executing tests with VSTest from here
Not sure if it’s helpful yet, but it might be.
# Create a secret variable - powershell: | Write-Host '##vso[task.setvariable variable=mySecret;issecret=true]abc' # Attempt to output the value in various ways - powershell: | # Using an input-macro: Write-Host "This works: $(mySecret)" # Using the env var directly: Write-Host "This does not work: $env:MYSECRET" # Using the mapped env var: Write-Host "This works: $env:MY_MAPPED_ENV_VAR" env: MY_MAPPED_ENV_VAR: $(mySecret) How to run NUnit tests in VSTS
Filtering Tests
Instead of using Nunit test filters, you need to use the MSTest filters. For example, you need to convert:
cat == unit to /TestCaseFilter:”Category=unit”
This will execute all Nunit tests that have the Category “unit”.
