Continuous Delivery with TFS, VSTS, and Azure DevOps

by | Nov 25, 2019 | DevOps | 0 comments

Azure_DevOps

In this article we tackle all of the common challenges encountered when trying to do continuous integration with Azure DevOps aka VSTS aka TFS. Some critical topics that are tackled here are:

  • How to work with Azure DevOps environment variables
  • How to create a build pipeline

By the way, if you aren’t aware, TFS, VSTS and Azure DevOps are all technically the same solution. Over several years, Microsoft did a lot of rebranding that created the confusion between all of these products. Hence it went from TFS to VSTS to Azure DevOps relatively quickly. Causing lots of confusion. The latest version is called Azure DevOps. I hope that it stays with this name for the foreseeable future 🙏

Sauce Labs with Azure DevOps

Sauce Labs is an extremely popular service for performing 360 degree testing in the cloud. Let’s talk about how to set it up to work with Azure DevOps.

Limitations

  • As of March 2022, Sauce Labs does not have any plugins that can be used in Azure DevOps. You must write your own test commands to run tests and start Sauce Connect
  • It’s not possible to view the videos and images of your Sauce Labs tests in Azure DevOps. Instead, we must navigate to saucelabs.com to see that information. However, you will get pass/fail information.

Pre-requisites

C# with Sauce Labs and Azure DevOps

1. Add your Sauce username and access key in ADO

Open your Pipeline and click Variables.

Adding Variables to your pipeline
Paste the values of your sauce username and access key
My recommendation is that you name the variables to something similar that I have above. Do not name your ADO variables the same as your Environment variables as that will cause you issues when you are trying to read them. So don't name your ADO variable SAUCE_USER_NAME for example

2. C# source code should reference environment variables like this

// testFile.cs
var sauceUserName = Environment.GetEnvironmentVariable("SAUCE_USERNAME");
var sauceAccessKey = Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY");

Make sure that you do NOT set the EnvironmentVariableTarget.User as your 2nd parameter as Azure DevOps will not be able to read that variable.

3. Read values from Azure Variables in your .yml

- task: DotNetCoreCLI@2
  displayName: 'Run tests'
  env:
    SAUCE_USERNAME: $(sauceUsername) #this will store the value from 'sauceUsername' into SAUCE_USERNAME
    SAUCE_ACCESS_KEY: $(sauceKey)

Here’s an example of displaying an environment variable

- bash: echo $(SAUCE_USERNAME) # will output our username stored in SAUCE_USERNAME env variable

Learn more about VSTS aka Azure DevOps logging commands. Check out this repo with a fully working example.

An excellent blog about environment variables in ADO.

Use YML and it will be easy. Use Classic Editor and this process will be hard.

NodeJS with SauceLabs and Azure DevOps

  1. Go to Pipelines and create a new pipeline
  2. Link the new pipeline to your repository. You will likely need to provide permissions to Azure Pipelines in your repository management system.
  3. Create a YML file that looks like the code below
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/javascript

trigger:
- main

pool:
  vmImage: ubuntu-latest

# multiple pipelines can re-use variables that are stored in a variable group
variables:
- group: sauce-labs-variables

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '14.x'
  displayName: 'Install Node.js'
  
# Below we navigate to the working directory, install node packages, run tests on Sauce.
# Make sure to set SAUCE_USERNAME and SAUCE_ACCESS_KEY variables
- script: |
    cd ./webdriverio/webdriver/examples/w3c/
    npm install
    npm run test.saucelabs.us -- --logLevel "debug" #it's really wise to enable a high level of logging for CI
  env: 
    SAUCE_USERNAME: $(sauceUsername) #this will read the value from 'sauceUsername' in ADO and will store it into SAUCE_USERNAME env variable
    SAUCE_ACCESS_KEY: $(sauceAccessKey)
  displayName: 'install and run WebdriverIO tests in Sauce Labs'

Ensure that you have your environment variables stored in Azure DevOps.

Learn more by following Azure DevOps docs on JavaScript and Node.js.

Java with Azure Pipelines and Sauce Labs

# Maven
# Build your Java project and run tests with Apache Maven.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/java
trigger:
– main
pr:
– main
pool:
vmImage: 'ubuntu-latest'
#Set the environment variables for the pipeline.
# We create a variable sauce_user and assign it a value of $(SAUCE_USERNAME), which comes from the ADO
variables:
– name: sauce_user
value: $(SAUCE_USERNAME)
– name: sauce_key
value: $(SAUCE_ACCESS_KEY)
steps:
– bash: echo $SAUCE_USER
– bash: echo $SAUCE_KEY
#This will build and run the tests in the Maven project
– task: Maven@3
inputs:
mavenPomFile: 'pom.xml'
mavenOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.8'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/surefire-reports/TEST-*.xml'
goals: 'package'
view raw azureOnSauceWithJava.yml hosted with ❤ by GitHub

How to create environment variables to share across pipelines

If you would like to reuse environment variables across multiple pipelines, then you need to use variable groups. It’s important to know that if use “group”, then you now need to list your other variables in a key/value pairs.

variables:
- group: sauce-labs-variables
- name: solution
  value: '**/*.sln'
- name: buildPlatform
  value: 'Any CPU'
- name: buildConfiguration
  value: 'Release'

Automation pipeline in Azure DevOps with .NET Framework

In this section, we will cover automated testing pipelines in Azure Pipelines.

Working Azure DevOps pipeline with automated tests in .yml

pool:
name: Hosted VS2017
demands:
– msbuild
– visualstudio
– vstest
steps:
– script: set
displayName: print all variables
– task: NuGetToolInstaller@0
displayName: 'Use NuGet 4.4.1'
inputs:
versionSpec: 4.4.1
– task: NuGetCommand@2
displayName: 'NuGet restore'
inputs:
restoreSolution: '**\*.sln'
– task: VSBuild@1
displayName: 'Build solution'
inputs:
solution: '**\*.sln'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactstagingdirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
logProjectEvents: true
– task: PowerShell@2
displayName: 'Set Sauce Environment Variables'
inputs:
targetType: filePath
filePath: ./setEnvironmentVariables.ps1
arguments: '$env:SAUCE_USER $env:SAUCE_KEY'
# Using powershell ##vso command to set an environment variable in the system
– powershell: |
Write-Host "Sauce Username stored in ADO variables is=>$($env:SAUCE_USER)";
Write-Host "Sauce Access Key stored in ADO variables is=>$($env:SAUCE_KEY)";
Write-Host "Sauce Username stored in Env variables is=>$($env:SAUCE_USERNAME)";
Write-Host "Sauce Access Key stored in Env variables is=>$($env:SAUCE_ACCESS_KEY)";
Write-Host "Sauce Build Repository URI stored in Env variables is=>$($env:BUILD_REPOSITORY_URI)";
#checking to make sure that env variables were set between yml tasks
– powershell: |
Write-Host "Sauce Username stored in Env Variables variables is=>$($env:SAUCE_USERNAME)";
Write-Host "Sauce Access Key stored in ADO variables is=>$($env:SAUCE_ACCESS_KEY)";
displayName: display env variables bw posh tasks
– task: VSTest@2
displayName: 'Run Best Practices Framework'
inputs:
searchFolder: '$(System.DefaultWorkingDirectory)'
testSelector: 'testAssemblies'
testAssemblyVer2: |
**\*Selenium*.dll
!**\SauceExamples\packages\*.dll
!**\packages\*.dll
testFiltercriteria: 'TestCategory=BestPractices'
runInParallel: true
codeCoverageEnabled: true
testRunTitle: 'NUnit Automation Framework'
rerunFailedTests: true
rerunFailedThreshold: 10
rerunMaxAttempts: 2
failOnMinTestsNotRun: true
view raw azurePipelines.yml hosted with ❤ by GitHub
Example Azure DevOps pipeline using .NET Framework with automated UI tests

Staged test execution in Azure DevOps

What if you want to run different suites in your Azure pipeline so that you have a gated release. For example, you should be trying to run unit tests first, followed by integration tests, followed by acceptance tests. In that case, you need the ability to filter.

By default, Azure DevOps will attempt to run all of your tests in your solution. Regardless of whether you are using NUnit or MsTest or something weird like XUnit (although I never tried with this and wouldn’t)

When you add a VS Test task to your pipeline, this default configuration will run all your tests

YAML file to run tests

pool:
name: Hosted VS2017
demands:
– msbuild
– visualstudio
– vstest
steps:
– script: set
displayName: print all variables
– task: NuGetToolInstaller@0
displayName: 'Use NuGet 4.4.1'
inputs:
versionSpec: 4.4.1
– task: NuGetCommand@2
displayName: 'NuGet restore'
inputs:
restoreSolution: '**\*.sln'
– task: VSBuild@1
displayName: 'Build solution'
inputs:
solution: '**\*.sln'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactstagingdirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
logProjectEvents: true
– task: PowerShell@2
displayName: 'Set Sauce Environment Variables'
inputs:
targetType: filePath
filePath: ./setEnvironmentVariables.ps1
arguments: '$env:SAUCE_USER $env:SAUCE_KEY'
# Using powershell ##vso command to set an environment variable in the system
– powershell: |
Write-Host "Sauce Username stored in ADO variables is=>$($env:SAUCE_USER)";
Write-Host "Sauce Access Key stored in ADO variables is=>$($env:SAUCE_KEY)";
Write-Host "Sauce Username stored in Env variables is=>$($env:SAUCE_USERNAME)";
Write-Host "Sauce Access Key stored in Env variables is=>$($env:SAUCE_ACCESS_KEY)";
Write-Host "Sauce Build Repository URI stored in Env variables is=>$($env:BUILD_REPOSITORY_URI)";
#checking to make sure that env variables were set between yml tasks
– powershell: |
Write-Host "Sauce Username stored in Env Variables variables is=>$($env:SAUCE_USERNAME)";
Write-Host "Sauce Access Key stored in ADO variables is=>$($env:SAUCE_ACCESS_KEY)";
displayName: display env variables bw posh tasks
– task: VSTest@2
displayName: 'Run Best Practices Framework'
inputs:
searchFolder: '$(System.DefaultWorkingDirectory)'
testSelector: 'testAssemblies'
testAssemblyVer2: |
**\*Selenium*.dll
!**\SauceExamples\packages\*.dll
!**\packages\*.dll
testFiltercriteria: 'TestCategory=BestPractices'
runInParallel: true
codeCoverageEnabled: true
testRunTitle: 'NUnit Automation Framework'
rerunFailedTests: true
rerunFailedThreshold: 10
rerunMaxAttempts: 2
failOnMinTestsNotRun: true
view raw azurePipelines.yml hosted with ❤ by GitHub

Important points about YAML file

I couldn’t get searchFolder to filter out directories

However, using testAssemblyVer2 worked really well

Filtering tests using Azure DevOps

In order to be able to have a staged test execution in our continuous integration pipeline, we need to be able to filter our tests. I have 100s of tests in this repo but I would only like to run the tests that have [Category(“BestPractices”)] from Nunit.

Furthermore, I would like to exclude all of the performance tests, even though they are also categorized as “BestPractices”. As a result, the configuration of our Azure DevOps task looks like this.

test filtering in azure devops

And here is what the same exact thing looks like in a .yml

steps:
- task: VSTest@2
  displayName: 'Run Best Practices Tests without performance'
  inputs:
    testFiltercriteria: 'TestCategory=BestPractices&TestCategory!=Performance'
    runInParallel: true
    testRunTitle: 'Best Practices Tests'

More on the filtering capabilities in Azure DevOps aka VSTS aka TFS

How to package Nuget using Azure DevOps?

Phenomenal resource that will walk you through the process.

This is a resource that helps you to understand how to work with Nuget in .NET Standard.

Set up a Service connection to Nuget in your project settings

You do need to have an MSDN account and then you can create a Nuget.org account.

How to publish the Nuget package

  1. Download Nuget.exe
  2. Add a nuget.config file
  3. Run command nuget restore
  4. Run command .\nuget.exe push -Source "Simple.Sauce" -ApiKey az "C:\Source\SauceLabs\simple_sauce\dotnet\Simple.Sauce\bin\Debug\Simple.Sauce.0.1.1-debug.nupkg"

Why DevOps?

value of dev ops

value of dev-ops

Adding continuous integration

  1. Select Builds > Edit > Triggers. Under Continuous integration, select on the name of your repository.
    1. Toggle on the checkbox labeled Enable continuous integration.
    2. You can select CI for both merges and Pull Requests – https://prnt.sc/kb0g6n

How to create scheduled builds

  1. Picking up from the section above, click the Add button under Scheduled section
  2. Update the time and the day of when you want to run your builds
  3. Save & queue

How to set up your test .dll paths

Docs from MS on file matching patterns

The hardest time I had was how to configure my paths for my automation .DLL files. It’s not intuitive to know what is the working directory of your code base.

Here are some examples of what’s worked for me:

Executable Paths

"C:\Source\Github\dot-net-sauce\SauceExamples\Web.Tests\bin\Debug\Web.Tests.dll"'**\$(BuildConfiguration)\**\Web.Tests.dll'
"C:\Source\Github\dot-net-sauce\SauceExamples\SeleniumNunit\bin\Debug\SeleniumNunit.dll"'**\$(BuildConfiguration)\**\SeleniumNunit.dll'
Column A shows the local path. Column B shows how the path looks in Azure DevOps.

How to pass parameters to test code from a build or release pipeline?

A: Use a runsettings file to pass values as parameters to your test code. For example, in a release that contains several stages, you can pass the appropriate app URL to each the test tasks in each one. The runsettings file and matching parameters must be specified in the Visual Studio Test task.

Source

How to add a status badge to your Github repo

The instructions on this are really good from Microsoft and you can follow them here in the Get the status badge section

All about YAML

Share
Tweet
Share

Trackbacks/Pingbacks

  1. Intro to Selenium WebDriver with .NET Core - Part 1 - Ultimate QA - […] If you want to integrate this code into Azure DevOps aka VSTS, you can read this post… […]

Leave a Reply

Share
Tweet
Share