Testing in DevOps is not a trivial process, as it requires continuous testing, automated analysis, and fast-paced release verification. So TestOps, being a subset of DevOps focused on these tasks, is a set of technologies and approaches that brings testing documentation automation, test execution on CI/C, full control over automated testing via reporting, and defect management.
This is the fourth post in a series. Today, we’ll talk about how to create a single space for testing execution with Allure TestOps.
Why is it important? When your project has thousands of tests on three frameworks and a couple of CI’s to run them, the task starts to be intimidating and time-hungry.
In this post, we’ll learn how to run and rerun tests automatically in any possible way and configuration!
Run build jobs
The first thing that we need to set up is the environment. In previous parts, we created several variables for the build job on Jenkins CI side:
- J_HOST we are using to run and tell our tests, what is the host that we need to use to run the tests on.
- J_BRANCH is to select a branch from the VCS.
- J_BROWSER is to provide information to our test, on which browser we want to run our tests
Before we start running the tests from Allure TestOps we need to feed environment information to the system. First of all, let’s check the environment variables. By default, it’s Host, Browser and Branch.
Everything is set up on the Allure TestOps side globally, but for the project we need to link the environment from CI to these global labels. In Settings → Environment we need to create custom fields:
- J_BROWSER linked to the global Browser variable
- J_HOST to linked Host
- J_BRANCH linked to Branch
From this moment on, Allure will recognize these variables in test results and provide this information to the Launches. If we run the tests again, Allure TestOps launch area will show the variables used in the Launch:
As we close the launch and go to the Jobs section, we’ll see the job created in Jenkins. We need to pull the default variables and their values from CI to this job, just update from the build server:
To run a whole build job on the CI side, we select “Run” in the Jobs tab. And if we want to provide some alternative environment to run, we create a set of environment variables:
There is also an option to define two or more sets of parameters. All of them will be run on Jenkins side and provide the test results for these environments. As well, you may easily change the set of tests to be executed by selecting specific groups or tests. The build job will just run with all the setting on “Submit”.
So Allure feeds the environment information to CI and then CI will provide this environment information to the tests and tests will be running on different environments.
Running tests
So, the most common case for running tests is “run all tests and see what happens”, right? Let’s take a look at how it works with Allure TestOps.
Just go to the Test Cases tab, select all the tests, and click “Run” in bulk actions.
After that, we have a one-window run setup: launch name, tags, and environment configuration. Submit. Our job starts on Jenkins!
Another common case is running a set of tests, aimed at a specific microservice, component, or an API. With Allure TestOps, we just check the tests (or a group of them) to run via UI, click “Run”, and configure the run:
After that, Jenkins will trigger a new build job with the specified set of tests. You can see that environment information provided by Allure is stored here and we run just several tests.
All the test results will be transformed into test cases documentation and will be in just one test case. If the launch is closed, test cases will be updated accordingly.
Rerun failed tests / Flaky detection
So, the tests are executed, right? And the first thing we get are results. Quite often, they are not all-green. And in many cases, we might want to know if the fail signals an actual fail or is just a randomly red one. Let’s check it out!
A good way to test a test (pun intended) for flakiness without long investigations is to rerun it. Select the failing tests with a filter and rerun it in the open Launch. This action creates another launch with the same configuration. Results of it will be stored in the Retries tab.
So, at this point, you get failed tests executed again, and if some of them get green – it’s up to you to decide if you want to quarantine (mute) them, leave them be, or create a Defect (LINK) and keep an eye on the issue.
And unresolved test cases moved to Retries and we can check the Retries.
Test Plans: Create and Run
Another big story in the test execution process is creating suites that are regularly run on specific conditions and triggers. It can be any sort of filtering and combination: a set of API tests for a microservice, or UI tests for a component.
To show the way Test Plans work in Allure TestOps, let’s take smoke tests, as an example. First of all, we need to get all the smoke tests done with a tag filter.
To create a set of tests, select all the filtered tests and “Create test plan” in Bulk actions. We will name it, surprise-surprise, “smoke tests”! Now, in Test plans you can see “smoke tests”.
Execution works in a single Run button, the same as with other test runs. The magic here is in the “Sync test cases” button that will add all the new tests with the Smoke tag to the plan and will run them in the future.
On Run, Allure TestOps will initiate a job with smoke tests only, providing the necessary scope for the team to investigate.
Try Allure TestOps Cloud
Qameta Software focuses on developing amazing tools that help software testers. Allure TestOps Cloud is now available, so you are in several clicks from trying next-gen test case execution management!