Testing in DevOps is not a trivial process, as it requires continuous testing, automated analysis, and fast-paced release verification. So TestOps, being a subset of DevOps focused on these tasks, is a set of technologies and approaches that brings testing documentation automation, test execution on CI/C, full control over automated testing via reporting, and defect management.
This is the third post in a series. Today, we’ll talk about how to automate failed tests resolution and bug tracking with Allure TestOps Defect feature.
Why is it important? Automated tests don’t talk and fail frequently. Automated resolution of fails and errors allows to save up to 40% of QA engineer’s time.
Create a Defect
What is a defect? It’s a solid way to work with known issues. If a single error causes 20 automated tests to fail, we don’t want to deal with each failure, right? In this case, a Defect will gather failed tests, link them to a bug, and will keep them out of your sight.
Launch is created, build almost finalized and we will see the test results. This is our test results with one failed test. What do we do? Go to the Overview tab to see the failing test. We can see that two tests are failing and they are marked as unresolved.
Let’s go deeper into the test and see what is happening there. We can see that some data does not match with the test’s expectations. After we analyzed the error and, let it be an actual update in the service under test, understood the reason, we can create a Defect.
A defect is created via an automation rule that assigns it and provides some explicit description as “Content under test has been modified / deleted”. To create the rule, we need a part of the message or the stack trace to create a regular expression: ‘Message pattern: Element should text.*new-feature’.
On creation, the Defect will be linked to all the matching failing tests. If the regular expression is correct, you’ll see the test. And the Rule name “based on message”.
Now, the Defect will appear in the Launch Overview tab. If we go to the test results, 1 test result is linked to an open defect and another test result is ‘an unresolved’ one.
After that, the Defect will resolve the failing test cases. So if you have 1 000 test results and some of them pass here, for example, that will be 5, or 6, or 7, or 10, you won’t need to analyze this again, all the test results with the same error message will be linked to the Defect in current and further launches.
As for the test results not related to this error, they go to the Unresolved category. Thus, you don’t need to analyze any failures related to the created Defect.
Reproduction of an error linked to a Defect
As soon as the test cases with errors are analyzed and sorted to Defects, we can close the launch and just check what will happen if we start another build job. And we will check, what will happen, if the same error occurred. We just check our defects. Here, for example, just one test case affected several launches. At the same time, we can see the history of several test cases failing for the same reason across several launches.
Running a couple of launches highlights that there is a defect. What can we do? In the launches tab we will see the defect without plus sign, which means there is one test in this launch.
In fact, if there is a defect and there are no unresolved test results, we don’t need to analyze the launch results. We don’t need to analyze the root cause of failures and define the root cause as they are known issues reported in a defect. So, this failed test is already registered, understood and processed, and thus is resolved.
Linking a Defect to a JIRA issue
And for this particular defect, for example, we can do other work. We can link an issue on a Jira server. And we can see that there is an issue created “Ads widget is not working”. As you link the Defect to the issue, you will see the issue link and status:
Note. For further screenshots we have taken another issue and screenshots.
There are test cases assigned to this issue. They’re already active, so someone is checking. If we make any changes to this particular issue, for example, the status of the life cycle, the Defect will be updated. And vice versa, the more test cases fail with the expression we have created for the Defect, the more cases you will see in the JIRA issue. As soon as the issue gets resolved, the defect becomes Closed.
It’s important, as, with the issue, we should solve the root cause of the defect and all the automatically resolved failures. So if the error pops up again, we have to reopen the ticket, and update its progress state to reopen the defect.
If we run our Launches again and again, failures that match the defect will be hidden within it. So you will have to look through the failed tests landing in the Unresolved test results section of the Overview tab. These will be only the test cases that you need to analyze within the scope of the launch.
So again, what is a defect? A defect is the acknowledgement that a failed test is analyzed, processed, and ready to be resolved (or is in the process of resolving). And that exact approach saves you a lot of time on the analysis.
Allure TestOps Cloud is now available!
Qameta Software focuses on developing amazing tools that help software testers. Starting Jul 7, 2022 Allure TestOps Cloud went publicly available, so trying the Defects is in a couple of clicks from you now!