Automated testing best practices are critical if you want to succeed with test automation!
Test automation has been around for decades! So why not learn the best design patterns from others?
As opposed to re-learning them all yourself.
This post has some of the most important design patterns that you will encounter.
Some patterns mentioned here include how we were able to achieve a 500% reduction in test suite execution time.
There’s an automated testing best practice that we used to achieve a 98% reduction in test case execution time.
Another Selenium design pattern helped this client execute 30,000 tests a week with a 96% pass rate:
Here’s another one:
I recently worked with a client and in about four hours we were able to take a very simple test, detach it from their complicated framework, and
✅ decrease the test case execution time by 60%,
✅ decrease the number of unnecessary Selenium commands by 87% (from >160 calls to 21 😁),
✅ increase test reliability to 100%
There are many more automated testing best practices like this. Keep reading…
About the author:
You might be asking, “What gives me the credibility to recommend automated testing best practices”?
First, I do it out of the desire to help the community learn from my mistakes and my own experiences. I have no funding or affiliations with any agency that would pay me to argue in one direction or another.
Second, as of the end of 2019, I work as part of a team of Solution Architects. There are about 8 of us and we have close to 100 years of combined test automation experience!
Most of these automated testing best practices we have been following for years, as we learned them independently from other luminaries. And we agree on most ideas here, especially the automated testing best practices to avoid.
Finally, if that’s not enough, as a Solutions Architect, I work with about a dozen clients every single year. And probably a hundred different automation engineers. Through every interaction, I see what works and what doesn’t. I mean we run over 3 Million tests per day! We have the data!
That’s when I come to this post and document it for you. So that you avoid the automated testing worst practices and follow the automated testing best practices.
So enjoy and take this seriously, this ain’t no bs 😎
Automated Testing Best Practices
Best Practice: Tests Should Be Atomic
What is an atomic automated test?
An automated atomic test (AAT) is one that tests only a single feature or component. AAT have very few UI interactions and typically touch a maximum of two screens. The “typical” UI end-to-end tests break the AAT pattern.
Furthermore, AATs meet several requirements of good tests as specified by Kent Beck
As an aside, this concept is already well understood in unit and integration tests, but UI tests continue to lag behind.
A good rule of thumb that I use on my teams is:
Automated acceptance test should not run longer than 30 sec on your local resources
Here are some examples of atomic GUI tests:
500% Reduction in Test Suite Execution Time (Case Study)
In a recent case study, we found that 18 long end to end tests ran in ~20 min. Using a much larger suite of 180 atomic tests, with the same exact code coverage, running in parallel, we were able to decrease the entire suite execution time to ~4min.
Advantages of atomic tests
1. Atomic tests fail fast
First, writing atomic tests allows you to fail fast and fail early. This implies that you will get extremely fast and focused feedback. If you want to check the state of a feature, it will take you no longer than 1 minute.
2. Atomic tests decrease flaky behavior
Tests that complete in 2 minutes or less are twice as likely to pass as tests lasting longer than two minutes. In other words, the longer a test takes to run, the more likely it is to fail
Second, writing atomic tests reduces flakiness because it decreases the number of possible breaking points in that test. Flakiness is less of a problem with unit or integration tests. But it is a large problem with acceptance UI automation.
Here’s an example:
- Open UltimateQA.com home page
- Assert that the page opened
- Assert that each section on the page exists
- Open Blog page
- Search for an article
- Assert that the article exists
For UI automation, every single step is a chance for something to go wrong…
A locator may have changed, the interaction mechanism may have changed, your synchronization strategy may be broken, and so on.
Therefore, the more steps that you add, the more likely your test is to break and convey false positives.
3. Atomic checks allow for better testing
The third benefit of writing atomic tests is that if it fails, it will not block other functionality from being tested. For example, the test I mentioned above. If it fails on Step 3, then you might never get to check if the Blog page works or that the Search functionality works. Assuming that you don’t have other tests to check this functionality. As a result of a large test, you will reduce your test coverage.
4. Atomic tests are short and fast
Finally, another great benefit of writing atomic tests is that they will run quicker when parallelized…
How I got a 98% performance enhancement in test execution speed with a single change?
In the scenario above, I had a suite of 18 end-to-end tests that were NOT atomic and were not running in parallel.
Maintaining the same code coverage, I broke down my tests into 180 tiny, atomic tests…
Ran them in parallel and decreased the average time of test case to 1.76s from 86s!
Your entire automation suite run time will be as fast as your slowest test
By the way, I have seen automated tests that take 30 – 90 minutes to execute. These tests are extremely annoying to run because they take so long. Even worse, I’ve never seen such a test produce valuable feedback in my entire career. Only false positives.
Are you having errors in your Selenium automation code? Maybe this post of the Most Common Selenium Errors can help you.
Case Study of Amazing Automation:
✅96% pass rate
✅Average test run time: 30 sec (In the cloud!)
✅The average number of Selenium commands per test: <30
By The Way, Atomic tests are covered in depth in the article below 👇👇
How to break up giant end-to-end UI tests?
Okay, you, believe me, atomic tests are good.
But how can you break up your large end-to-end tests, right?
Trust me, you’re not the only one struggling with this situation…
It gets worse:
Daily, I encounter clients that have the same issue.
Furthermore, I wish that I could provide a simple answer to this. But I cannot…
For most individuals, this challenge is one of technology and culture.
However, I will provide a step by step guide to help you have atomic tests.
It won’t be easy… But when you achieve it, it will be SO Sweet!
Here is a simple scenario:
- Open Amazon.com
- Assert that the page opens
- Search for an item
- Assert that item is found
- Add item to cart
- Assert that item is added
- Assert that checkout is complete
The first problem is that many automation engineers assume that you must do an entire end-to-end flow for this automated test.
You must complete step 1 before step 2 and so on… Because how can you get to the checkout process without having an item in the cart?
The automated testing best practices approach is to be able to inject data to populate the state of the application before any UI interactions.
If you would like to see a live code demonstration, I tackle that in this video at ~12:35 (click Play)
How to manipulate test data for UI automation?
You can inject data via several options:
- Using something like a RESTful API to set the application into a specific state
- Injecting data into the DB to set the application in a certain state
- Using cookies
If you can inject data between the seams of the application, then you can isolate each step and test it on its own
- Use an API to send a web request that will generate a user
- Use an API that will generate an item in your Amazon cart
- Now you can pull up the UI to the cart page and checkout using web automation
- Clean up all test data after
This is the best practice approach. You tested the checkout process without using the UI for steps one through three.
Using an API is extremely fast… A web request can execute in like 100 ms.
This means that steps 1,2,4 can take less than one second to execute. The only other step you will need to do is to finish the checkout process.
It gets better:
Using an API is much more robust than using a UI for test steps. As a result, you will drastically decrease test flakiness in your UI automation.
Probably the most common impediment to atomic testing is the login screen. And most of our apps have one.
So how do we remove this from our test so that our test can be atomic?
Here’s one example:
Congratulations, we are now logged in 🙂
Now use your UI testing tool to perform the single operation that you want to test.
Here’s how a full atomic test would look like:
Notice how the test only has one UI action and one assertion…
That’s a sign of an atomic test in UI automation.
Here’s a tutorial that shows you how to create an atomic test by controlling the state of the UI using JS.
How To Coordinate API and UI Interactions In One Test?
What if you wanted to coordinate API and UI actions in a single test? Well, that would look something like this.
The most common situation is not to alternate Selenium and API calls. Normally we do API stuff to set up state. UI stuff to assert functionality. API stuff to clean up. It’s not common to do Selenium, API, Selenium, API, Selenium and so on.
However, it’s possible for a user case for sure!
Just remember that Selenium or any other UI framework has nothing to do with your App API. A UI automation library is all about the front-end, the UI. The API part of your test is all about managing state and test data, the back end.
What if you can’t inject data for testing?
I know that the world isn’t perfect and many of us aren’t lucky enough to have applications that are developed with testability in mind.
So what can you do?
You have two options:
1. Work with Developers to make the application more testable
Yes, you should work with the developers to make your application more testable. Not being able to easily test your application is a sign of poor development practices.
This does mean that you will need to leave your cube and communicate across silos to break down communication barriers.
Frankly, this is part of your job. You need to communicate across teams and work together to create a stable product.
If the product fails, the whole team fails, not just a specific group
Again, it’s not easy…
I’ve worked at one company where it took me two years to simply integrate Developers, automation, and manual QA into a single CI pipeline.
It was a weekly grind to have everyone caring about the outcome of the automation suite.
And in the end, our team was stronger and more agile than ever.
Trust me, this is doable and most developers are happy to help. But you must be willing to break down these barriers.
Here’s the second option, and you won’t want to hear it:
2. If your application is not automation-friendly, don’t automate
If you can’t work with the developers because you’re unwilling…
Or if the company culture doesn’t allow you to…
Then just don’t automate something that won’t provide value. I know that your manager asked you to automate it…
However, we are the automation engineers. We are the professionals.
We must decide what to automate and not to automate based on our understanding of application requirements.
We were hired because of our technical expertise, because of our abilities to say what is possible, what is not possible, and what will help the project to succeed.
Although it might feel easy to say “yes, I will automate your 30-minute scenario”, it’s not right to do so.
If your manager is non-technical, they should not be telling you how to do your job. You don’t see managers telling developers how to code. Why is it okay for managers to tell an automation engineer what to automate?
The answer is it’s not okay!
You must be the expert and decide on the correct approach to do your job.
If you don’t agree with me…
Check out this video from Robert Martin – arguably one of the best Software Developers of this century.
He does a better job explaining professionalism than I ever could 🙂
Tests Should Run In Parallel
Parallelization is one of the most powerful ways to speed up test suite execution time. I helped one of my clients go from a suite that runs in 41 min to one that runs in 7 min. We did this by working for less than one hour and enabling parallelization.
Other than deleting tests from the suite, there’s nothing else that can achieve such dramatic results so quickly.
However, parallelization is not so easy to achieve and has four mandatory requirements.
1. Your tests cannot use the static keyword
It’s actually a bit more complex than simply not using static. If you have objects that are static and that store state, then you will not be able to run in parallel. This is more technically correct. However, thread-safety is hard and it’s better to simply avoid this keyword with anything mutable.
I worked with a client that wanted to speed up their test suite execution time by running in parallel. After looking at their suite, they used a static WebDriver. This means that their tests cannot run in parallel without heavy refactoring.
2. Tests must be autonomousAn autonomous test is one that doesn't rely on the outcome of another test. If test 1 executes and fails, test 2 should not fail as a result. That's a false positive that isn't indicative of a failure in the system. Rather, it's a failure in the… Click To Tweet
Don’t link your tests together. There is no good reason for it. If you want to speed up your test suite execution time then parallelize, quarantine, break up giant tests.
3. Just-in time data management
Test data management is critical to parallelization. The optimal solution is to be able to create your test data, use it in your test case, and then destroy it afterwards. In this way, there are no dependencies or states of objects that can be changed when running in parallel.
Best Practice: Follow Page Object Pattern
Probably one of the most important automated testing best practices is to follow the Page Object Pattern. I’m sure you heard this one before.
However, I would say at least 80% of people doing test automation still don’t follow the pattern as defined. The pattern that’s existed for over a decade.
In my point of view, that’s a mistake. This is a tried and true pattern that works when followed.
Our team of SAs all follow this pattern and have not had to deviate away from it for any reason. It’s easy and it works. No reason to complicate anything or reinvent the wheel.
A Good Page Object Pattern:
- The page object class has a great name that tells us exactly what the HTML page or HTML component does
- The page class contains methods to interact with the HTML page or component
- Properties and methods live in a single class
- The page object exposes only methods that an end-user would use to interact with the HTML
- A page object doesn’t need to be an entire HTML page, it can be a small component as well
Examples of good page objects
1. The page object class has a great name that tells us exactly what the HTML page or HTML component does
If you cannot name your page object so that it’s 100% clear what’s inside of that page object, then it’s likely your page object does too much.
What does this page object do SauceDemoLoginPage? What methods will be exposed for a user to interact with the HTML?
Are you surprised by what’s inside of this class? Probably not.
2. The page class contains methods to interact with the HTML page or component
The only public methods that are allowed in your page objects are those that an end-user can perform to your web application.
On this HTML page, the only two actions that an end-user can perform are Open() and Login()
The user cannot ConnectToSQL(), OpenExcel(), ReadPDF(). Hence, such actions should never be found in your automated UI tests.
3. Page objects should directly store properties and methods or be composed of objects that expose access
Page objects should allow us the capability to interact with the application through a single interface, the page object. This means that if we want to interact with the login page, we interact with it through
For example, we might want to login
What we want to avoid happening is for all the locators to live in a separate class for example. This makes the code more complicated unnecessarily. Although there are many articles claiming that separating methods from properties for a page object helps to maintain Single Responsibility Principle, that’s actually not correct. The SRP is viewed from the actor who can cause a change to our code. Technically, there is only a single actor that can ever break our class, the Developer. There is no other actor that will solely modify HTML elements without modifying the HTML page.
In conclusion, avoid separating out locators from methods for a page object. It’s over-optimization. However, a page may be composed of multiple components that are relevant to that page object.
4. The page object exposes only methods that an end-user would use to interact with the HTML
On this HTML page, the only two actions that an end-user can perform are Open() and Login()
The user cannot ConnectToSQL(), OpenExcel(), ReadPDF(). Hence, such actions should never be found in your automated UI tests.
5. A page object doesn’t need to be an entire HTML page, it can be a small component as well
This concept is known as composition in programming. And we want to prefer composition over inheritance. Here’s a code example of how composition is used to create cleaner and tighter page objects.
Here’s a video tutorial on Page Objects where I tackle this topic in depth
Front Door First Principle
Objects have several kinds of interfaces. There is the “public” interface that clients are expected to use and there may also be a “private” interface that only close friends should use. Many objects also have an “outgoing interface” consisting of the used part of the interfaces of any objects on which they depend.
The types of interfaces we use has an influence on the robustness of our tests. The use of Back Door Manipulation (page X) to set up the fixture or verify the expected outcome or a test can result in Overcoupled Software (see Fragile Test on page X) that needs more frequent test maintenance. Overuse of Behavior Verification (page X) and Mock Objects (page X) can result in Overspecified Software (see Fragile Test) and tests that are more brittle and that may discourage developers from doing desirable refactorings.
When all choices are equally effective we should use round trip tests to test our system under test (SUT). To do this, we test an object through it’s public interface and use State Verification (page X) to determine whether it behaved correctly. If this is not sufficient to accurately describe the expected behavior we can make our tests layer-crossing tests and use Behavior Verification to verify the calls the SUT makes to depended-on components (DOCs). If we must replace a slow or unavailable DOC with a faster Test Double (page X), using a Fake Object (page X) is preferable because it encodes fewer assumptions into the test (the only assumption being that the component that the Fake Object replaces is actually needed.)XUnit Patterns
Best Practice: Tests should be under 30 sec on your local resources
Another automated testing best practice is that your automated tests should be fast! I can’t stress this enough. Unit and integration tests are already really fast, they run in milliseconds. A unit test can be as fast as one millisecond. So we don’t need to worry about this.
Here’s the kicker:
UI tests are really slow because they run in seconds. Thousands of times slower than unit and integration tests! Even if you do everything perfectly.
So it’s really important to focus on fast UI tests in your test framework design.
Your automated UI tests should take no longer than 30 seconds on your local resources.Nikolay Advolodkin
Automated tests should take no longer than two minutes to executeDanny MacKeown, Automation Guild 2019
The reason for this boils down to the fact that you want your builds to be fast. Meaning no more than 20 minutes. Otherwise, nobody will have the patience to wait so long for automation feedback.
To accomplish a requirement of fast builds. You need small, fast tests, running in parallel. I’m sorry, there is no other way.
Automated Testing Best Practices Video
If you like video format, I recorded a video presentation of the most important automation best practices that I could pack in 60 minutes. I include lost of code examples.
Not! Automated Testing Best Practices
Our industry pretty much sucks as a whole. The majority of us have no clue how to do test automation well.
For example, look at this analysis of 2 Billion Tests from all over the globe:
How disappointing is that?
Some of my unit tests have been running without failure for years. While our GUI automation can’t even pass more than 9 times out of 10 runs.
This means that if you run your test every workday, in 2 weeks, your test will fail at least one time.
It gets better:
Now that we know the problem, we can work towards a solution! The question is why are we so bad at GUI automation?
It’s because of these anti-patterns below…
Don’t use Cucumber-like tools without following BDD
There is one anti-pattern that is guaranteed to kill your test automation success and that’s using Cucumber or other Gherkin style tools without following the process of BDD. Hence, this pattern is at the top of the list.
7 different solution architects with 7 different careers. 75+ years of combined test automation experience. 3 million tests per month executed. And approximately 50 clients per year seen all over the world.
I can confidently say that using a Gherkin style tool without following the process of BDD will kill your automation project. No questions asked.Nikolay Advolodkin
Don’t want to listen to me and the other six test automation experts? Here is the creator of Cucumber.
If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. There are other tools that are designed to do this with far less abstraction and typing overhead than Cucumber.Aslak Hellesøy, creator of Cucumber
I shouldn’t have to say any more. I pray that this is enough to help you reconsider your decision 🙏. And save your test automation project from certain death 🙏. However, you can read about why that’s the case.
Using afterEach or afterAll hooks for test clean up
Avoid using afterEach of afterAll hooks for cleaning up test state. Instead, we want to use beforeEach for cleaning up test state.
This best practice came from Gleb Bahmutov (Cypress.io best practices). Big Thanks 🙏
Tests must not rely on outcome of previous tests
This automation anti-pattern is one of the worst that we can ever make. Let’s imagine that we have test B that relies on the outcome of test A. If test A fails then test B will also fail. However, test B will fail simply because it didn’t get the appropriate state from test A. Not because of an actual bug.
Furthermore, creating this anti-pattern will make parallelization impossible. Without parallelization, an automation project cannot succeed.
Here is some more documentation from Cypress
Anti-Pattern: UI tests should not expose interactions with web elements
The benefit of using Page Objects is that they abstract implementation logic from the tests. The tests can be focused on the scenarios and not implementation. The idea is that the scenario doesn’t change, but the implementation does.
For example, this method is performing a bunch of operations for some actions. At any point, we may need to change our steps. Maybe a new field got added and now we need to check a checkbox. Or, maybe one of the fields gets removed.
Even more common, you want to add logging. In that case, every test will need to be accommodated for this new flow (could be 1000s of tests).
It gets better:
The right way to solve this problem is to encapsulate all of the steps into a method called Login().
Now, it doesn’t matter if we have to add logging, add an extra field, remove a field and so on. There will be a single place to update the Login steps, in the Login() method. Take a look below.
This test will only need to change for a single reason…
If requirements change, and there’s no way around that:
Anti-Pattern: Assuming that more UI automation is better
What do you think are some guaranteed ways to lose trust in automation and kill an automation program?
If you read the title and guessed the answer, great job 🙂
There are very few automation anti-patterns that will kill an automation program faster than using UI automated testing to try and test everything.
More automation is not necessarily better. I would argue that for an organization that is starting, less stable automation is magnitudes of times better than more automation.
Here’s some cool info:
I’m super lucky that I get to consult and work with clients all over the world. So I’ve seen all sorts of organizations.
This organization has executed 123K automated UI tests in 7 days!
Here’s the kicker:
Take a look at this graph and how only 15% of the tests passed.
Can this organization say that out of 100% of the features that are being tested here, that 85% of those features contain bugs?
In that case, this would mean that approximately ~104,000 bugs were logged in the 7 days. That seems, highly unlikely, if not impossible…
So then, what are all of these failures and errors?
They’re called false positives. Failing tests that are not a result of actual faults in the software being tested.
Who is sorting through all of these failures?
Is there someone on the team that is sitting and sorting through all of these failures?
~104,000 non-passing tests…
So what is the reason that these tests failed?
A. Because there is one bug in the application that caused all of these failures?
B. Because two or more bugs are causing all of these problems?
C. Because there are ~zero bugs found and the majority of the failures are a result of worthless automation efforts?
I’d bet $104,000 that it’s option C 😂
It gets worse:
How many automation engineers do you need to sort through 104,000 non-passing tests in one week?
When I ran a team of four automation engineers, we could barely keep up with a few non-passing automated tests per week.
So let’s be honest… nobody is analyzing these non-passing automated tests…
would you agree? Please comment below what you think!
So then what value are these test cases serving the entire organization? What decision do they help the business to make about the quality of the software?
If there was an 85% failure rate in your manual testing, do you move your software to production? Of course not…
So why is it acceptable for so many automated tests to run, not pass, and continue to run?
It’s because this automation is just noise now… The noise that nobody listens to… Not even the developers of the automation.
But, there’s hope…
Some organizations do automation correctly as well. Here’s an example of one…
Why is this automation suite more successful?
First, notice that it was executed over a year. And over a year there were not that many failures…
Yes, this doesn’t necessarily imply that the automation is successful.
Which automation would you trust more?
A. One that is passing for months at a time and gets a failure once every couple of months?
B. Or the automation that has only 15% passing tests of which 104,000 are not passing?
Here’s where it gets interesting:
If you think about a single feature – Facebook login or Amazon search for example.
How often does that featured break based on your experience? Very rarely, if ever (based on my experience at least)
So if you have an automated test case for one of these features, which of the graphs above look more like how the development of the feature behaves?
That’s your answer…
Your automated UI tests should behave almost identical to how the actual development of a feature happens.
Meaning, passing the majority of the time, like a minimum 99.5% of the time and failing once in a blue moon, due to a real regression.
It gets better:
So what can you do to make your automation valuable?
If our UI automation is not providing a correct result more than 99.5% of the time, then stop automating! and fix the reliability. We're only allowed a max of 5 false positives out of 1000 test executions! Running our test once per day= ~5 false… Click To Tweet
If your automation is not providing a correct result more than 99.5% of the time, then stop automating and fix your reliability! You’re only allowed 5 false positives out of 1000 test executions. That’s called quality automation.
Is that impossible?
Not at all. I ran the team that had these execution results below…
Sadly, I no longer have the exact passing percentage of these metrics. But if you do a little estimation, you’ll be able to see that the pass rate of this graph is extremely high.
You can see the red dots on the graph, which signify a failure. Note one of the long non-failure gaps between build ~1450 and ~1600. That’s ~150 builds of zero failures.
Furthermore, I can say that every failure on this graph was a bug that was introduced into the system. Not a false positive which is so common in UI automation.
By the way, I’m not saying this to impress you…
Rather, to impress upon you the idea that 99.5% reliability from UI automation is possible and I’ve seen it.
It gets better:
I recently came across an excellent post by a Microsoft Engineer talking about how they spent two years moving tests from UI automation to the right level of automation and the drastic improvement in automation stability. Here’s the chart:
Check out after Sprint 116 when they introduced their new system!
Just another success story of a company that does automation at the right system level.
Want an in depth explanation with real code? Check out this video
Anti-Pattern: Using complicated data store such as Excel
One of the most common questions from my students and clients is how to use Excel for test data management in test automation.
Don’t use Excel to manage your automation test data
I understand the rationale behind using Excel for your test data. I’ve been doing test automation for a long time and I know about Keyword Driven Frameworks and trying to let manual testers create automated tests in Excel. My friends…
It just doesn’t work… I wasn’t able to make it work myself and I’ve never seen anyone else make it work successfully.
Why using Excel is an anti-pattern?
- The logic to read and manage Excel adds extra overhead to your test automation that isn’t necessary. You will need to write 100s of lines of code just to manage an Excel object and read data based on column headers and row locations. It’s not easy and is prone to error. I’ve done it many years ago. All of this will eat into your automation time and provide no value to the business that employs you.
- You will be required to manage an external solution component for your test automation. This means that you can never simply pull the code and have everything work. You will need to have a license for Excel. You will need to download and install it. And you will need to do this for all of your automation environments. Usually local, Dev, Test, and Prod. This means that you need to manage this Excel instance in all of these environments. This is simply another waste of your time.
What are your solutions?
- The best solution is if you have an API that you can use to read test data. This is a robust and lightweight solution
- If you don’t have an API, you can talk directly to the Database. This takes much less code and it’s much easier to manage than working with and external Excel object.
- If you must use some data source, use a .csv or .json file. CSZ and JSON files are extremely lightweight, easy to read, and can be directly inserted into your automation code. This means that you will be able to simply download the code and have everything work without needing to perform other installations.
Anti-Pattern: Trying to use UI automation to replace manual testing
Automated testing CANNOT replace manual testing
I have not read or seen any automation luminary who claims that automation can replace manual testing. Not right now at least… Our tools have a long way to go.
However, I know and have worked with managers and teams whose goal with test automation is exactly what cannot be done.
And so these individuals pursue an impossible goal… Leading to failure.
The use of automation can drastically enhance the testing process. If used correctly, automation can reduce, not replace the manual testing resources required.
Why can’t automation replace manual testing?
First, it’s impossible to get 100% automated code coverage. It’s impossible to get 100% code coverage in general…
That’s why we still have bugs on all the apps in the world, right?
Anyways, if you can’t automate 100% of the application, that means that you will need some sort of manual testing to validate the uncovered portions.
Second, UI automation is too flaky, too hard to write, and too hard to maintain to get over 25% code coverage…
This is based on experience and is an approximation… I don’t have any hard data on this.
However, you don’t want higher coverage than 25%. I guess it’s possible that with a well designed, modular system, you might be able to get higher UI automation coverage.
But this is an exception, not the rule.
Here’s the kicker:
Using a combination of unit, integration, and UI automation, you might get close to 90% test coverage…
But that’s hard. And this is just a side note.
Finally, some manual activities cannot be automated technologically…
That’s for now and for at least a few years in my opinion.
Some examples include UX Testing and Exploratory Testing.
So again, if you are trying to use automation to replace manual testing, it will be a futile effort.
What is the solution?
Use the automation testing pyramid and don’t try to replace manual testing with UI automation.
Use UI automation and any automation to enhance the testing process.
- A combination of manual testing and automated testing will create the best end-user experience.
Anti-Pattern: Mixing functional automation with performance testing
Description coming soon… In the meantime, do your research about whether this makes sense. Remember, that at the end of the day, it is always faster to run ten, one-minute tests in parallel than to run a single five-minute test.
It’s one minute to suite feedback time versus five minutes. Even if each test takes longer because of the setup and teardown, parallelization is still the most powerful way to scale your automation. Trying to scale your automation by combining tests is not the right approach.
Anti-Pattern: Keyword Driven Testing
Keyword Driven Testing (KDT) is a remnant from the early to mid-2000s when QTP aka UFT was popular.
At one point, we believed that somehow we can write test automation code in such a way that manual testers will be able to string together a bunch of keywords aka functions to make tests.
We used Excel sheets to design test cases by stringing together a bunch of functions (using Excel sheets is also an anti-pattern)
Here’s an example:
What are the problems with KDT?
1. Almost nobody in the world is doing KDT any more
First, let’s define MY success criteria:
- 100% reliability from test automation
- Automation executes on every pull request
- Full automation suite runs in less than 10 min
- Automation is used as a quality gate by the entire organization
I’m fortunate enough to work with dozens of clients and hundreds of automation engineers, every single year.
In my entire career, I have never seen a successful implementation of Keyword Driven Testing. That doesn’t mean that there isn’t one. But…
Why are they so rare? And do you want to take the chance to beat these odds?
Even worse, no automation luminary (Simon Stewart, Titus Fortner, Angie Jones, Alan Richardson, Paul Merrill…) even mentions KDT when they talk UI test automation…
There must be a reason why don’t you think?
Sure, I’ve talked to people online that claim that KDT is working for them. Yet it’s also funny that these people don’t want to share anything publicly…
2. Unnecessary code must be written to read some external source like Excel
Someone has to write the logic to manage the reading and writing to an Excel spreadsheet. Or some other data source. But the Excel spreadsheet is most common with this approach because it’s the most user-friendly for manual testers.
That logic is pretty absurd and takes A LOT of time to get right. Here’s what mine looked like:
Criticize it all you like, I know it’s bad. It was many years ago when I was learning C#.
Regardless, if I did this today with better programming patterns, it would still be a waste of time.
It gets better:
Here is how I data drive my tests today:
This even shows how the test case looks. But really, all of the logic for how to read test data for all of my tests is in this single line of code:
Concise, isn’t it? From 100s of lines of code to one (honestly, it’s not a 100% apples to apple comparison, but it’s close enough).
Thanks so much, NUnit testing framework and all of the Developers that spent years maintaining this code 🙂
Without you, I might still be stuck writing my code to do the same thing 🙁
PS. Don’t get me started on all the logic required to parse and manage strings from an Excel sheet.
3. More complexity by introducing external teams
If you’ve done test automation for some time, I think that you will agree with me that the success criteria that I defined above are not easy to meet?
Even doing test automation without KDT, with simple Page Objects, most of us struggle to even achieve one of the KPIs.
It gets worse:
Imagine trying to do automation where you have to teach manual testers how to use your Excel spreadsheet. Not that there is anything wrong with manual testers.
However, the problem is that now we need to add more communication, which means more potential for miscommunication aka problems.
It’s just another unnecessary variable.
Codeless test automation tools have been failing for decades, why are we trying to recreate them?
4. KDT forces you to create your test runners
Because we want to read a spreadsheet and turn those keywords into functions using reflection, we can’t use the beautiful testing frameworks that have been developed for us by really smart people from JUnit (Java), NUnit (C#), Mocha (JS) and so on…
It seems like a waste of time to me, how about you?
5. KDT forces your tests to be written with interaction commands in mind
The problem here is that Keyword Driven Testing encourages your tests into an improper mindset of
Your automated tests should be written through user actions versus element interactions:
SendKeys(), SendKeys, ClickButton()
Keyword Driven Frameworks encourage imperative tests not declarativeTitus Fortner, Selenium Ruby Bindings maintainer, Watir automation framework maintainer. Check out this talk at ~9:00 https://www.youtube.com/watch?v=vg8um4bYQXI
Here are some of my videos showing these problems:
Trying to automate CAPTCHA
CAPTCHA, short for Completely Automated Public Turing test to tell Computers and Humans Apart, is explicitly designed to prevent automation, so do not try! There are two primary strategies to get around CAPTCHA checks:
Disable CAPTCHAs in your test environment
Add a hook to allow tests to bypass the CAPTCHAhttps://www.selenium.dev/documentation/en/worst_practices/captchas/
Gmail, email, and Facebook logins
… logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.https://www.selenium.dev/documentation/en/worst_practices/gmail_email_and_facebook_logins/
Anti-Pattern: Giant BDD Tests
The description is coming soon…
However, you do not want to have large BDD tests with many “When” and “Then” because it means that you are testing too much. It means that your tests will not be atomic. See Atomic Tests Pattern at the top.
Anti-Pattern: Imperative BDD Tests
The biggest mistake BDD beginners make is writing Gherkin without a behavior-driven mindset. They often write feature files as if they are writing “traditional” procedure-driven functional tests: step-by-step instructions with actions and expected results. These procedure-driven tests are often imperative and trace a path through the system that covers multiple behaviors. As a result, they may be unnecessarily long, which can delay failure investigation, increase maintenance costs, and create confusion.
Automation Panda, https://automationpanda.com/2017/01/30/bdd-101-writing-good-gherkin/
Anti-Pattern: Large Classes
Every class in your test automation project should be no longer than 200 lines of code.
Ultimately, this is a software programming problem. But developing test automation is a programming exercise, so the same concepts apply.
Having small classes (less than 200 lines long) provides the following benefits:
- Classes are more likely to follow the Single Responsibility Principle
- It’s easier to understand what a class is doing. Think
- As a result of the first two points, small classes are easier to maintain
What if I don’t want to have too many classes?
I believe this is a fallacy. I have never encountered anyone who complained of having too many classes or files. Even if it’s possible, it seems likely an exception, rather than the rule.
Do not use soft assertions
What is the point of an assertion that will not fail your test? If you don’t care about the outcome of that assertion, then why are you making it? And if one of your soft assertions fails, does that mean your test will fail as well?
Hypothetically, there is a test that’s validating the checkout process. This test fails because it failed to locate a Contact Us link as it was stepping through the pages. This Contact Us link check was an assertion. So, is the checkout process broken or is there simply a missing link? Do we fail our UserCanCheckoutTest ? If we do, then the test is lying because the user can check out.A soft-assertion in an automated test is simply an excuse for a test that doesn't know how to follow the atomic test principle and doesn't know how to make use of correct parallelization. #automation #programming #anti-pattern Click To Tweet
If you follow atomic testing pattern for test automation, you cannot have soft assertions.
Red Flags (Avoid These Automated Testing Practices)
This section is a collection of automation techniques that I have seen with my clients that cause them a lot of problems. I can’t quite classify them as “anti-patterns” because they are not widely accepted as such in the automation community, by the automation experts. However, I do believe that they are on the brink of being bad practices that you should strongly reconsider.
Using PageFactory from Selenium
Inspired by a post from Titus Fortner (maintainer of Selenium Ruby), the use of PageFactory in Selenium automation is a perverse problem that’s causing plenty of issues for engineers all over the world.
Don’t use PageFactory… It was a terrible mistake putting it [in Selenium], right? Because now I can’t take it out of the project.Simon Stewart, creator of WebDriver
Page Factory suffers from at least two major concerns:
- Increasing the amount of Selenium commands. As a result, making tests slower and more brittle
- Leading to StaleElementException
Increasing the amount of Selenium commands
Use of PageFactory with a ExpectedConditions.elementToBeClickable(), followed by a click() will result in an inefficient chain of requests.
- Locate Element
- Check if Element Displayed
- Locate Element
- Check if Element Enabled
- Locate Element
- Click Element
Typically, this can actually be done with two commands. Hence, to speed up and stabilize the tests, we can use @CacheLookup.
This may seem like a great idea, but before you go and add it to each of your element definitions, beware of its primary drawback, the dreadedTitus Fortner, Page Factory Optimization
StaleElementException. Chromedriver references an element by its location in the Document Object Model (DOM). If the DOM is reloaded by refreshing the page, or it changes in such a way that the element moves positions within the DOM, Selenium will consider that element stale.
Using BDD tools for UI automation
I work on a team of highly skilled solution architects. Combined we have close to 100 years of test automation experience. We have seen ONE good BDD automation implementation in our lives by Aslak Hellesoy. Hopefully, this alone makes you reconsider using BDD automation tools. If not, let’s talk about why failure is drastically magnified.
It all starts with a simple question…
What is Behavior Driven Development?
Behaviour-Driven Development (BDD) is a set of practices that aim to reduce some common wasteful activities in software development:
– Rework caused by misunderstood or vague requirements.
– Technical debt caused by reluctance to refactor code.
– Slow feedback cycles caused by silos and hand-overs
BDD aims to narrow the communication gaps between team members, foster better understanding of the customer and promote continuous communication with real world examples.
Now that we understand that, let me ask…
Did you see the word tool or tools used a single time?
No, we didn’t.
This implies that BDD is NOT a tool, it is a set of practices.
Aslak Hellesoy, the creator of Cucumber says,
If you think Cucumber is a testing tool, please read on, because you are wrong.
There is a process to follow that involves many roles on the software team.
This process is called BDD. It’s what came out of that clique I mentioned. BDD is not a tool you can download
I’m fortunate in that I’m a Solutions Architect (SA) and I get to talk to dozens of new customers and hundreds of automation engineers every year.
Here’s where it gets bad:
The common problem that I encounter and my fellow SAs…
is that almost nobody uses BDD as a set of practices.
No practices are implemented to remove all the waste and technical debt.
Instead, tools such as Cucumber, Serenity, SpecFlow are used to write automated tests and then we claim that we are “doing BDD”.
It gets worse:
The problem is that using BDD tools adds an extra layer of complexity and dependency to the test automation code.
If we aren’t using the BDD process for the actual advantages then all we are doing is adding extra complexity for no benefits 🙁
These are the problems that I see when a BDD tool is used for automation without actually following the BDD practices:
Problem 1: BDD tools create more dependencies
Let’s take a look at a diagram that shows all the dependencies that are added when using a BDD automation framework (I didn’t include all the other dependencies such as test runners and so on as they’re not related to BDD).
When you add a BDD tool to your automation suite, you have the BDD framework such as Cucumber used by Feature Files. Feature Files use Step Definitions and Step Definitions use Page Objects.
Hold on to that thought for a second…
What if you don’t use a BDD automation tool?
By not using a BDD tool, we can remove two extra dependencies.
Dependencies in software development are almost always bad. We want to limit the number of dependencies because each one is a chance for something to go wrong.
Most of the software development design patterns focus on dependency management…
Think Single Responsibility Principle or Open-Closed Principle.
In software development, we strive towards having our modules doing less while limiting the number of dependencies.
So why are we adding extra BDD tool dependencies to our automation if we aren’t using the process?
BDD tools help to create more readable code:
Yes, this is true. I would agree that a test written in good Gherkin syntax is very readable.
However, is it that big of a difference when compared to a non-BDD test like this:
I don’t believe it’s that drastic of a difference.
Is it worth it to take the risk of extra dependency management for slightly more readable tests?
It gets worse:
The other problem that seems to happen with the majority of the BDD tests is that they don’t follow actual BDD best practices.
Problem 2: BDD is being done by the wrong team!
I recently read an interesting article that pointed out another problem of using BDD Tools for automation. As most organizations currently do BDD, it’s the automation engineers that are simply converting manual tests to BDD tests, right?
What that ultimately means is that the completely wrong team is writing BDD.
The developers who are designing the code aren’t writing the test specifications. Even though they are most familiar with the system.
The BAs aren’t writing the specifications either because they view BDD as an automation tool. Even though the BAs are the ones that are the closest to the client.
It gets worse:
The BDD specs are being written by the isolated team that rarely communicates with the client and doesn’t have an intimate knowledge of the system being tested.
How does that make any sense?
Problem 3: BDD tools struggle with parallelization
As far as I’ve seen with BDD automation frameworks such as Cucumber or SpecFlow, parallelization with them is a problem.
You can only parallelize at the Feature File level. This means that you can only run as many tests in parallel as you have feature files. This is a major problem if you are trying to scale your test automation.
Here’s the secret:
If you can’t run in parallel, you can’t scale. Hence, you can’t get fast enough feedback. As a result, your automation program will fail, I can guarantee that.
Problem 4: Poor Gherkin is coded
Let me start by saying that I’m NOT a BDD expert by any means. In my career, I have written about ten BDD tests in my life. All in an attempt to help other teams understand how a correct BDD implementation should look. And I don’t plan on ever writing a production BDD test since I find it an unnecessary burden for little gain.
However, I read the documentation as I work with 100s of automation engineers every single year to resolve the problems that they encounter with BDD. So my Gherkin may not be perfect, but if you doubt my ideas, check the resources for yourself 😉
Here are some examples that I have seen in real life (This example is from Karate framework that also uses the BDD syntax):
Here’s another one from a real customer… 😨😨😨☠
The purpose of Behavior Driven Development scenarios is to document end-user behavior. User’s don’t come to our page to send keys, click buttons and synchronize on elements. Users come to our pages to login or book a hotel or check a reservation.
The danger in exposing UI interactions in the BDD tests is that any time any UI flow changes, you must update all tests to reflect that change. For example, if today your login doesn’t have a captcha, but tomorrow it does.
Even more dangerous are the exposed sleeps. Locally, 10 sec may be enough. In a cloud grid, it will not. Now every test that uses that functionality needs to be updated to wait longer. If this was in a single step of a page object, we would just need to update a single place.
Problem 5: Unnecessary Overhead
Every technology brings overhead with it because you need to learn and answer some of the following questions:
- How does it work?
- What are the best practices?
- How do I use it?
- How do I install it?
Here’s a real example of some of the rules that you will need to learn if you want to do BDD correctly (there are many more than this):
Write all steps in third-person point of view
Write steps as a subject-predicate action phrase
Given-When-Then steps must appear in order and cannot repeat
The most important question that we need to ask ourselves is
Why are we trying to answer all of these questions? What is the problem that we are actually trying to solve?
For slightly better readability… Because our boss told us so…
Those don’t seem like valid reasons to spend extra time mastering another tool.
Avoid Retrying Failed Tests
A concept that is not an automated testing best practice is having retry logic in your test automation. In most cases, probably 99% of the time, retry logic should not exist in your test automation code.
First, if you have retry logic in your test automation, this immediately implies that you don’t trust your test automation code!
You are telling yourself, your team, and the organization that my tests sometimes pass and sometimes fail. You don’t know why. So you just rerun them until they pass.
What this also means is that if they fail, your default assumption is that it’s not a bug in the application, but instability in your test automation.Retry logic in test automation, implies that your #automation isn't trustworthy. The default behavior is to rerun a failed test rather than explore the failure. If the test isn't capable of finding a regression, what's the value of the test?Click To Tweet
It gets worse:
The second and much more sinister problem is that you may cover up real 🐛
I once worked on a project where on random occasions I had an automated UI test that would fail. The behavior was so sporadic and unpredictable that it just didn’t make any sense.
And it was so infrequent that I couldn’t detect a good pattern.
I sat down with a developer and our DevOps engineer and after some time troubleshooting, we figured out the root cause.
The problem was that we had a load balancer and a bunch of servers behind it.
One of the 12 servers was corrupt. And under specific conditions, the automated scenario would be routed to this server.
In this case, the scenario would fail because the functionality on that server was broken.
Our load balancer handled 1000s of web requests per second. This means that potentially a 12th of those requests were failing due to the bug in the software.
If I had added a retry logic to this test, I would have covered up this bug and the company would be losing a lot of money. Potentially 10s of thousands of dollars/month would be my estimate.
This type of error is something that engineers get fired for. When we claim that we have coverage and the risk is mitigated.
When in fact, it’s not.
Hence, this is why we don’t want to have retry logic in our test framework design.
There maybe a very few instances where retry logic is acceptable, but that’s more of an exception than the rule.
Don’t use Cucumber without following BDD
What do you think?
What do you think about these automated testing best practices? Have you encountered any of these problems?