In the fast-paced world of software development, ensuring high-quality products is crucial for success. Automated testing has emerged as a powerful ally, enabling organizations to accelerate their testing processes, improve efficiency, and deliver robust software solutions to market. However, implementing effective automated testing requires more than just the right tools and frameworks—it demands a well-defined set of best practices. In this article, we delve into the top 17 automated testing best practices, presenting a comprehensive guide that will empower developers, testers, and quality assurance teams to optimize their testing efforts and achieve unparalleled software quality. Whether you’re a seasoned professional or a novice in the realm of automation, these best practices will serve as invaluable resources to supercharge your testing process, bolster product reliability, and meet the ever-evolving demands of the software industry.
Using a combination of these patterns customers have:
Another Selenium design pattern helped this client execute 30,000 tests a week with a 96% pass rate:
This is what quality automation looks like!
Here’s another one:
In about four hours a customer was able to take a very simple test, detach it from their complicated framework, and
✅ decrease the test case execution time by 60%,
✅ decrease the number of unnecessary Selenium commands by 87% (from >160 calls to 21 😁),
✅ increase test reliability to 100%
There are many more automated testing best practices like this. Keep reading…
About the author:
You might be asking, “What gives me the credibility to recommend automated testing best practices”?
First, I do it out of the desire to help the community learn from my mistakes and my own experiences. I have no funding or affiliations with any agency that would pay me to argue in one direction or another.
Second, as of the end of 2019, I work as part of a team of Solution Architects. There are about 8 of us and we have close to 100 years of combined test automation experience!
Most of these automated testing best practices we have been following for years, as we learned them independently from other luminaries. And we agree on most ideas here, especially the automated testing best practices to avoid.
There’s more:
Finally, if that’s not enough, as a Solutions Architect, I work with about a dozen clients every single year. And probably a hundred different automation engineers. Through every interaction, I see what works and what doesn’t. I mean we run over 3 Million tests per day! We have the data!
That’s when I come to this post and document it for you. So that you avoid the automated testing worst practices and follow the automated testing best practices.
So enjoy and take this seriously, this ain’t no bs 😎
Automated Testing Best Practices
Best Practice: Tests Should Be Atomic
What is an atomic automated test?
An automated atomic test (AAT) is one that tests only a single feature or component. AAT have very few UI interactions and typically touch a maximum of two screens. The “typical” UI end-to-end tests break the AAT pattern.
500% Reduction in Test Suite Execution Time (Case Study)
In a recent case study, we found that 18 long end to end tests ran in ~20 min. Using a much larger suite of 180 atomic tests, with the same exact code coverage, running in parallel, we were able to decrease the entire suite execution time to ~4min.
Advantages of atomic tests
1. Atomic tests fail fast
First, writing atomic tests allows you to fail fast and fail early. This implies that you will get extremely fast and focused feedback. If you want to check the state of a feature, it will take you no longer than 1 minute.
2. Atomic tests decrease flaky behavior
Tests that complete in 2 minutes or less are twice as likely to pass as tests lasting longer than two minutes. In other words, the longer a test takes to run, the more likely it is to fail
Second, writing atomic tests reduces flakiness because it decreases the number of possible breaking points in that test. Flakiness is less of a problem with unit or integration tests. But it is a large problem with acceptance UI automation.
Here’s an example:
Open UltimateQA.com home page
Assert that the page opened
Assert that each section on the page exists
Open Blog page
Search for an article
Assert that the article exists
For UI automation, every single step is a chance for something to go wrong…
A locator may have changed, the interaction mechanism may have changed, your synchronization strategy may be broken, and so on.
Therefore, the more steps that you add, the more likely your test is to break and convey false positives.
3. Atomic checks allow for better testing
The third benefit of writing atomic tests is that if it fails, it will not block other functionality from being tested. For example, the test I mentioned above. If it fails on Step 3, then you might never get to check if the Blog page works or that the Search functionality works. Assuming that you don’t have other tests to check this functionality. As a result of a large test, you will reduce your test coverage.
4. Atomic tests are short and fast
Finally, another great benefit of writing atomic tests is that they will run quicker when parallelized…
How I got a 98% performance enhancement in test execution speed with a single change?
98% improvement in average test case execution time by having atomic, parallel tests
In the scenario above, I had a suite of 18 end-to-end tests that were NOT atomic and were not running in parallel.
Maintaining the same code coverage, I broke down my tests into 180 tiny, atomic tests…
Ran them in parallel and decreased the average time of test case to 1.76s from 86s!
Your entire automation suite run time will be as fast as your slowest test
Nikolay Advolodkin
By the way, I have seen automated tests that take 30 – 90 minutes to execute. These tests are extremely annoying to run because they take so long. Even worse, I’ve never seen such a test produce valuable feedback in my entire career. Only false positives.
Now use your UI testing tool to perform the single operation that you want to test.
Here’s how a full atomic test would look like:
[Test]
public void ShouldBeAbleToCheckOutWithItems()
{
//Arrange
var overviewPage = new CheckoutOverviewPage(Driver);
overviewPage.Open();
//We don't need to actually use th UI to add items to the cart.
//I'm injecting Javascript to control the state of the cart
overviewPage.Cart.SetCartState();
//Act - very few UI interactions
overviewPage.FinishCheckout().
IsCheckoutComplete.Should().BeTrue("we finished the checkout process"); //Assert
}
Notice how the test only has one UI action and one assertion…
That’s a sign of an atomic test in UI automation.
Here’s a tutorial that shows you how to create an atomic test by controlling the state of the UI using JS.
https://www.youtube.com/embed/wOnjvh7EB8s?feature=oembedHow to bypass a login mechanism using JavaScript injection
How To Coordinate API and UI Interactions In One Test?
What if you wanted to coordinate API and UI actions in a single test? Well, that would look something like this.
public void AtomicTest()
{
//This is an API call that will create a new user
//Known as the Arrange in unit testing
var api = new AppApi();
var testData = api.CreateNewUser();
//Now we perform our UI interactions
//The Act in unit testing
var loginPage = new LoginPage(driver);
loginPage.Login(testData);
//Now the assertion, the Act
new ProductsPage(driver).IsLoaded().Should().BeTrue();
//Now clean up, which can also be done in a TearDown hook
api.DeleteUser(testData.UserId);
}
The most common situation is not to alternate Selenium and API calls. Normally we do API stuff to set up state. UI stuff to assert functionality. API stuff to clean up. It’s not common to do Selenium, API, Selenium, API, Selenium and so on.
However, it’s possible for a user case for sure!
Just remember that Selenium or any other UI framework has nothing to do with your App API. A UI automation library is all about the front-end, the UI. The API part of your test is all about managing state and test data, the back end.
What if you can’t inject data for testing?
I know that the world isn’t perfect and many of us aren’t lucky enough to have applications that are developed with testability in mind.
So what can you do?
You have two options:
1. Work with Developers to make the application more testable
Yes, you should work with the developers to make your application more testable. Not being able to easily test your application is a sign of poor development practices.
This does mean that you will need to leave your cube and communicate across silos to break down communication barriers.
Frankly, this is part of your job. You need to communicate across teams and work together to create a stable product.
If the product fails, the whole team fails, not just a specific group
Again, it’s not easy…
I’ve worked at one company where it took me two years to simply integrate Developers, automation, and manual QA into a single CI pipeline.
It was a weekly grind to have everyone caring about the outcome of the automation suite.
And in the end, our team was stronger and more agile than ever.
Trust me, this is doable and most developers are happy to help. But you must be willing to break down these barriers.
Here’s the second option, and you won’t want to hear it:
2. If your application is not automation-friendly, don’t automate
If you can’t work with the developers because you’re unwilling…
Or if the company culture doesn’t allow you to…
Then just don’t automate something that won’t provide value. I know that your manager asked you to automate it…
However, we are the automation engineers. We are the professionals.
We must decide what to automate and not to automate based on our understanding of application requirements.
We were hired because of our technical expertise, because of our abilities to say what is possible, what is not possible, and what will help the project to succeed.
Although it might feel easy to say “yes, I will automate your 30-minute scenario”, it’s not right to do so.
If your manager is non-technical, they should not be telling you how to do your job. You don’t see managers telling developers how to code. Why is it okay for managers to tell an automation engineer what to automate?
The answer is it’s not okay!
You must be the expert and decide on the correct approach to do your job.
If you don’t agree with me…
Check out this video from Robert Martin – arguably one of the best Software Developers of this century.
He does a better job explaining professionalism than I ever could 🙂
Tests Should Run In Parallel
Parallelization is one of the most powerful ways to speed up test suite execution time. I helped one of my clients go from a suite that runs in 41 min to one that runs in 7 min. We did this by working for less than one hour and enabling parallelization.
Other than deleting tests from the suite, there’s nothing else that can achieve such dramatic results so quickly.
However, parallelization is not so easy to achieve and has four mandatory requirements.
1. Your tests cannot use the static keyword
It’s actually a bit more complex than simply not using static. If you have objects that are static and that store state, then you will not be able to run in parallel. This is more technically correct. However, thread-safety is hard and it’s better to simply avoid this keyword with anything mutable.
I worked with a client that wanted to speed up their test suite execution time by running in parallel. After looking at their suite, they used a static WebDriver. This means that their tests cannot run in parallel without heavy refactoring.
2. Tests must be autonomous
Don’t link your tests together. There is no good reason for it. If you want to speed up your test suite execution time then parallelize, quarantine, break up giant tests.
3. Just-in time data management
Test data management is critical to parallelization. The optimal solution is to be able to create your test data, use it in your test case, and then destroy it afterwards. In this way, there are no dependencies or states of objects that can be changed when running in parallel.
Best Practice: Follow Page Object Pattern
Probably one of the most important automated testing best practices is to follow the Page Object Pattern. I’m sure you heard this one before.
However, I would say at least 80% of people doing test automation still don’t follow the pattern as defined. The pattern that’s existed for over a decade.
In my point of view, that’s a mistake. This is a tried and true pattern that works when followed.
Our team of SAs all follow this pattern and have not had to deviate away from it for any reason. It’s easy and it works. No reason to complicate anything or reinvent the wheel.
A Good Page Object Pattern:
The page object class has a great name that tells us exactly what the HTML page or HTML component does
The page class contains methods to interact with the HTML page or component
Properties and methods live in a single class
The page object exposes only methods that an end-user would use to interact with the HTML
A page object doesn’t need to be an entire HTML page, it can be a small component as well
Examples of good page objects
class CheckoutPersonalInfo extends Base {
constructor() {
super(SCREEN_SELECTOR);
}
// Make it private so people can't mess with it
// Source: https://github.com/tc39/proposal-class-fields#private-fields
get #screen() {
return $(SCREEN_SELECTOR);
}
get #cancelButton() {
return $('.cart_cancel_link');
}
/**
* Submit personal info
*
* @param {object} personalInfo
* @param {string} personalInfo.firstName
* @param {string} personalInfo.lastName
* @param {string} personalInfo.zip
*/
submitPersonalInfo(personalInfo) {
const {firstName, lastName, zip} = personalInfo;
this.waitForIsDisplayed();
this.#firstName.addValue(firstName);
triggerOnChange('[data-test="firstName"]');
this.#lastName.addValue(lastName);
triggerOnChange('[data-test="lastName"]');
this.#postalCode.addValue(zip);
triggerOnChange('[data-test="postalCode"]');
this.#continueCheckoutButton.click();
}
}
public class ProductsPage : BasePage
{
private readonly string _pageUrlPart;
public ProductsPage(IWebDriver driver) : base(driver)
{
_pageUrlPart = "inventory.html";
}
// An element can be located using ExpectedConditions through an explicit wait
public bool IsLoaded => Wait.UntilIsDisplayedById("inventory_filter_container");
//elements are not accessible for the external test API
private IWebElement LogoutLink => _driver.FindElement(By.Id("logout_sidebar_link"));
// An element can also be located without ExpectedConditions
private IWebElement HamburgerElement => _driver.FindElement(By.ClassName("bm-burger-button"));
public int ProductCount =>
_driver.FindElements(By.ClassName("inventory_item")).Count;
//We are using Composition to have one page object living in another page object
public CartComponent Cart => new CartComponent(_driver);
public void Logout()
{
HamburgerElement.Click();
LogoutLink.Click();
}
}
1. The page object class has a great name that tells us exactly what the HTML page or HTML component does
If you cannot name your page object so that it’s 100% clear what’s inside of that page object, then it’s likely your page object does too much.
What does this page object do SauceDemoLoginPage? What methods will be exposed for a user to interact with the HTML?
using System.Reflection;
using Common;
using OpenQA.Selenium;
namespace Web.Tests.Pages
{
public class SauceDemoLoginPage : BasePage
{
public SauceDemoLoginPage(IWebDriver driver) : base(driver)
{
}
private readonly By _loginButtonLocator = By.ClassName("btn_action");
public bool IsLoaded => new Wait(_driver, _loginButtonLocator).IsVisible();
private IWebElement PasswordField => _driver.FindElement(By.Id("password"));
private IWebElement LoginButton => _driver.FindElement(_loginButtonLocator);
private readonly By _usernameLocator = By.Id("user-name");
private IWebElement UsernameField => _driver.FindElement(_usernameLocator);
public SauceDemoLoginPage Open()
{
_driver.Navigate().GoToUrl(BaseUrl);
return this;
}
public ProductsPage Login(string username, string password)
{
SauceJsExecutor.LogMessage(
$"Start login with user=>{username} and pass=>{password}");
var usernameField = Wait.UntilIsVisible(_usernameLocator);
usernameField.SendKeys(username);
PasswordField.SendKeys(password);
LoginButton.Click();
SauceJsExecutor.LogMessage($"{MethodBase.GetCurrentMethod().Name} success");
return new ProductsPage(_driver);
}
}
}
Are you surprised by what’s inside of this class? Probably not.
2. The page class contains methods to interact with the HTML page or component
The only public methods that are allowed in your page objects are those that an end-user can perform to your web application.
sauce demo login page
On this HTML page, the only two actions that an end-user can perform are Open() and Login()
The user cannot ConnectToSQL(),OpenExcel(), ReadPDF(). Hence, such actions should never be found in your automated UI tests.
3. Page objects should directly store properties and methods or be composed of objects that expose access
Page objects should allow us the capability to interact with the application through a single interface, the page object. This means that if we want to interact with the login page, we interact with it through SauceDemoLoginPage class.
For example, we might want to login SauceDemoLoginPage.Login("username", "password");
What we want to avoid happening is for all the locators to live in a separate class for example. This makes the code more complicated unnecessarily. Although there are many articles claiming that separating methods from properties for a page object helps to maintain Single Responsibility Principle, that’s actually not correct. The SRP is viewed from the actor who can cause a change to our code. Technically, there is only a single actor that can ever break our class, the Developer. There is no other actor that will solely modify HTML elements without modifying the HTML page.
In conclusion, avoid separating out locators from methods for a page object. It’s over-optimization. However, a page may be composed of multiple components that are relevant to that page object.
4. The page object exposes only methods that an end-user would use to interact with the HTML
On this HTML page, the only two actions that an end-user can perform are Open() and Login()
The user cannot ConnectToSQL(),OpenExcel(), ReadPDF(). Hence, such actions should never be found in your automated UI tests.
5. A page object doesn’t need to be an entire HTML page, it can be a small component as well
This concept is known as composition in programming. And we want to prefer composition over inheritance. Here’s a code example of how composition is used to create cleaner and tighter page objects.
// A car is Composed of an enginer, wheels, and so on.
// Hence, it contains these objects inside of the class
public class Car
{
Engine engine;
Wheels wheels;
}
public class Engine
{
//The Engine.java does stuff related to the engine
}
Here’s a video tutorial on Page Objects where I tackle this topic in depth
Objects have several kinds of interfaces. There is the “public” interface that clients are expected to use and there may also be a “private” interface that only close friends should use. Many objects also have an “outgoing interface” consisting of the used part of the interfaces of any objects on which they depend.
The types of interfaces we use has an influence on the robustness of our tests. The use of Back Door Manipulation (page X) to set up the fixture or verify the expected outcome or a test can result in Overcoupled Software (see Fragile Test on page X) that needs more frequent test maintenance. Overuse of Behavior Verification (page X) and Mock Objects (page X) can result in Overspecified Software (see Fragile Test) and tests that are more brittle and that may discourage developers from doing desirable refactorings.
When all choices are equally effective we should use round trip tests to test our system under test (SUT). To do this, we test an object through it’s public interface and use State Verification (page X) to determine whether it behaved correctly. If this is not sufficient to accurately describe the expected behavior we can make our tests layer-crossing tests and use Behavior Verification to verify the calls the SUT makes to depended-on components (DOCs). If we must replace a slow or unavailable DOC with a faster Test Double (page X), using a Fake Object (page X) is preferable because it encodes fewer assumptions into the test (the only assumption being that the component that the Fake Object replaces is actually needed.)XUnit Patterns
Best Practice: Tests should be under 30 sec on your local resources
Another automated testing best practice is that your automated tests should be fast! I can’t stress this enough. Unit and integration tests are already really fast, they run in milliseconds. A unit test can be as fast as one millisecond. So we don’t need to worry about this.
Here’s the kicker:
UI tests are really slow because they run in seconds. Thousands of times slower than unit and integration tests! Even if you do everything perfectly.
So it’s really important to focus on fast UI tests in your test framework design.
Your automated UI tests should take no longer than 30 seconds on your local resources.Nikolay Advolodkin
Automated tests should take no longer than two minutes to executeDanny MacKeown, Automation Guild 2019
The reason for this boils down to the fact that you want your builds to be fast. Meaning no more than 20 minutes. Otherwise, nobody will have the patience to wait so long for automation feedback.
To accomplish a requirement of fast builds. You need small, fast tests, running in parallel. I’m sorry, there is no other way.
Automated Testing Best Practices Video
If you like video format, I recorded a video presentation of the most important automation best practices that I could pack in 60 minutes. I include lost of code examples.
ℹ️ We have limited space for only 1 assessment per quarter
Don’t use Cucumber-like tools without following BDD
There is one anti-pattern that is guaranteed to kill your test automation success and that’s using Cucumber or other Gherkin style tools without following the process of BDD. Hence, this pattern is at the top of the list.
7 different solution architects with 7 different careers. 75+ years of combined test automation experience. 3 million tests per month executed. And approximately 50 clients per year seen all over the world.
I can confidently say that using a Gherkin style tool without following the process of BDD will kill your automation project. No questions asked.Nikolay Advolodkin
Don’t want to listen to me and the other six test automation experts? Here is the creator of Cucumber.
If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. There are other tools that are designed to do this with far less abstraction and typing overhead than Cucumber.Aslak Hellesøy, creator of Cucumber
I shouldn’t have to say any more. I pray that this is enough to help you reconsider your decision 🙏. And save your test automation project from certain death 🙏. However, you can read about why that’s the case.
Using afterEach or afterAll hooks for test clean up
Avoid using afterEach of afterAll hooks for cleaning up test state. Instead, we want to use beforeEach for cleaning up test state.
This automation anti-pattern is one of the worst that we can ever make. Let’s imagine that we have test B that relies on the outcome of test A. If test A fails then test B will also fail. However, test B will fail simply because it didn’t get the appropriate state from test A. Not because of an actual bug.
Furthermore, creating this anti-pattern will make parallelization impossible. Without parallelization, an automation project cannot succeed.
Anti-Pattern: UI tests should not expose interactions with web elements
The benefit of using Page Objects is that they abstract implementation logic from the tests. The tests can be focused on the scenarios and not implementation. The idea is that the scenario doesn’t change, but the implementation does.
For example, this method is performing a bunch of operations for some actions. At any point, we may need to change our steps. Maybe a new field got added and now we need to check a checkbox. Or, maybe one of the fields gets removed.
Even more common, you want to add logging. In that case, every test will need to be accommodated for this new flow (could be 1000s of tests).
The right way to solve this problem is to encapsulate all of the steps into a method called Login().
Now, it doesn’t matter if we have to add logging, add an extra field, remove a field and so on. There will be a single place to update the Login steps, in the Login() method. Take a look below.
This test will only need to change for a single reason…
If requirements change, and there’s no way around that:
[Test]
public void ShouldNotBeAbleToLoginWithLockedOutUser()
{
_loginPage.Open();
//Although I would likely never test a login through the UI. This is just a small example
var productsPage = _loginPage.Login("locked_out_user", "secret_sauce");
productsPage.IsLoaded.Should().BeFalse("we used a locked out user who should not be able to login.");
}
Anti-Pattern: Assuming that more UI automation is better
What do you think are some guaranteed ways to lose trust in automation and kill an automation program?
If you read the title and guessed the answer, great job 🙂
There are very few automation anti-patterns that will kill an automation program faster than using UI automated testing to try and test everything.
There are very few automation anti-patterns that will kill an automation program faster than using UI automation to try and test everything.
More automation is not necessarily better. I would argue that for an organization that is starting, less stable automation is magnitudes of times better than more automation.
Here’s some cool info:
I’m super lucky that I get to consult and work with clients all over the world. So I’ve seen all sorts of organizations.
An organization that ran 123K UI tests in 7 days
This organization has executed 123K automated UI tests in 7 days!
Here’s the kicker:
Take a look at this graph and how only 15% of the tests passed.
the very low passing rate
Can this organization say that out of 100% of the features that are being tested here, that 85% of those features contain bugs?
In that case, this would mean that approximately ~104,000 bugs were logged in the 7 days. That seems, highly unlikely, if not impossible…
So then, what are all of these failures and errors?
They’re called false positives. Failing tests that are not a result of actual faults in the software being tested.
Who is sorting through all of these failures?
Is there someone on the team that is sitting and sorting through all of these failures?
~104,000 non-passing tests…
So what is the reason that these tests failed?
A. Because there is one bug in the application that caused all of these failures?
B. Because two or more bugs are causing all of these problems?
C. Because there are ~zero bugs found and the majority of the failures are a result of worthless automation efforts?
I’d bet $104,000 that it’s option C 😂
It gets worse:
How many automation engineers do you need to sort through 104,000 non-passing tests in one week?
When I ran a team of four automation engineers, we could barely keep up with a few non-passing automated tests per week.
So let’s be honest… nobody is analyzing these non-passing automated tests…
would you agree? Please comment below what you think!
So then what value are these test cases serving the entire organization? What decision do they help the business to make about the quality of the software?
If there was an 85% failure rate in your manual testing, do you move your software to production? Of course not…
So why is it acceptable for so many automated tests to run, not pass, and continue to run?
It’s because this automation is just noise now… The noise that nobody listens to… Not even the developers of the automation.
Automation Failed!
But, there’s hope…
Some organizations do automation correctly as well. Here’s an example of one…
automated tests executed over a year
Why is this automation suite more successful?
First, notice that it was executed over a year. And over a year there were not that many failures…
Yes, this doesn’t necessarily imply that the automation is successful.
However…
Which automation would you trust more?
A. One that is passing for months at a time and gets a failure once every couple of months?
B. Or the automation that has only 15% passing tests of which 104,000 are not passing?
Here’s where it gets interesting:
If you think about a single feature – Facebook login or Amazon search for example.
How often does that featured break based on your experience? Very rarely, if ever (based on my experience at least)
So if you have an automated test case for one of these features, which of the graphs above look more like how the development of the feature behaves?
That’s your answer…
Your automated UI tests should behave almost identical to how the actual development of a feature happens.
Meaning, passing the majority of the time, like a minimum 99.5% of the time and failing once in a blue moon, due to a real regression.
It gets better:
So what can you do to make your automation valuable?
It’s simple…
If your automation is not providing a correct result more than 99.5% of the time, then stop automating and fix your reliability! You’re only allowed 5 false positives out of 1000 test executions. That’s called quality automation.
Is that impossible?
Not at all. I ran the team that had these execution results below…
automated tests executed over a year
Sadly, I no longer have the exact passing percentage of these metrics. But if you do a little estimation, you’ll be able to see that the pass rate of this graph is extremely high.
You can see the red dots on the graph, which signify a failure. Note one of the long non-failure gaps between build ~1450 and ~1600. That’s ~150 builds of zero failures.
Furthermore, I can say that every failure on this graph was a bug that was introduced into the system. Not a false positive which is so common in UI automation.
By the way, I’m not saying this to impress you…
Rather, to impress upon you the idea that 99.5% reliability from UI automation is possible and I’ve seen it.
It gets better:
I recently came across an excellent post by a Microsoft Engineer talking about how they spent two years moving tests from UI automation to the right level of automation and the drastic improvement in automation stability. Here’s the chart:
Check out after Sprint 116 when they introduced their new system!
Just another success story of a company that does automation at the right system level.
Want an in depth explanation with real code? Check out this video
Anti-Pattern: Using complicated data store such as Excel
One of the most common questions from my students and clients is how to use Excel for test data management in test automation.
Don’t use Excel to manage your automation test data
I understand the rationale behind using Excel for your test data. I’ve been doing test automation for a long time and I know about Keyword Driven Frameworks and trying to let manual testers create automated tests in Excel. My friends…
It just doesn’t work…
Why using Excel is an anti-pattern?
The logic to read and manage Excel adds extra overhead to your test automation that isn’t necessary. You will need to write 100s of lines of code just to manage an Excel object and read data based on column headers and row locations. It’s not easy and is prone to error. I’ve done it many years ago. All of this will eat into your automation time and provide no value to the business that employs you.
You will be required to manage an external solution component for your test automation. This means that you can never simply pull the code and have everything work. You will need to have a license for Excel. You will need to download and install it. And you will need to do this for all of your automation environments. Usually local, Dev, Test, and Prod. This means that you need to manage this Excel instance in all of these environments. This is simply another waste of your time and effort.
Here’s a great example from Andrejs Doronis comparing the complexity of Excel vs a JSON object
Reading data from Excel vs reading data from an object by Andrejs Doronins
What are your solutions?
The best solution is if you have an API that you can use to read test data. This is a robust and lightweight solution
If you don’t have an API, you can talk directly to the Database. This takes much less code and it’s much easier to manage than working with and external Excel object.
If you must use some data source, use a .CSV, .JSON, or .YML. CSV and JSON files are extremely lightweight, easy to read, and can be directly inserted into your automation code. This means that you will be able to simply download the code and have everything work without needing to perform other installations (examples of how to do this can be found in the Complete Selenium w/ Java Bootcamp).
You can even use data classes that make code extremely readable and easy to maintain. See examples below.
Data-Driven Test Using a Data Object in C#
Take a look at how the [TestFixtureSource(typeof(TestConfigData), nameof(TestConfigData.PopularDesktopCombinations))] is used to read data from TestConfigData.PopularDesktopCombinations. You can see the code below.
namespace Core.BestPractices.Web.Tests.Desktop
{
// We read data from TestConfigData.PopularDesktopCombinations
[TestFixtureSource(typeof(TestConfigData), nameof(TestConfigData.PopularDesktopCombinations))]
[TestFixture]
[Parallelizable]
public class DesktopTests : WebTestsBase
{
[SetUp]
public void SetupDesktopTests()
{
if (BrowserOptions.BrowserName == "chrome")
((ChromeOptions) BrowserOptions).AddAdditionalCapability("sauce:options", SauceOptions, true);
else
BrowserOptions.AddAdditionalCapability("sauce:options", SauceOptions);
Driver = GetDesktopDriver(BrowserOptions.ToCapabilities());
}
public string BrowserVersion { get; }
public string PlatformName { get; }
public DriverOptions BrowserOptions { get; }
public DesktopTests(string browserVersion, string platformName, DriverOptions browserOptions)
{
if (string.IsNullOrEmpty(browserVersion))
BrowserVersion = browserVersion;
if (string.IsNullOrEmpty(platformName))
PlatformName = platformName;
BrowserOptions = browserOptions;
}
[Test]
[Category("ci")]
public void LoginWorks()
{
var loginPage = new LoginPage(Driver);
loginPage.Visit();
loginPage.Login("standard_user");
new ProductsPage(Driver).IsVisible().Should().NotThrow();
}
Anti-Pattern: Trying to use UI automation to replace manual testing
Automated testing CANNOT replace manual testing
I have not read or seen any automation luminary who claims that automation can replace manual testing. Not right now at least… Our tools have a long way to go.
However, I know and have worked with managers and teams whose goal with test automation is exactly what cannot be done.
And so these individuals pursue an impossible goal… Leading to failure.
Side note:
The use of automation can drastically enhance the testing process. If used correctly, automation can reduce, not replace the manual testing resources required.
Why can’t automation replace manual testing?
First, it’s impossible to get 100% automated code coverage. It’s impossible to get 100% code coverage in general…
That’s why we still have bugs on all the apps in the world, right?
Anyways, if you can’t automate 100% of the application, that means that you will need some sort of manual testing to validate the uncovered portions.
Second, UI automation is too flaky, too hard to write, and too hard to maintain to get over 25% code coverage…
This is based on experience and is an approximation… I don’t have any hard data on this.
However, you don’t want higher coverage than 25%. I guess it’s possible that with a well designed, modular system, you might be able to get higher UI automation coverage.
But this is an exception, not the rule.
Here’s the kicker:
Using a combination of unit, integration, and UI automation, you might get close to 90% test coverage…
But that’s hard. And this is just a side note.
Finally, some manual activities cannot be automated technologically…
That’s for now and for at least a few years in my opinion.
So again, if you are trying to use automation to replace manual testing, it will be a futile effort.
What is the solution?
Use the automation testing pyramid and don’t try to replace manual testing with UI automation.
Use UI automation and any automation to enhance the testing process.
A combination of manual testing and automated testing will create the best end-user experience.
Anti-Pattern: Mixing functional automation with performance testing
Description coming soon… In the meantime, do your research about whether this makes sense. Remember, that at the end of the day, it is always faster to run ten, one-minute tests in parallel than to run a single five-minute test.
It’s one minute to suite feedback time versus five minutes. Even if each test takes longer because of the setup and teardown, parallelization is still the most powerful way to scale your automation. Trying to scale your automation by combining tests is not the right approach.
Anti-Pattern: Keyword Driven Testing
Keyword Driven Testing (KDT) is a remnant from the early to mid-2000s when QTP aka UFT was popular.
At one point, we believed that somehow we can write test automation code in such a way that manual testers will be able to string together a bunch of keywords aka functions to make tests.
Automation is used as a quality gate by the entire organization
I’m fortunate enough to work with dozens of clients and hundreds of automation engineers, every single year.
In my entire career, I have never seen a successful implementation of Keyword Driven Testing. That doesn’t mean that there isn’t one. But…
Why are they so rare? And do you want to take the chance to beat these odds?
Even worse, no automation luminary (Simon Stewart, Titus Fortner, Angie Jones, Alan Richardson, Paul Merrill…) even mentions KDT when they talk UI test automation…
There must be a reason why don’t you think?
2. Unnecessary code must be written to read some external source like Excel
Someone has to write the logic to manage the reading and writing to an Excel spreadsheet. Or some other data source. But the Excel spreadsheet is most common with this approach because it’s the most user-friendly for manual testers.
That logic is pretty absurd and takes A LOT of time to get right. Here’s what mine looked like:
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Runtime.InteropServices;
using log4net;
using log4net.Config;
using Microsoft.Office.Core;
using Microsoft.Office.Interop.Excel;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using Environment = System.Environment;
namespace AutomatedTesting.DriverScript
{
public class Program
{
#region Fields and Properties
private static string _emailTo, _timeStamp;
private static ILog _log = LogManager.GetLogger(typeof(Program).FullName);
/// A suffix to append to the class name / namespace of a class to form a default assembly qualified name.
private const string DEFAULT_ASSEMBLY_QUALIFIED_SUFFIX = " Version=1.0.0.0, Culture=neutral, PublicKeyToken=null";
private static Dictionary<string, bool> _testResults = new Dictionary<string, bool>();
#endregion
public static int Main(string[] args)
{
var exitCode = ExitCode.Success;
GlobalContext.Properties["hostname"] = Environment.MachineName;
XmlConfigurator.Configure(new FileInfo(string.Format(@"{0}\log4net.config", AppDomain.CurrentDomain.BaseDirectory)));
BaseTester bt = null;
Application xlApp = null;
Workbook wb = null;
Workbooks workBooks = null;
Worksheet testCasesSheet = null;
Worksheet browsersControlSheet = null;
CommandLineParameters parameters = null;
var exHandler = new ExceptionHandler();
var allTestData = new AllTestData();
var testMetadata = new TestMetadata();
var externalTestData = new TestDataFromExternalSource();
if (args.Length == 0)
{
_log.Fatal("args is null");
exitCode = ExitCode.EmptyArguments;
//TODO send email
}
else
{
try
{
_log.Info("STARTED FRAMEWORK....");
_timeStamp = DateTime.Now.ToString("yyyy.MM.dd.HH.mm.ss.fff");
allTestData.Metadata = testMetadata;
allTestData.External = externalTestData;
parameters = FillInCommandLineParameters(args);
Notifier.SendHeartbeat(parameters.Environment);
Notifier.ResetExceptionMessage();
//This is the path of the entire suite of tests for some application that is specified from command line
var originalTestSuiteFilePath =
Path.GetFullPath("TestControls/" + string.Format(ExcelManager.TestSuiteFileName, parameters.App));
var uniqueTestSuiteFilePath = ExcelManager.CreateUniqueWorkbookInstance(originalTestSuiteFilePath,
parameters, _timeStamp);
//2. Create the DB objects to work with the test cases
xlApp = new Application {FileValidation = MsoFileValidationMode.msoFileValidationSkip};
workBooks = xlApp.Workbooks;
wb = workBooks.Open(uniqueTestSuiteFilePath, ReadOnly: false, Editable: true);
testCasesSheet = (Worksheet) wb.Worksheets["TestCases"];
browsersControlSheet = (Worksheet) wb.Worksheets["CrossPlatformTestingConfig"];
var testCasesUsedRange = testCasesSheet.UsedRange;
var testCasesHeadersDictionary = ExcelManager.GetColumnNamesAndIndices(testCasesUsedRange,
testCasesSheet);
var crossPlatformUsedRange = browsersControlSheet.UsedRange;
var crossPlatformDictionary = ExcelManager.GetColumnNamesAndIndices(crossPlatformUsedRange,
browsersControlSheet);
//The buildName represents a single unique time where all of the tests are ran for some application.
//Caution, BrowserStack removes any special characters and replaces them with a space, therefore, don't put special characters in the name
//or the buildId will not be found in BrowserSTack
var desiredTags = string.Join(" ", parameters.Tags);
allTestData.Metadata.Build = string.Format("app:{0}. env:{1}. tags:{2}. user:{3}. time:{4}",
parameters.App,
parameters.Environment.ToUpper(), desiredTags, Environment.UserName, _timeStamp);
ExcelManager.NumberOfRowsForCrossPlatformTesting = crossPlatformUsedRange.Rows.Count;
var dataRetreiver = new ExcelDataRetreiver(browsersControlSheet, crossPlatformDictionary);
//For every single row in the CrossPlatformTestingConfig sheet, we will perform the actions
for (var i = ExcelManager.StartingRow; i < ExcelManager.NumberOfRowsForCrossPlatformTesting; i++)
{
//ConfigureBrowsersAndPlatformsBasedOnExecutionMode(browsersControlSheet, crossPlatformDictionary, i, _smokeTestMode);
if (parameters.SmokeTestMode || parameters.LocalTestingMode)
//if we're running smoke tests or local tests, we only need to run through 1 iteration of the tests
{
//set rowCount to 1 above the starting row so that only one iteration occurs
ExcelManager.NumberOfRowsForCrossPlatformTesting = 4;
//get only the first row of the spreadsheet which will signify the default os/browser combination
allTestData.Metadata.Browser = dataRetreiver.RetreiveValue(i, "browser");
allTestData.Metadata.Version = dataRetreiver.RetreiveValue(i, "version");
allTestData.Metadata.Os = dataRetreiver.RetreiveValue(i, "os");
allTestData.Metadata.OsVersion = dataRetreiver.RetreiveValue(i, "os_version");
allTestData.Metadata.Resolution = dataRetreiver.RetreiveValue(i, "resolution");
}
else
{
allTestData.Metadata.Browser = dataRetreiver.RetreiveValue(i, "browser");
allTestData.Metadata.Version = dataRetreiver.RetreiveValue(i, "version");
allTestData.Metadata.Os = dataRetreiver.RetreiveValue(i, "os");
allTestData.Metadata.OsVersion = dataRetreiver.RetreiveValue(i, "os_version");
allTestData.Metadata.Resolution = dataRetreiver.RetreiveValue(i, "resolution");
}
//None of those should be null
if (string.IsNullOrEmpty(allTestData.Metadata.Browser) ||
string.IsNullOrEmpty(allTestData.Metadata.Version) ||
string.IsNullOrEmpty(allTestData.Metadata.Os) ||
string.IsNullOrEmpty(allTestData.Metadata.OsVersion)) continue;
_log.InfoFormat(
"Iterating through every single row of the spreadsheet to find the test cases that have an Execute flag.");
for (var j = 3; j <= testCasesUsedRange.Rows.Count; j++)
{
var executeColumn = "execute" + parameters.Environment;
var cellValue = ExcelManager.GetCellValue(testCasesSheet, testCasesHeadersDictionary, j,
executeColumn);
//sometimes there may be formatted cells in the spreadsheet that do not contain any data
//we may iterate through them, but should not do any further action if there are no values
if (string.IsNullOrEmpty(cellValue)) continue;
allTestData.External.TestCaseName = ExcelManager.GetCellValue(testCasesSheet, testCasesHeadersDictionary, j,
"classname");
var strOfTagsFromExternalSource = ExcelManager.GetCellValue(testCasesSheet, testCasesHeadersDictionary, j,
"tags");
string[] tagsFromSpreadsheet;
if (strOfTagsFromExternalSource != null)
tagsFromSpreadsheet = strOfTagsFromExternalSource.ToLower().Split(',');
else
{
throw new Exception(
string.Format("There are no tags set up for this test:{0}. Please add tags.",
allTestData.External.TestCaseName));
}
//check to see if the tags in the test cases sheet were called for in the
//execute file. If they match up, then those test cases will be ran
var tagsMatch = tagsFromSpreadsheet.Any(x => parameters.Tags.Contains(x)) ||
parameters.Tags.Contains(allTestData.External.TestCaseName.ToLower());
_log.Debug($"row:{j}. cellValue:{cellValue}. tags retrevied from external source:{strOfTagsFromExternalSource}. tagsMatch:{tagsMatch}");
//if the execute flag is equal 'y' then that means that we are going to execute this test
if (cellValue.ToLower().Trim() != "y" || !tagsMatch)
continue;
allTestData.External.NameSpace = ExcelManager.GetCellValue(testCasesSheet, testCasesHeadersDictionary, j,
"namespace");
//All of the optional parameters that don't necessarily need values
var testLevelType = ExcelManager.GetCellValue(testCasesSheet, testCasesHeadersDictionary, j,
"testlevel");
//if this is an 'api' leve test case, then we dont need it iterating through the regression browsers
//because that is not relevant. Therefore it will only do 1 loop, on 1 browser, through all the test cases
if (!string.IsNullOrEmpty(testLevelType) && testLevelType.ToLower() == "api" || tagsFromSpreadsheet.Contains("smoke"))
ExcelManager.NumberOfRowsForCrossPlatformTesting = 4;
allTestData.External.TestCaseDescription =
testCasesSheet.Cells[j, testCasesHeadersDictionary["description"]].value !=
null
? Convert.ToString(
testCasesSheet.Cells[j, testCasesHeadersDictionary["description"]].value)
: "Description was not specified in the test suite control";
bt = CreateBaseTesterInstance(allTestData);
bt.LocalTestingMode = parameters.LocalTestingMode;
var testPass = false;
//Execute the test case
try
{
//Initialize the Driver
allTestData.Metadata.Project = "WI_" + parameters.App.ToLower() + "_" + parameters.Environment;
_log.InfoFormat("--Running Automation Check.");
_log.InfoFormat("------App:'{0}'.", parameters.App);
_log.InfoFormat("------Test Case Name:'{0}'.", allTestData.External.TestCaseName);
//Set up envs
//TODO, if Test set up fails, then we should not update the previous session with our results
bt.TestSetUp(parameters, allTestData);
bt.RunTest(parameters.Environment);
testPass = true;
}
catch (WebDriverException e)
{
//"Failed to start up socket within 45000 ms. Attempted to connect to the following addresses:"
Notifier.MyException = e;
Notifier.SetEmailToGroup(parameters);
}
catch (WebException e)
{
Notifier.MyException = exHandler.HandleWebException(e);
}
catch (InvalidOperationException e)
{
//The browser/OS was not compatible for BrowserStack and Driver was never initialized.
//This failure is not critical because the applications are still working and only the QA should be notified
//Or it could be the "Not able to reach BrowserStack..." message.
Notifier.MyException = e;
Notifier.SetEmailToGroup(parameters);
}
catch (AssertFailedException e)
{
Notifier.MyException = e;
Notifier.SetEmailToGroup(parameters);
}
catch (Exception e)
{
Notifier.MyException = e;
}
finally
{
_testResults.Add(string.Join(".", Guid.NewGuid(), parameters.Environment, parameters.App, allTestData.External.TestCaseName), testPass);
bt.CleanUp(Notifier.MyException);
exitCode = Notifier.LogUpdateAndEmailFailures(allTestData.External.TestCaseName, parameters, allTestData, bt);
Notifier.ResetExceptionMessage();
}
} //End of the for loop that treverses the entire DB to count all the test cases
} //End of the For loop for the CrossPlatformTestingConfig sheet
} //End of the try block where all the main logic lives
catch (ArgumentNullException e)
{
_log.FatalFormat(
"Error creating a BaseTester class as a result of missing classes. Not found:'{0}'",
allTestData.External.NameSpace + "." + allTestData.External.TestCaseName);
exitCode = ExitCode.BrowserStackSessionRetreivalFailure;
//TODO move into SetEmailToGroup
_emailTo = ConfigurationManager.AppSettings["qaEmailGroup"];
Notifier.SendFailureNotification("Error creating a BaseTester class as a result of missing classes.",
"Error creating a BaseTester class as a result of missing classes.", e, parameters, allTestData);
}
catch (COMException e)
{
exitCode = exHandler.HandleCOMException(e);
Notifier.SendFailureNotification("COMException", parameters.App, e, parameters, allTestData);
}
catch (Exception e)
{
Notifier.SendFailureNotification("Uknown Exception", parameters.App, e, parameters, allTestData);
exitCode = exHandler.HandleGenericException(e);
}
finally
{
try
{
ExcelManager.ReleaseExcelObjects(ref wb, ref xlApp, ref testCasesSheet, ref browsersControlSheet, ref workBooks);
}
catch (COMException e)
{
exitCode = exHandler.HandleCOMException(e);
}
KillAllProcesses("EXCEL");
}
} //End of the else loop that handles all of the logic for if the args != empty
Console.ForegroundColor = ConsoleColor.Cyan;
Console.WriteLine(Environment.NewLine + Environment.NewLine + "------------------ Automated Check Results ----");
Console.WriteLine("Application:{0}", parameters.App);
Console.Write("Environment:{0}", parameters.Environment);
Console.WriteLine(Environment.NewLine);
if (_testResults.Count != 0)
{
var testCount = 1;
foreach (var test in _testResults)
{
Console.ForegroundColor = ConsoleColor.Green;
if (test.Value == false)
{
Console.ForegroundColor = ConsoleColor.Red;
exitCode = ExitCode.TestFailed;
}
Console.WriteLine("{2}. Automated Check:{0} - Result:{1}", test.Key, test.Value, testCount);
testCount++;
}
Console.ForegroundColor = ConsoleColor.Cyan;
Console.WriteLine(
"Docs for checking status of tests are here: http://sdlc.devcentral.equifax.com/confluence/display/WAL/How+to+execute+the+automated+functional+tests+through+Posh");
}
else
{
_log.WarnFormat(
"The framework did not find any tests to run. Check to see if any tests actually exist for your Tags. App:'{0}'. Env:'{1}' that you passed in through command line.",
parameters.App, parameters.Environment);
exitCode = ExitCode.NoTestsWereExecuted;
}
Notifier.SendHeartbeat(parameters.Environment);
_log.InfoFormat("{1}Exiting Framework. Result of batch run:{0}", exitCode, Environment.NewLine);
return (int)exitCode;
}
/// <summary>
/// Will kill the process that is specified
/// </summary>
/// <param name="processName">the name of the process such as Excel</param>
private static void KillAllProcesses(string processName)
{
foreach (var proc in Process.GetProcessesByName(processName))
{
proc.Kill();
}
}
private static BaseTester CreateBaseTesterInstance(AllTestData allTestData)
{
// var assemblyQualifiedName = nameSpace + "." + className + "," + nameSpace + "," + DEFAULT_ASSEMBLY_QUALIFIED_SUFFIX;
var assemblyQualifiedName =
$"{allTestData.External.NameSpace}.{allTestData.External.TestCaseName},{allTestData.External.NameSpace},{DEFAULT_ASSEMBLY_QUALIFIED_SUFFIX}";
var typeDynamic = Type.GetType(assemblyQualifiedName);
BaseTester bt;
if (typeDynamic != null)
{
bt = (BaseTester)Activator.CreateInstance(typeDynamic);
bt.TestName = allTestData.External.TestCaseName;
}
else
{
throw new ArgumentOutOfRangeException(
"When trying to create the dynamic class, we did not find a reference of this project in the Main method." +
"Please Add a reference of your project from which you are running a test to the Main method of the Program.cs class." +
"Located in this project IXI.AutomatedTesting.DriverScript");
}
return bt;
}
/// <summary>
/// A method to decide who should receive the email based on different parameters.
/// For example, if it's Dev or Test, then BJ should not be receiving any of these failures.
/// If there are certain Exceptions that are not related to actual test failures but due to BrowserStack, then BJ should not receive those emails.
/// </summary>
/// <param name="parameters"></param>
/// <param name="type">Represents the exception that was thrown because sometimes we want to send an email to a special group based on some exception.</param>
private static CommandLineParameters FillInCommandLineParameters(string[] args)
{
var parameters = new CommandLineParameters();
parameters.AcceptSsl = false;
parameters.LocalTestingMode = false;
for (var i = 0; i < args.Length; i++)
{
switch (args[i].ToLower())
{
case "-app":
parameters.App = args[i + 1];
break;
case "-env":
parameters.Environment = args[i + 1].ToLower();
break;
//-debug switch will override all
case "-debug":
parameters.DebugMode = true;
Notifier.SetEmailToGroup(parameters);
break;
case "-tags":
var tagsLine = args[i + 1];
parameters.Tags = Array.ConvertAll(tagsLine.ToLower().Split(','), p => p.Trim());
break;
case "-localtesting":
parameters.LocalTestingMode = true;
break;
case "-acceptssl":
parameters.AcceptSsl = true;
break;
}
}
//run through all of the tags, if one of them says Smoke test, then that means that IXI-Bugs needs to be notified
//TODO update query to the same one as the one in Excel using the .contains
foreach (var tag in parameters.Tags)
{
if (tag.ToLower() == "smoke")
{
//set the smoke test mode to true so that we only run through one iteration of a default browser
parameters.SmokeTestMode = true;
Notifier.SetEmailToGroup(parameters);
}
}
//TODO add error handling to make sure that things like app, env and tags are passed in to let the user know to pass them in
_log.DebugFormat("Tags list- app:{0} -env:{1} -debug:{2} -tags:{3} -acceptSsl:{4}", parameters.App, parameters.Environment, parameters.DebugMode, string.Join(",", parameters.Tags, parameters.AcceptSsl));
return parameters;
}
}
}
Criticize it all you like, I know it’s bad. It was many years ago when I was learning C#.
Regardless, if I did this today with better programming patterns, it would still be a waste of time.
It gets better:
Here is how I data drive my tests today:
namespace Web.Tests.BestPractices
{
[TestFixture]
[TestFixtureSource(typeof(CrossBrowserData), "LastTwoOnLinuxFirefoxChrome")]
[Parallelizable]
public class LogoutFeature : BaseTest
{
public LogoutFeature(string browser, string version, string os) :
base(browser, version, os)
{
}
[Test]
public void ShouldBeAbleToLogOut()
{
var loginPage = new SauceDemoLoginPage(Driver);
loginPage.Open();
var productsPage = loginPage.Login("standard_user", "secret_sauce");
productsPage.Logout();
loginPage.IsLoaded.Should().BeTrue("we successfully logged out, so the login page should be visible");
}
}
}
This even shows how the test case looks. But really, all of the logic for how to read test data for all of my tests is in this single line of code:
One line of code to read test data
Concise, isn’t it? From 100s of lines of code to one (honestly, it’s not a 100% apples to apple comparison, but it’s close enough).
Thanks so much, NUnit testing framework and all of the Developers that spent years maintaining this code 🙂
Without you, I might still be stuck writing my code to do the same thing 🙁
PS. Don’t get me started on all the logic required to parse and manage strings from an Excel sheet.
3. More complexity by introducing external teams
If you’ve done test automation for some time, I think that you will agree with me that the success criteria that I defined above are not easy to meet?
Even doing test automation without KDT, with simple Page Objects, most of us struggle to even achieve one of the KPIs.
It gets worse:
Imagine trying to do automation where you have to teach manual testers how to use your Excel spreadsheet. Not that there is anything wrong with manual testers.
However, the problem is that now we need to add more communication, which means more potential for miscommunication aka problems.
It’s just another unnecessary variable.
Codeless test automation tools have been failing for decades, why are we trying to recreate them?
4. KDT forces you to create your test runners
Because we want to read a spreadsheet and turn those keywords into functions using reflection, we can’t use the beautiful testing frameworks that have been developed for us by really smart people from JUnit (Java), NUnit (C#), Mocha (JS) and so on…
It seems like a waste of time to me, how about you?
5. KDT forces your tests to be written with interaction commands in mind
The problem here is that Keyword Driven Testing encourages your tests into an improper mindset of ClickButton(), SendKeys(), ClickElement()…
Your automated tests should be written through user actions versus element interactions:
Login() versus SendKeys(), SendKeys, ClickButton()
Keyword Driven Frameworks encourage imperative tests not declarativeTitus Fortner, Selenium Ruby Bindings maintainer, Watir automation framework maintainer. Check out this talk at ~9:00 https://www.youtube.com/watch?v=vg8um4bYQXI
Here are some of my videos showing these problems:
CAPTCHA, short for Completely Automated Public Turing test to tell Computers and Humans Apart, is explicitly designed to prevent automation, so do not try! There are two primary strategies to get around CAPTCHA checks:
… logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.https://www.selenium.dev/documentation/en/worst_practices/gmail_email_and_facebook_logins/
Anti-Pattern: Giant BDD Tests
The description is coming soon…
However, you do not want to have large BDD tests with many “When” and “Then” because it means that you are testing too much. It means that your tests will not be atomic. See Atomic Tests Pattern at the top.
Anti-Pattern: Imperative BDD Tests
The biggest mistake BDD beginners make is writing Gherkin without a behavior-driven mindset. They often write feature files as if they are writing “traditional” procedure-driven functional tests: step-by-step instructions with actions and expected results. These procedure-driven tests are often imperative and trace a path through the system that covers multiple behaviors. As a result, they may be unnecessarily long, which can delay failure investigation, increase maintenance costs, and create confusion.
Every class in your test automation project should be no longer than 200 lines of code.
Nikolay Advolodkin
Ultimately, this is a software programming problem. But developing test automation is a programming exercise, so the same concepts apply.
Having small classes (less than 200 lines long) provides the following benefits:
Classes are more likely to follow the Single Responsibility Principle
It’s easier to understand what a class is doing. Think GoogleHomePage class versus WebApplicationManager class
As a result of the first two points, small classes are easier to maintain
What if I don’t want to have too many classes?
I believe this is a fallacy. I have never encountered anyone who complained of having too many classes or files. Even if it’s possible, it seems likely an exception, rather than the rule.
Do not use soft assertions
What is the point of an assertion that will not fail your test? If you don’t care about the outcome of that assertion, then why are you making it? And if one of your soft assertions fails, does that mean your test will fail as well?
Hypothetically, there is a test that’s validating the checkout process. This test fails because it failed to locate a Contact Us link as it was stepping through the pages. This Contact Us link check was an assertion. So, is the checkout process broken or is there simply a missing link? Do we fail our UserCanCheckoutTest ? If we do, then the test is lying because the user can check out.
If you follow atomic testing pattern for test automation, you cannot have soft assertions.
Red Flags (Avoid These Automated Testing Practices)
This section is a collection of automation techniques that I have seen with my clients that cause them a lot of problems. I can’t quite classify them as “anti-patterns” because they are not widely accepted as such in the automation community, by the automation experts. However, I do believe that they are on the brink of being bad practices that you should strongly reconsider.
Using PageFactory from Selenium
Inspired by a post from Titus Fortner (maintainer of Selenium Ruby), the use of PageFactory in Selenium automation is a perverse problem that’s causing plenty of issues for engineers all over the world.
Don’t use PageFactory… It was a terrible mistake putting it [in Selenium], right? Because now I can’t take it out of the project.Simon Stewart, creator of WebDriver
Page Factory suffers from at least two major concerns:
Increasing the amount of Selenium commands. As a result, making tests slower and more brittle
Leading to StaleElementException
Increasing the amount of Selenium commands
Use of PageFactory with a ExpectedConditions.elementToBeClickable(), followed by a click() will result in an inefficient chain of requests.
Locate Element
Check if Element Displayed
Locate Element
Check if Element Enabled
Locate Element
Click Element
Typically, this can actually be done with two commands. Hence, to speed up and stabilize the tests, we can use @CacheLookup.
StaleElementException
This may seem like a great idea, but before you go and add it to each of your element definitions, beware of its primary drawback, the dreaded StaleElementException. Chromedriver references an element by its location in the Document Object Model (DOM). If the DOM is reloaded by refreshing the page, or it changes in such a way that the element moves positions within the DOM, Selenium will consider that element stale.Titus Fortner, Page Factory Optimization
Using BDD tools for UI automation
I work on a team of highly skilled solution architects. Combined we have close to 100 years of test automation experience. We have seen ONE good BDD automation implementation in our lives by Aslak Hellesoy. Hopefully, this alone makes you reconsider using BDD automation tools. If not, let’s talk about why failure is drastically magnified.
It all starts with a simple question…
What is Behavior Driven Development?
Behaviour-Driven Development (BDD) is a set of practices that aim to reduce some common wasteful activities in software development:
– Rework caused by misunderstood or vague requirements.
– Technical debt caused by reluctance to refactor code.
– Slow feedback cycles caused by silos and hand-overs
BDD aims to narrow the communication gaps between team members, foster better understanding of the customer and promote continuous communication with real world examples.
https://docs.cucumber.io/bdd/overview/
Now that we understand that, let me ask…
Did you see the word tool or tools used a single time?
No, we didn’t.
This implies that BDD is NOT a tool, it is a set of practices.
Aslak Hellesoy, the creator of Cucumber says,
If you think Cucumber is a testing tool, please read on, because you are wrong.
There is a process to follow that involves many roles on the software team. This process is called BDD. It’s what came out of that clique I mentioned. BDD is not a tool you can download
I’m fortunate in that I’m a Solutions Architect (SA) and I get to talk to dozens of new customers and hundreds of automation engineers every year.
Here’s where it gets bad:
The common problem that I encounter and my fellow SAs…
is that almost nobody uses BDD as a set of practices.
No practices are implemented to remove all the waste and technical debt.
Instead, tools such as Cucumber, Serenity, SpecFlow are used to write automated tests and then we claim that we are “doing BDD”.
It gets worse:
The problem is that using BDD tools adds an extra layer of complexity and dependency to the test automation code.
If we aren’t using the BDD process for the actual advantages then all we are doing is adding extra complexity for no benefits 🙁
These are the problems that I see when a BDD tool is used for automation without actually following the BDD practices:
Problem 1: BDD tools create more dependencies
Let’s take a look at a diagram that shows all the dependencies that are added when using a BDD automation framework (I didn’t include all the other dependencies such as test runners and so on as they’re not related to BDD).
bdd
When you add a BDD tool to your automation suite, you have the BDD framework such as Cucumber used by Feature Files. Feature Files use Step Definitions and Step Definitions use Page Objects.
Hold on to that thought for a second…
What if you don’t use a BDD automation tool?
Without a BDD tool
By not using a BDD tool, we can remove two extra dependencies.
Dependencies in software development are almost always bad. We want to limit the number of dependencies because each one is a chance for something to go wrong.
Most of the software development design patterns focus on dependency management…
Think Single Responsibility Principle or Open-Closed Principle.
In software development, we strive towards having our modules doing less while limiting the number of dependencies.
Nikolay Advolodkin
So why are we adding extra BDD tool dependencies to our automation if we aren’t using the process?
BDD tools help to create more readable code:
Yes, this is true. I would agree that a test written in good Gherkin syntax is very readable.
However, is it that big of a difference when compared to a non-BDD test like this:
[Test]
public void ShouldNotBeAbleToLoginWithLockedOutUser()
{
_loginPage.Open();
//Although I would likely never test a login through the UI. This is just a small example
var productsPage = _loginPage.Login("locked_out_user", "secret_sauce");
productsPage.IsLoaded.Should().BeFalse("we used a locked out user who should not be able to login.");
}
I don’t believe it’s that drastic of a difference.
Is it worth it to take the risk of extra dependency management for slightly more readable tests?
It gets worse:
The other problem that seems to happen with the majority of the BDD tests is that they don’t follow actual BDD best practices.
Problem 2: BDD is being done by the wrong team!
I recently read an interesting article that pointed out another problem of using BDD Tools for automation. As most organizations currently do BDD, it’s the automation engineers that are simply converting manual tests to BDD tests, right?
What that ultimately means is that the completely wrong team is writing BDD.
The developers who are designing the code aren’t writing the test specifications. Even though they are most familiar with the system.
The BAs aren’t writing the specifications either because they view BDD as an automation tool. Even though the BAs are the ones that are the closest to the client.
It gets worse:
The BDD specs are being written by the isolated team that rarely communicates with the client and doesn’t have an intimate knowledge of the system being tested.
How does that make any sense?
Problem 3: BDD tools struggle with parallelization
As far as I’ve seen with BDD automation frameworks such as Cucumber or SpecFlow, parallelization with them is a problem.
You can only parallelize at the Feature File level. This means that you can only run as many tests in parallel as you have feature files. This is a major problem if you are trying to scale your test automation.
Here’s the secret:
If you can’t run in parallel, you can’t scale. Hence, you can’t get fast enough feedback. As a result, your automation program will fail, I can guarantee that.
Problem 4: Poor Gherkin is coded
Let me start by saying that I’m NOT a BDD expert by any means. In my career, I have written about ten BDD tests in my life. All in an attempt to help other teams understand how a correct BDD implementation should look. And I don’t plan on ever writing a production BDD test since I find it an unnecessary burden for little gain.
However, I read the documentation as I work with 100s of automation engineers every single year to resolve the problems that they encounter with BDD. So my Gherkin may not be perfect, but if you doubt my ideas, check the resources for yourself 😉
Here are some examples that I have seen in real life (This example is from Karate framework that also uses the BDD syntax):
Here’s another one from a real customer… 😨😨😨☠
Scenario Outline: Registration - Successful - eshop products
Given I navigate to "<url>"
And I should see page title having partial text as "Login"
When User clicks on "jointoday"
Then I should see the "Choose your package" message
And User selects "<packagecountry>" option
And I should see page title having partial text as "Package Selection | Registration"
And I wait for 5 sec
When User forcefully click on "Continue"
#registration - medical check
Then I should see page title having partial text as "Medical Check | Registration"
And User provided following "first name,last name,dob month,dob day,dob year" input data
And User forcefully click on "gender male"
And User selects no information options
And User forcefully click on "kilograms label"
And User provided following "current weight,height" input data
When User forcefully click on "Continue"
#registration - personal details
Then I should see page title having partial text as "Personal Details | Registration"
And User enters "reg email address" into "reg email address" field
And User enters "reg email address" into "reg confirm email address" field
And User enters "reg password" into "reg password" field
And User enters "reg password" into "reg reconfirm password" field
And I select "<securityquestion>" optionby "text" from dropdown "securityquestion"
Then User enters "billingaddresscity" into "securityanswer" field
And User selects all check boxes
When User forcefully click on "Continue"
#registration - summary
Then I should see page title having partial text as "Summary | Registration"
And User forcefully click on "Enter address manually"
And User provided following "billing address Line1,billing address city,billing address country" input data
And User enters "<billingaddresspostcode>" into "billing address postcode" field
And I select "<state>" optionby "text" from dropdown "billing address state"
And User enters "<contactnumber>" into "billing address contact number" field
And I select "<irelandstate>" optionby "text" from dropdown "billingaddressirstate"
And User selects all check boxes
When User forcefully click on "continue to payment"
#registration - payment
Then I wait for 3 sec
And I should see the "payment method heading" message
And I should see page title having partial text as "Choose your Payment Method"
And User provided following "cc number,cc holder name,cc cvv code" input data
And I select "cc expiry month" optionby "value" from dropdown "cc expiry month"
And I select "cc expiry year" optionby "value" from dropdown "cc expiry year"
When User forcefully click on "Complete Payment"
#registration - successful
Then I wait for 5 sec
And I should see the "payment successful" message
And I should see page title having partial text as "Finished | Registration"
#profilesetup - choose account details
Given User sets headers and navigates to "ukbypassauthentation"
Then I should see page title having partial text as "Personal Details | Profile Builder”
And User enters "choose a username" into "username" field
And I select "skip" optionby "value" from dropdown "time zone"
When User forcefully click on "Continue"
#profilesetup - target weight
Then I should see the "choose your target weight" message
And I should see page title having partial text as "Target Weight | Profile Builder"
And User enters "target weight" into "target weight" field
When User forcefully click on "Continue"
And I wait for 5 sec
#profilesetup - your health
Then I should see the "your health" message
And I should see page title having partial text as "Medical Check | Profile Builder"
And User forcefully click on "donthaveanyofthehealthconditions"
When User forcefully click on "Continue"
#profilesetup - your dietary preferences
Then I should see the "your dietary preferences" message
And I should see page title having partial text as "Dietary Requirements | Profile Builder"
Then User forcefully click on "donthaveanyofthedietarypreferences"
And User forcefully click on "Continue"
Then I should see the "your journey with us starts here" message
And I should see page title having partial text as "Quick Start"
And I should see the "Close" message
And User forcefully click on "Close"
#shop
Given User sets headers and navigates to "shopurl"
Then I wait for 5 sec
And User clicks on "<shopurl>"
And I should see page title having partial text as "Shop"
Then I wait until home page load
And User adds "<icount>" items of "same" boxes "<boxtoadd>" to basket
And User forcefully click on "basketcheckout"
And I wait for 5 sec
Then I should see the "check out header" message
And I wait for 5 sec
Then User completes the payment either by one click payment method or using new card method
#shop - confirmation
And I wait for 5 sec
And I should see the "congratulations" message
And I should see page title having partial text as "Homepage"
Examples:
| url | state | billingaddresspostcode | shopurl | icount | boxtoadd | contactnumber | packagecountry | securityquestion | irelandstate |
| ukurl | ukstate | ukbillingaddresspostcode | skip | 5 | biscuit | irbillingaddresscontactnumber | UK | cityborn | skip |
| ukurl | eustate | eubillingaddresspostcode | skip | 10 | biscuit | irbillingaddresscontactnumber | Europe | cityborn | irstate |
The purpose of Behavior Driven Development scenarios is to document end-user behavior. User’s don’t come to our page to send keys, click buttons and synchronize on elements. Users come to our pages to login or book a hotel or check a reservation.
The danger in exposing UI interactions in the BDD tests is that any time any UI flow changes, you must update all tests to reflect that change. For example, if today your login doesn’t have a captcha, but tomorrow it does.
Even more dangerous are the exposed sleeps. Locally, 10 sec may be enough. In a cloud grid, it will not. Now every test that uses that functionality needs to be updated to wait longer. If this was in a single step of a page object, we would just need to update a single place.
Problem 5: Unnecessary Overhead
Every technology brings overhead with it because you need to learn and answer some of the following questions:
How does it work?
What are the best practices?
How do I use it?
How do I install it?
…
Here’s a real example of some of the rules that you will need to learn if you want to do BDD correctly (there are many more than this):
Write all steps in third-person point of view
Write steps as a subject-predicate action phrase
Given-When-Then steps must appear in order and cannot repeat
The most important question that we need to ask ourselves is
Why are we trying to answer all of these questions? What is the problem that we are actually trying to solve?
For slightly better readability… Because our boss told us so…
Those don’t seem like valid reasons to spend extra time mastering another tool.
Avoid Retrying Failed Tests
In most cases retry logic to rerun failed tests should not exist in test automation code. It’s a dangerous pattern for the following reasons.
First, retry logic in test code immediately implies that you don’t trust your test automation code!
You are telling yourself, your team, and the organization that my tests sometimes pass and sometimes fail. You don’t know why. So you just rerun them until they pass.
The second and much more sinister problem is that you may cover up real 🐛
I once worked on a project where on random occasions I had an automated UI test that would fail. The behavior was so sporadic and unpredictable that it just didn’t make any sense.
And it was so infrequent that I couldn’t detect a good pattern.
I sat down with a developer and our DevOps engineer and after some time troubleshooting, we figured out the root cause.
The problem was that we had a load balancer and a bunch of servers behind it.
One of the 12 servers was corrupt. And under specific conditions, the automated scenario would be routed to this server.
In this case, the scenario would fail because the functionality on that server was broken.
Our load balancer handled 1000s of web requests per second. This means that potentially a 12th of those requests were failing due to the bug in the software.
If I had added a retry logic to this test, I would have covered up this bug and the company would be losing a lot of money. Potentially 10s of thousands of dollars/month would be my estimate.
This type of error is something that engineers get fired for. When we claim that we have coverage and the risk is mitigated.
When in fact, it’s not.
Hence, this is why we don’t want to have retry logic in our test framework design.
However, if the root cause of the failure is 100% clear, and it’s out of our control, retry logic may be an acceptable solution.
Here’s an example:
iPhone 12.1 doesn’t connect to the internet
This test failed because, ” An element could not be located on the page using the given search parameters.”. After root cause analysis, it’s clear that the reason for the failure was because of the iPhone failure. For whatever reason it wasn’t able to connect to the internet. This test is using a 3rd party service provider for it’s device farm.
We can add retry logic to this test and have it rerun on a failure. I find this to be an acceptable solution, especially for mobile automation.
Also, we can try to find a cloud provider that isn’t as flaky in its infrastructure. This option is drastically more difficult and unreasonable in this situation.
What do you think?
What do you think about these automated testing best practices? Have you encountered any of these problems?
Find out how our clients are running automation 66% faster! Get your Free framework assessment.
ℹ️ We have limited space for only 1 assessment per quarter.
25 Comments
Daryl Flowers
on September 3, 2018 at 6:28 am
Thanks, great read and leads me wanting to learn more and implement this.
Only thing that makes this hard is finding these practice as videos to attempt to carry out/ Learn some of these changes, do you have a tutorial on these? or do you know of some that would allow this learning of these easier to understand (more so around the API and DB level)
For BDD, the real anti-pattern is POM, not the feature files or step definitions. Regarded as the de facto design pattern, automation testers push POM to every test automation suite whether it needs it or not. It is safe to say POM is the only design pattern they know, as most of them do not have a proper software engineering background or simply their only experience of code are badly implemented test automation suites.
Anything implemented poorly will be a problem. I’ve seen Page Objects implemented poorly and I’ve seen BDD tests implemented poorly.
However, implemented well, Page Objects are a logical abstraction of an HTML page into a class. They help to enforce Single Responsibility Principle, decrease duplication, and improve structure.
Check out some of my page objects here and let me know what you believe is wrong with them: https://github.com/saucelabs-training/demo-csharp/tree/master/SauceExamples/Web.Tests/Pages
With that said. Why do I need to add a BDD framework on top of this? More complexity and more dependency. For what gain?
I’ve seen few clients succeed with BDD automation frameworks. I’ve seen plenty of clients succeed without them. Again, more complexity just makes test automation even harder.
hello mr.nikolay,
i have a lot of question for you. could you please accept my linkedin? (like my name). you denied BDD and POM, what automation framework do you recommend for us that help test automation UI?
Not POM, I strongly recommend POM. It’s the simplest way to have the easiest to maintain code. BDD is unnecessarily complicated for little extra benefit
I dont think so. BDD give whole team and customers work 2gether. So the product will do its intent. Developers will write a feature and unit test base on feature file. tester can use it to write specific checklist and send to devs. A new comer is able to learn about the systeam with existing living document. So you reject BDD and KDT, i wonder do u use DDT? Can you share me a Github of your fw? thanks.
no, i do not. most of company i have ever worked, they applied cucumber for ui test but they make what mistake that u said. i wish you have a automation UI and API on JAVA :)). but i can exchange 2 languages. thank you so much, sir. c u on linkedin.
Leonardo Arakaki
on December 11, 2019 at 9:09 am
Hi Nikolay, thanks for these eyes-opening concepts.
Have you seen any disadvantage by implementing Atomic Testing in automation projects? I’ve seen this natural movement in a manual testing project, and the only thing that cause some problems but only initially is our test report, the numbers has increased and the client had some resistance about it, but they understood during our progress.
I guess the only “disadvantage” is that it can potentially take a lot of time and effort to make it possible and work. But that’s like saying that testing takes a lot of time and effort and we shouldn’t do it. It’s honestly the only way that an automation project can potentially succeed. The further the tests are from atomic, the less success the project has. That’s the correlation that I’ve observed.
Manual tests =! automated tests. One cannot be translated into another and shouldn’t be. The goal is rapid feedback that let’s us know the quality of the application. If automated tests tell us the quality regarding the features of interest, it doesn’t matter how they are performed as long as the feedback is correct.
Absolutely loved your article. More so about the design practices then the article on cucumber itself (since I do not have enough experience to comment on it). Most practices that are recommended in Selenium.dev are also what you share with examples. I have to start on a front end project and I think I am going to take a lot of lessons from here in that. Thanks for making this!
“Do not use soft assertions” – disagree. Soft asserts allow to catch simple minor issues like ‘error message text NOT correct’ or ‘some minor element did not appear’ and continue flow.
Define soft assertions. I define a soft assertion as a verification that fails the test but does not prevent the test from continuing. It appears you are defining it as a verification that does not fail the test. In that case I would agree it is a wrong pattern. Allowing it to continue allows the test to check for minor changes as well as critical failures. Both are bugs but with different priorities.
This is a soft assertion “verification that fails the test but does not prevent the test from continuing.” You manually check every test to know what fails and what doesn’t?
This is the most detailed and best article on automated testing that I have ever come across. Thank you so much for taking the time to share this invaluable information!
I think your question is whether we check the test reports to see what failed. Yes of course. Even if there was only one assert the test may have failed prior to this with a critical exception so yes, every failed test has to be checked. If you rely on that single assert and don’t check for any unexpected exceptions or errors preventing the assert from even being executed then you are not doing due diligence.
We run thousands of automated tests on a regular basis. This is not a significant effort to review the failures.
I found about in this course https://testautomationu.applitools.com/test-automation-framework-csharp/chapter5.html and we are using it like this without any problem. It was because we created a nuget package with the helpers that we needed and we didn’t want to pass the driver all over the page classes. Another think that I am curious about is how can you run 100 tests in only one minute? if I increase the number of the workers in most cases the running time will increase. we now run them with 4 workers.
Hi Nikolay, thanks for the article. While I agree that following BDD practices, with buy-in from all stakeholders, is the ideal, there are aspects of using Cucumber that provide distinct benefits even outside of a canonical BDD process. In particular, the cucumber step data tables methods, described here, https://github.com/cucumber/cucumber-js/blob/master/features/data_tables.feature, whereby input data can be automatically converted into various data structures depending on input format, lend themselves to highly organized, intuitive, streamlined data input, both for input and expected validation data. These confer an advantage over the obsolescent external data source Excel or csv approaches, as you’ve noted, and constitute an API of sorts for data input that is powerfully integrated with Gherkin’s readability.
As far as Aslak Hellesøy’s comment that Cucumber is overkill for “driving a mouse and keyboard”, that is certainly true; of course the BDD layer is an abstraction on top of another framework that takes case of the browser interactions. For simple applications, certainly balancing the overhead of a BDD layer against the benefits makes sense. However assuming an application’s business logic is complex enough to take advantage of the data handling capabilities, it can make sense even if requirements themselves are not written in Gherkin per formal BDD guidelines.
Furthermore in some implementations of Agile methodologies, it can be harnessed more in the spirit of TDD to backfill the requisite level of detail to user stories that do not specify flow variants and edge cases to the resolution that would be expected in a more traditional development process. If BAs, POs, developers and other stakeholders can agree that once a Gherkin-style test plan is approved via a joint review process, it constitutes an extension of requirements, then some of the benefits of BDD can be accessed.
Getting Started with Appium - Ultimate QA - […] Want to learn more about Automated Testing Practices? I have selected the Top 17 Automated Testing Practices here. […]
Thanks, great read and leads me wanting to learn more and implement this.
Only thing that makes this hard is finding these practice as videos to attempt to carry out/ Learn some of these changes, do you have a tutorial on these? or do you know of some that would allow this learning of these easier to understand (more so around the API and DB level)
Happy to help. Yes, definitely not that many resources out there. I will work on creating some. However, I know some great API testing instructors:
https://beaufortfairmont.com/webinars/
https://www.ontestautomation.com/articles/
Don’t know much about DB instructors 🙁
For BDD, the real anti-pattern is POM, not the feature files or step definitions. Regarded as the de facto design pattern, automation testers push POM to every test automation suite whether it needs it or not. It is safe to say POM is the only design pattern they know, as most of them do not have a proper software engineering background or simply their only experience of code are badly implemented test automation suites.
Dependency Injection: Write Tests, Not Page Objects
https://www.linkedin.com/pulse/dependency-injection-write-tests-page-objects-katherine-rollo
Anything implemented poorly will be a problem. I’ve seen Page Objects implemented poorly and I’ve seen BDD tests implemented poorly.
However, implemented well, Page Objects are a logical abstraction of an HTML page into a class. They help to enforce Single Responsibility Principle, decrease duplication, and improve structure.
Check out some of my page objects here and let me know what you believe is wrong with them: https://github.com/saucelabs-training/demo-csharp/tree/master/SauceExamples/Web.Tests/Pages
With that said. Why do I need to add a BDD framework on top of this? More complexity and more dependency. For what gain?
I’ve seen few clients succeed with BDD automation frameworks. I’ve seen plenty of clients succeed without them. Again, more complexity just makes test automation even harder.
hello mr.nikolay,
i have a lot of question for you. could you please accept my linkedin? (like my name). you denied BDD and POM, what automation framework do you recommend for us that help test automation UI?
Not POM, I strongly recommend POM. It’s the simplest way to have the easiest to maintain code. BDD is unnecessarily complicated for little extra benefit
I dont think so. BDD give whole team and customers work 2gether. So the product will do its intent. Developers will write a feature and unit test base on feature file. tester can use it to write specific checklist and send to devs. A new comer is able to learn about the systeam with existing living document. So you reject BDD and KDT, i wonder do u use DDT? Can you share me a Github of your fw? thanks.
You’re right, in theory. My team of SAs with a combined experience of 100 years in test automation has seen one correct implementation of BDD. That was by Aslak Hellesoy, the creator of Cucumber. Is your team following BDD automation?
https://github.com/saucelabs-training/demo-csharp/tree/master/SauceExamples/Web.Tests/BestPractices
no, i do not. most of company i have ever worked, they applied cucumber for ui test but they make what mistake that u said. i wish you have a automation UI and API on JAVA :)). but i can exchange 2 languages. thank you so much, sir. c u on linkedin.
Hi Nikolay, thanks for these eyes-opening concepts.
Have you seen any disadvantage by implementing Atomic Testing in automation projects? I’ve seen this natural movement in a manual testing project, and the only thing that cause some problems but only initially is our test report, the numbers has increased and the client had some resistance about it, but they understood during our progress.
I guess the only “disadvantage” is that it can potentially take a lot of time and effort to make it possible and work. But that’s like saying that testing takes a lot of time and effort and we shouldn’t do it. It’s honestly the only way that an automation project can potentially succeed. The further the tests are from atomic, the less success the project has. That’s the correlation that I’ve observed.
Manual tests =! automated tests. One cannot be translated into another and shouldn’t be. The goal is rapid feedback that let’s us know the quality of the application. If automated tests tell us the quality regarding the features of interest, it doesn’t matter how they are performed as long as the feedback is correct.
Absolutely loved your article. More so about the design practices then the article on cucumber itself (since I do not have enough experience to comment on it). Most practices that are recommended in Selenium.dev are also what you share with examples. I have to start on a front end project and I think I am going to take a lot of lessons from here in that. Thanks for making this!
“Do not use soft assertions” – disagree. Soft asserts allow to catch simple minor issues like ‘error message text NOT correct’ or ‘some minor element did not appear’ and continue flow.
Why not just fail the test then?
Define soft assertions. I define a soft assertion as a verification that fails the test but does not prevent the test from continuing. It appears you are defining it as a verification that does not fail the test. In that case I would agree it is a wrong pattern. Allowing it to continue allows the test to check for minor changes as well as critical failures. Both are bugs but with different priorities.
This is a soft assertion “verification that fails the test but does not prevent the test from continuing.” You manually check every test to know what fails and what doesn’t?
This is the most detailed and best article on automated testing that I have ever come across. Thank you so much for taking the time to share this invaluable information!
Thank you so much for the kind words Marco Cruz! We are very happy that you liked this article! 😉
Hi Nikolay, thank you so much for sharing a lot of amazing concepts! Do you have an example project using these ones? Thanks!
Hi Calrissa,
You can check Nikolay’s GitHub profile here : https://github.com/nadvolod
Hope this helps!
One thing I disagree is using the static keyword. I understand why you said it. But you have the alternative to use the attribute [TreadSatic]
Hi Ramona, I’m not familiar with [TreadSatic], do you have a link to the docs? Also, curious, why do you prefer this option?
I think your question is whether we check the test reports to see what failed. Yes of course. Even if there was only one assert the test may have failed prior to this with a critical exception so yes, every failed test has to be checked. If you rely on that single assert and don’t check for any unexpected exceptions or errors preventing the assert from even being executed then you are not doing due diligence.
We run thousands of automated tests on a regular basis. This is not a significant effort to review the failures.
I found about in this course https://testautomationu.applitools.com/test-automation-framework-csharp/chapter5.html and we are using it like this without any problem. It was because we created a nuget package with the helpers that we needed and we didn’t want to pass the driver all over the page classes. Another think that I am curious about is how can you run 100 tests in only one minute? if I increase the number of the workers in most cases the running time will increase. we now run them with 4 workers.
Hi Nikolay, thanks for the article. While I agree that following BDD practices, with buy-in from all stakeholders, is the ideal, there are aspects of using Cucumber that provide distinct benefits even outside of a canonical BDD process. In particular, the cucumber step data tables methods, described here, https://github.com/cucumber/cucumber-js/blob/master/features/data_tables.feature, whereby input data can be automatically converted into various data structures depending on input format, lend themselves to highly organized, intuitive, streamlined data input, both for input and expected validation data. These confer an advantage over the obsolescent external data source Excel or csv approaches, as you’ve noted, and constitute an API of sorts for data input that is powerfully integrated with Gherkin’s readability.
An example of such a framework is described in this OpenJS World 2021 talk, https://www.youtube.com/watch?v=86zerP8DMHE&list=PLyspMSh4XhLPVo7CroDYZ8O3rGBs5jp7S&index=11.
As far as Aslak Hellesøy’s comment that Cucumber is overkill for “driving a mouse and keyboard”, that is certainly true; of course the BDD layer is an abstraction on top of another framework that takes case of the browser interactions. For simple applications, certainly balancing the overhead of a BDD layer against the benefits makes sense. However assuming an application’s business logic is complex enough to take advantage of the data handling capabilities, it can make sense even if requirements themselves are not written in Gherkin per formal BDD guidelines.
Furthermore in some implementations of Agile methodologies, it can be harnessed more in the spirit of TDD to backfill the requisite level of detail to user stories that do not specify flow variants and edge cases to the resolution that would be expected in a more traditional development process. If BAs, POs, developers and other stakeholders can agree that once a Gherkin-style test plan is approved via a joint review process, it constitutes an extension of requirements, then some of the benefits of BDD can be accessed.