Unstable Selenium tests are probably the most common issues that we encounter. There are instances that they work, but there are other instances when they don’t. People may find this frustrating. In fact, most of them will just choose to ignore this if the issue is going on for quite a while. This means that they already lose the possibility of catching bugs due to flaky tests.

[Tweet “Here is a method that is helping world-class automation testers avoid flaky Selenium tests.”]

Before we proceed in enumerating the different issues, you need to remind yourself that Selenium is not flaky and your Selenium tests do not have to be unstable. There are different reasons why the tests will fail.

Poor Isolation of Test

During the tests, you might notice that the test is working perfectly fine when alone; however, in the build process, you will experience a random failure. The best solution to this problem is to isolate the resources. The data should be set-up within the Selenium tests to prevent relying on data stores. On the off chance that the application requires a login credential, several accounts should be created first and should only be allocated for the tests.

Flaky Tests Due to Flaky External Devices

When you are depending on an infrastructure that is beyond the control of the team, you will experience flaky Selenium tests with a similar underlying cause. In order to prevent this, avoid relying on the external services that you can’t control. Analyzing the run of your Selenium tests is the perfect opportunity to find the external services that are causing the problem. An easier way to avoid this issue is by running ‘health check’ before you run your tests. This will guarantee that the external services are running properly.

Short Timeouts

There are instances when you have to wait for 15 seconds in order for a request at the Selenium tests to complete. However, if it takes less than 10 seconds, then this is a short timeout. Perhaps the best thing to do here is to avoid using timeouts. If you can’t avoid using them, set the timeout generously. Make sure to utilize the timeout together with a User Interface notification.

Long Timeouts

Some flaky tests will show you 404 or 500 errors. You will also notice that the texts are never appearing. This will normally happen when your tests are continuously timing out. You should not poll only on your desired end-result; you should also consider polling for the usual error conditions. When the screen showed an error condition, fail the Selenium tests with error message.

When you are experiencing flaky Selenium tests, analysis of the root cause will help you find the best solution. Your Selenium tests will not be unstable just because you discover a bug in the framework. In order to guarantee the stability of the tests, you will basically need the help of the entire team. If you have a smaller team, this will not be a problem. However, for those who are working with a large team, the process may be difficult.

What kinds of problems do you have with flakiness in Selenium? Leave your comment below.

Sources:
Simon, Stewart. “My Selenium Tests Aren’t Stable!” Google Testing Blog, https://testing.googleblog.com/2009/06/my-selenium-tests-arent-stable.html