Testing in the cloud vs locally
Testing in the cloud can be fun. You have a shiny new tool and we all enjoy to play with tools.
However:
Testing in a cloud service can also be hard. There are challenges that you will face that I will describe below…
Advantages of cloud based automation
Automatic video recording
Have you ever found a failed automated test and wished that you had a video recording or better screenshot capture?
Of course you have. I never met an Automation Engineer that didn’t need a good video recording to reproduce their automation failure.
Well, I have good news:
With Browser Stack (BS) and Sauce Labs (SL), you can have video recording of your automation test simply by using the paid services.
This feature works out of the box when you link to the RemoteWebDriver using their hub. No extra work required on your end.
It gets even better:
Sauce Labs and Browser Stack will store these videos so that you can access them any time you want to play back your test automation session.
Automatic screenshot capture
Would you like a screenshot captured for every step that you perform using Selenium WebDriver?
How about if each of those screenshots were organized in order so that you can almost recreate the entire automation test by looking at chronological images?
Sounds awesome, right?
Browser Stack and Sauce Labs both provide this capability automatically. And you don’t need to do anything…
Simply perform your standard Web Driver calls as before.
This is an example of what happens in Browser Stack when you perform Selenium commands.
browser stack selenium step screenshot
Enhanced ability to run tests in parallel
If you have a decently sized automation suite, one of your goals is to always speed up the feedback time from those tests.
[Tweet “Have a decently sized automation suite? U must strive to decrease feedback time from that suite.”]
There is only so much optimization that you can do in the code…
So the best way to accomplish this is to execute your tests in parallel rather than in a sequential manner.
Now:
With functional GUI test automation, this idea is WAY harder than you think. First, you require a Selenium Grid setup. And that’s the easy part…
Next, you will need a complete environment that is capable of executing GUI automation. For every single parallel test.
Two tests running in parallel require two virtual machines. Three tests running in parallel require three virtual machines.
Here’s the kicker:
You need to maintain all of these virtual machines…
This means maintaining the Operating System, the security updates, the browsers, browser versions, and anything else that you may require to execute GUI automation.
This maintenance job truly sucks.
But it gets better:
With a cloud automation provider such as Sauce or Stack, these hardware and software resources are maintained by them.
Whenever you want to run five tests in parallel, you simply run your tests. The cloud provider will create the required resources and execute the tests for you.
I’m assuming that you paid for enough resources and your tests are configured to run in the cloud provider.
If you paid for ten virtual machines, you can run ten automation tests in parallel with the click of a button.
No VMs means no maintenance. Simply focused test automation, with extremely rich logging.
[Tweet “No VMs means no maintenance. Simply focused test automation, with extremely rich logging @sauceLabs”]
Disadvantages of cloud based automation
Executing in the cloud is slow
Honestly, functional test automation execution in the cloud can be slow…
From my experience, tests running in the cloud providers are fifty to one hundred percent slower. It makes sense if you think about it.
It takes time to spin up a virtual machine with the appropriate resources. It also takes time to send all of your Selenium information to the cloud service and have it execute and display the results in an easy to understand format.
That’s why you will require at least two or three virtual machines.
Multiple VMs required to make it worth it
All of us desire a faster feedback time. It’s always nice when your tests can run within seconds or minutes.
Therefore, at some point, you will begin to try and run your tests in parallel to drastically reduce the feedback time. Running on your local resources may not be so hard…
Honestly:
Running in a cloud automation provider like Sauce or Browser Stack is more difficult.
Since automated checks executed in the cloud are fifty to one hundred percent slower, you will need between two to three virtual machine instances for every one test that you were planning to run locally.
Here is an example:
If you execute a single test on your local hardware then that test takes thirty seconds. If you want to execute the same test in Sauce Labs for example, that test may take up to sixty seconds to finish.
Therefore, you can execute two tests locally in the same amount of time that you can execute a single test in the cloud.
Now:
This means that in order to outperform your local resources in terms of feedback time, you will need to execute three parallel tests on three parallel virtual devices.
This will be three tests executed in a minute in the cloud. While executing only two tests in a minute on your local hardware. Assuming that all of the tests have a similar run time.
Therefore, in order to make paying for the service actually worth the investment, you will need to pay for at least three consecutive virtual devices.
[Tweet “Paying for 1 cloud device for automation will almost never improve your feedback cycle time”]
Obviously, there is an argument to be made for the extra features that you receive such as video capture and logging…
I’m only considering feedback cycle time in this example.
Configuration of Remote WebDriver can be tricky
Configuring a local browser is as easy as it can possibly be.
This single line of code creates us a new ChromeDriver that we can use locally. Assuming you have Chrome browser installed.
If you would like to create a browser that works with a service like Sauce Labs, things can get a bit more complex…
Certainly not a single line of code any more.
Not that this is extremely hard. But there is a larger barrier to entry.
You need to know the correct values to pass in for each of the capabilities. These configurations simply take more time to learn.
Sauce Labs vs Browser Stack – Feature Comparison
Free Trial
Both of the services offer a free trial.
Sauce labs free trial options
Browser Stack’s free trial doesn’t clearly define what you get until you actually sign up. After you create your free account, you will notice this information.
Browser stack free trial options
100 free minutes just like Sauce Labs and 10 parallel testing sessions vs 2 for Sauce Labs.
However, with Browser Stack, there is a phenomenal trial feature…
If you use Edge Driver, then you can have unlimited amount of testing using Browser Stack.
This means that you can run ten tests in parallel with no limits on number of minutes with BS.
Winner: Browser Stack
Stability
Don’t you hate products that are supposed to help you with Quality Assurance, but then those products are full of bugs themselves? So frustrating when that happens.
Well with Sauce Labs, you don’t have to worry about this.
Sauce Labs is a well-built application that doesn’t have functional issues. Everything works well and is pretty easy to use.
Browser Stack is significantly less stable. From the years of experience that I had using this service, there have been tons of bugs. Some that I can remember include:
- Logs stopped working so that screenshots and videos weren’t possible
- Interactive videos stopped working
- They also got hacked
Winner: Sauce Labs
Innovation
Which of the tools is more innovative and pushes the boundaries of test automation?
Over the years, Sauce Labs was the first to market with the Jenkins plugin. Browser Stack followed. Sauce Labs has new and updated Team Management that makes managing of users for large organizations much easier. Sauce Labs is capable of tracking your network traffic and generating a HAR file to show you possible problems with your web requests.