Test Manager UI
In this video tutorial, I want to talk to you about the test manager, all of the different capabilities and also discuss the batches.
So, what you see over here is the test manager. In here you see a collection of all of the tests that have been executed.
On the left-hand side, you see a batch pane. Previous to this, you had no concept of batches, what you actually saw was a single test being executed one at a time. However, at some point, you might want to get more advanced and execute a suite of automated tests.
That’s where batch has come in handy, because now what you can do is execute multiple tests at the same time and place them into a single batch.
So, what you can see here is I actually was able now to provide a batch name. You can see I gave ultimate QA different resolutions. Inside of here, you can see the date and when it was executed, but also you can see how many tests are unresolved and how many passed.
And, you can actually expand this to see more and two test passed. And, actually, the batch is highlighted right here. When I click it, it shows me everything that’s happened and you can see all of the tests that I executed one by one, but also they’re inside of a batch.
Similar to like a test suite. So, this is my ultimate QA home page being tested on different resolutions.
There is also the concept of a batch summary view that you can see right here. So, the batch summary view basically allows you to see the statistics for this batch. So, you can see there’s a test result, there’s the diff status, so in this case, I had a 40% pass/fail pass rate and 60% unresolved rate.
I’ve got to get in and see why are these tests unresolved. Then you can see all of the different kinds of other metrics.
You can see the browsers, the operating systems. Whether there are any issues and remarks and any test
assignment, then you can see the different kinds of viewports that were executed here.
There’s this other view, the batch details view, which you can see you’ve already seen here. So, it shows you one test execute at a time. You can actually click on it and it will expand out to give you a little preview of what happens.
So, in this case, it’s passed, so you can see there’s a little equal sign that says it was a match. Everything is ok. You can expand the ones that didn’t pass and you can see that it gives you a preview of what might have failed.
This one as well, this one has a preview and you can actually if you want, you can open it in the step editor. You can show the baseline image, so you can toggle like this versus the different images. Okay?
You can highlight the diff locations. And, you can give it a thumbs-up or thumbs-down from here, then actually save it from here. So, for example, if I give this a thumbs down. Now, this has been rejected. Later I just need to save it, but I don’t want to give it a thumbs down so I’m not going to save it. The same thing with these other ones, as you can see, shows you the diff.
Now, one thing to keep in mind, this is actually a preview mode. Applitools uses a little bit of a different algorithm. Here, it’s not the full Applitools comparison algorithm that’s used here. So, what you see here might actually be different when you open it in step editor.
So, just keep that in mind, this is like a preview and then the full algorithm gets applied whenever you open it in the step editor.
Oh! By the way, something other very important here is you can actually group your tests. So, later I’ll show you how you can apply properties to your test cases. But, automatically, Applitools detect stuff like the application, the branch, the environment, name, and so on.
So, based on that you can actually group your test. So, for example, if you want to group it by an operating system, now everything is grouped by Windows 10 operating system. Because that’s where I ran it.
I can also group it by the viewport size, now you can see that under Windows 10 I have several different viewport sizes, and in each one, I only have a single test but in this one, I have two tests, in 1920×1080 resolution. You can see I have two tests in here if I wanted to look at them like that.
So, you can continue to drill down if you want by the environment name which I actually had never said so, I don’t think any of these will have. You see? yeah? See? Exactly!
Not applicable because I don’t have an environment name set up, I can remove it and so now everything just looks as before. This is a very nice way to drill down if you know you have a huge batch.
I’ll tell you at one of my previous workplaces, we ran about anywhere between 3,000 to 9,000 visual validations every two weeks.
So, those batches can look kind of big. We didn’t have all 3,000 to 9,000 in a single batch but there could be like over a thousand visual validations in a single batch. So, you might want to filter them down based on some specific criteria. That’s what we did using these group bys.
Then, there’s this other view batch, batch step view. Batch step view basically is really nice if you want to see all the images at the same time and see all the differences from here. Imagine in that scenario where we ran between 3,000 to 9,000 visual validations and we opened up a batch and there are about a thousand different visual validations.
We like to open it up and look at it from here. A lot of times what you might see is some common failures. And something like this you might be like, oh! Ok! That’s actually totally ok, so I’m just going to go ahead and give this some thumbs up because I know that’s changed based on the new requirement.Ft
I’m just going to give thumbs up from here, thumbs up from here, you know?
Then you can click a “save” in order to save the changes.
It’s very quick and easy to view from here. You can also even “star” these so that you can remember to come back to them in the future. Then you’ll be able to filter based on the starred ones.
Then there’s also issues and remarks view. Which is not going to have anything right now because we don’t have any issues and remarks. But later, we’ll make issues and remarks and you’ll be able to see it.
Then they’re just here reloading. You can also search based on the different filters. So, for example, if I “star” one of these, you can search and you can look where are my starred ones? See? Now it’s sort of showing you only the starred ones.
You can also check by unresolved, you know? Sign to me, you can apply more filters, maybe, you want, based on the match level. Then you can see, let’s see which ones had layout, okay? That’s all the layout ones.
Which one has strict? Okay? None of them had a strict match level, see that? So you can filter down here. It’s super useful to be able to play around with these different item types and filter, non-filter, and so on, and so forth.
Actually, one other thing I wanted to share is you can actually share this view. You can copy a link to the clipboard and then share it with your team. Your team does have to have an Applitools account in order to be able to log in and view it. But it makes life very easy because you just copy the URL and now you can share it.
Now, over here on the left-hand side, I already mentioned to you the batches. I just want to mention some other options. Obviously, you can do stuff with the batches here, you can toggle. So, for example, if you want to do something with these two batches, you can delete them if you want. I don’t want to do that.
You can reload the batches so if you have something running, I’m going to click “confirm” because I don’t want to save anything, if you want to have some batches running you can keep reloading them and then they will appear here.
You can filter based on the batches.
So, you can see that all of the batches automatically get a status. So, in this case, it’s unresolved because I have some unresolved tests. But this one had a past, you know? Other ones might have a failure, so you can filter based on them here. You can even apply more filters here based on the batch status, test status.
Also, I love to filter based on the batch name. So, if you come up with a really good batch, a naming convention, you can filter using the batch name. So, for example, if I do different resolutions and then hit enter, hit apply. It will only show me all the batches that have different resolutions in the name.
How awesome is that, right?
So, if I remove this, you’ll see that I have other batches that don’t have that in it. So, here, for example, see that? I had these other batches that I was doing. It’s very useful!
If you have used this in the past in different workplaces. I’ve actually used the application name and version in order to be able to see the screenshots for that application name and version. You might want to batch it, maybe, based on browser types. Maybe, based on operating systems, depending on what your large group category are, right?
This is, again, you’re running tests. You want to group all of your tests into a single place. You can filter it based on functionality, you can filter it based on some kind of organizational unit that makes sense to you.
For me, it made sense to group them here based on different resolutions per page. So, I’m going to have a page, and then, here, I have all the different resolutions for that page. So, that’s how I’ve used batches here.
All right! So, now you’ve got a whole of that covered. You understand basically everything in the test manager.
Let’s go ahead and dive into the code because I’m sure you’re dying to find out, how did I actually make all of this happen?
Batches
Sweet, so let’s go and look at the magical code that actually makes the batches happen.
So, I’ve created a class here called “Batches”. That’s going to run and create batches and give you an example.
Our batch’s code actually inherits from “BaseClass2”. The reason I had to create a “BaseClass2” was because I had to modify a bunch of the setup and teardown methods to make it do what I want. That’s why I had to create a “BaseClass2” and you’ll see that as we step through the code.
Now, what I want to do here is take you through this test case right here “HomePage1080p”. It’s a very simple test case, but it’s fantastic because it’s very realistic. It actually analyzes my ultimate QA home page and the reason why this is great is because it’s a real web application that actually gets real traffic and real production updates.
A lot of times, you know? These vendors they make seem like it’s so easy to use their tools. When in fact that may not be the case. Applitools has actually been one of the easiest tools to use that I found. But, it can have a lot of complexity and it can have a lot of little nuances depending on the webpage that you’re developing.
So, I wanted to do something real and to show you it’s working on a real application and the real problems that we would encounter along the way.
I think the best way for me to show this to you is to execute this test and as we debug and step through it, I’m going to describe and tell you everything that is happening.
What I’m going to do here is, come to my test explorer. In my test explorer, I’m going to expand this, go to Applitools and then go to batches. Then, I’m going to have my homepage 1080p and I’m going to go ahead and debug this selected test.
Now, one tip I want to tell you is to keep your test names short, so that they appear in the test results view of Applitools. You’ll see this when I’m showing you the test results view again.
But it’s really nice if you keep your test name short, then what you can actually do is use the method name using reflection. You can grab the method name and pass that into your test execution, then you can use that dynamically.
Anyway, so, the very first thing that happens here is in the “BaseClass2”. There’s a bunch of properties that you’ve seen before, that’s not a problem. One new property that you need to pay attention to is this “BatchInfo” property.
This “BatchInfo” property comes from Applitools. If I hover over it, it comes from the Applitools namespace, “Applitools.Selenium”. So, I’ve created a static property called “MyBatchInfo”. What this is actually going to do is store my batch information related to Applitools for all of my tests.
I’ve created a “OneTimeSetUp” attribute here. A “OneTimeSetUp” attribute from NUnit allows us to be able to initialize some stuff before an entire test class. So, why this is important is because inside of our test class batches, we can have many different test cases, which we do, right?
I am testing the homepage in many different resolutions and we want them to all go into a single batch. So, if that is the case you want to execute a “OneTimeSetUp” and then provide your batch information. Now, all of the test methods inside of your single class will go into that batch.
That’s exactly what I’m doing here, I’m setting “MyBatchInfo” which is that static property to this “NewBatchInfo” from Applitools. And, the “BatchInfo” constructor takes in a string, let me navigate to that definition. It takes in a string a name, okay?
So, the name of your batch or it has this other constructor that you can obviously supply and put in a daytime offset, but I am not using that right now.
So, I supplied a batch name which is this which what you’ve seen before, I’m calling it “UltimateQA-DifferentResolutions”. And, again, you can specify whatever is relevant for your application.
Now that we’re going to set. We have this, it is going to apply to all of our test methods. So, I’m going to go ahead and continue stepping over this. Now, I’m going to create a new “ChromeDriver”. You’ve seen that before, nothing different there.
Set an implicit wait, nothing different there. Creating a new Applitols eyes object, nothing new there. Now, this is something new. You’ve got “Eyes.Batch” which is coming from Applitools eyes. We’re setting the batch which is a batch property inside of the Applitools object to our batch info that we created before.
You can see that if I hover over “MyBatchInfo”, you can see that what’s supplied in here is, there’s an ID which is automatically generated. There’s a name, which is what I supplied and then the “StartedAt” date.
I configured this part. So, now, again, this allows me to have the batches in the test manager. I’m going to go ahead and step over that. Match level I just set to layout because that’s what I want to use for my home pages.
This is also another new thing that you have not seen. Applitools has previously been supplying properties for us automatically based on writing the operating system, the browser, browser version, and so on, stuff like that. Which it can automatically detect, but now these are custom properties that you can add to your tests.
So, remember that this right here is actually being executed in my “SetUp” attributes. So, it’s going to execute for every single test case. For every single test case, I want my page name to be set equal to a home page. Because in this case, my batches class is all about the home pages.
You can obviously apply this at a test case level as well. If you wanted to be inside of here and add some other properties, maybe like, a resolution 1080p. It may be in here you want to add a resolution 1192×969, like that.
If you wanted to do that, you can at the test case level as well. I’m doing it at the “SetUp” attribute level because I wanted to apply to every single test case regardless. So, every single one of them is receiving a string which represents the property name and a value, super simple, just property name and value.
So, now all of my tests will have this as their property. Then you’ve seen the Javascript, that’s normal. And, then we are going to a home page which will automatically open our application, then perform the appropriate actions.
I want to take you in here really quick because there are some other steps in here that are not so important right now. I’m going to tackle them in a moment, but I want to cover batches right now. So, don’t worry about this, but what’s important here as well is I’m using this “MethodBase” to dynamically get the test case name.
I’m supplying that into my test steps. I’ll show you that. What I’m actually going to do, just go ahead and execute this test. It’s actually a little slow because you’ll see what I actually have to do is deal with the popups that are on my home page. That may be a very common activity that you might encounter in your application.
That is why I think this test is extremely awesome because it’s a real-life test where you have to encounter real-life problems. So, I’m just going to go ahead and let it run, I’m going to speed it up through the video so that you don’t have to wait for everything. Then after we’ll go and see the outcome in Applitools.
All righty! Fantastic!
So, as you can see, the test has actually finished up!
Let’s go ahead and take a look at the UI to see what’s happened there. So, what I can do is go ahead and refresh. You can see that here is a batch and it has a single test inside of the batch. Why does it have a single test?
Well, because we only just ran a single test case. If we had run more, more tests would appear in this batch. Just like this one.
That’s pretty much all there is to batches. There is a lot more stuff that I have to show you, specifically how did I create these such long pages, and, how to deal with the popups.
Let’s go and cover all of that.
Your turn
What do you think is the most useful capability of Test Manager UI? Why? Let me know by leaving a comment below.