Exact match level
Now, in this section, I wanted to talk about some really important concepts, specifically related to match levels and different types of regions.
That’s why I actually left you hanging on all of these regions and these steps right here. Because these concepts are a bit more advanced, a bit more complex.
I wanted to leave enough time to discuss them on their own and focus on them on their own. So that I’m not just randomly throwing in large complicated concepts at you, through, you know, these little tiny videos that show you very simple concepts, such as these.
Actually, the options here are known as match level settings. They are very closely related to these regions. Once you understand match level settings, you can understand the different types of regions as well.
Match level settings
Let me go through the different match level settings. Applitools offers four different match level settings, the first one that we’re going to talk about is the exact match level setting. What this means is that, actually, Applitools is going to do an exact pixel by pixel comparison of the expected image to the actual image.
Now, actually Applitools does not recommend the exact match level, the only reason it’s there is for demo purposes. Because remember Applitools actually has very smart algorithms that will ignore changes that a human eye cannot detect. Exact is only there for demo purposes so it’s not recommended that you use it.
But, I will show you the kind of changes that an exact algorithm or other tools that use pixel to pixel comparison will catch when in reality they should probably not be catching those bugs. Therefore, causing false positives.
Take a look at this slide deck here, what I want you to pay attention to is the changes that you’re going to see happen in this area. It’s zoomed in and really, we are zoomed in on this region right here. So, this is what a human eye sees right, this is how it actually looks on the web application.
This is what’s happening on a pixel to pixel comparison, so if the strict mode was turned on or pixel to pixel comparison with other tools, besides Applitools was used. This is what you would see.
Take a look, you can obviously see that there are some pixel changes but if you look at the image below, the actual real image, there is nothing perceptible to the human eye, right. You can’t tell anything is even going on down there in that white circle.
This is something that pixel to pixel comparison tools will actually catch. You’re going to get a false positive, wasting your time as an automation engineer on dealing with this issue, that is not really an issue. No one is going to care about this slight pixel shift that’s not even visible to the human eye.
Here’s another example that would be caught in strict mode and pixel to pixel comparison, but would not be identified otherwise by Applitools smart algorithms that would ignore these kinds of changes.
Take a look at this over here, the word over here, this is what happens to it. It’s basically brightness changes that happen to it, that would be identified if it were not for the Applitools smarter algorithm.
You can see again, right? That’s what happens. It is really just the brightness changes, but it’s nothing significant. Everything remains the same, the strings, all the text is exactly the same. The word is exactly the same. But just the brightness changes a bit on different devices. Again, a false positive here.
Here’s another very good example, take a look at the image on top where we are zoomed in, to these, I guess, red lights on top of the car. Then, pay attention to how is visualized in the actual image over here.
I’m going to flip through them. See the shift in the pixels, right? But also pay attention to the bottom image, how there is absolutely no perceptible difference in the actual webpage, right? You can’t see anything shift, there’s nothing changing.
Again this would be called as a false positive in an exact mode or in pixel to pixel comparison. But, with Applitools smart algorithms this would not be the case.
So again, I just wanted to show you these examples of what would happen if you were doing just a very exact pixel of pixel comparison.
Again, you shouldn’t really need to use this mode. I just wanted to show you this that it’s there and that it does exist.
Strict match level
Next, I would like to bring your attention to the strict match level.
The strict match level is the next evolution of the pixel to pixel comparison. In Applitools, the strict match level actually uses artificial intelligence in order to figure out whether the change that has happened on the page is perceptible to the human eye.
That’s really what defines the strict algorithm. Is it perceptible to the human eye? And if it is then it’s going to spot it out as a discrepancy on the webpage. If it’s not a change that is perceptible to the human eye, it will not be spotted out.
In this case, in this example, as you can see, if I tap through. It’s very easy to see, right? This is perceptible to everybody, everyone can see all of these changes without a problem because they’re so obvious and of course Applitools will spot this out.
The strict algorithm will basically help you to identify any changes that can happen to the page. Color changes, text changes, right? If the font style changes. It can help you see changes in, like, shadows. If this changes to different colors, it will spot it all out.
Keep in mind: “Strict is your default match level”
Basically, the majority of this course is going to be performed using the strict level. The strict match level is actually the default level that comes in your code whenever you’re using Applitools.
Let me show you what I mean by that. Whenever you’re using the check window method or any other of the check methods and you didn’t specify a match level using Applitools. Then you get the strict match level.
I’m going to cover these, all this code in a second. But just keep that in mind, that strict is your default match level. For the majority of the situations, it’s usually the best level to use.
In fact for myself, in doing about 5,000 automated visual validations, using Applitools. I’ve discovered that using the strict match level, I’ve never had a false positive.
Anytime that Applitools tells me that there is an actual failure, that there’s an actual visual discrepancy. There has been a visual discrepancy.
Then, the question arises. Is it an actual bug?
Image source: https://gifer.com/i/79kQ.gif
Or is it an expected change? Maybe the requirements weren’t updated. That’s a conversation we always have.
But, I’ve never had it point out a difference that wasn’t actually a difference that needed to be looked at.
The strict algorithm is very smart and helps you to reduce those false positives
Content match level
Let’s talk about the content match level.
The content match level is basically the same exact thing as a strict match level, however, it does ignore colors.
This is useful for whenever you might have a website that includes many different colors and the colors aren’t relevant to your website. Then you can use the content level.
Let me show you here, as you can see, the differences are off. But, if I do turn on the differences, using the D key, you can now see that it’s highlighted this difference which is there, but it didn’t highlight this yellow.
Let me place the images side-by-side, see that? You can see that this used to be white and now I changed it to yellow. It didn’t highlight those differences. See that?
That is the content match level ignoring the colors. If the colors aren’t relevant, you can use it.
Coding content match level
Let’s take a look at the code. So, to get the code, what I’ve done here, I created a brand new class called match levels. I tagged it with category match levels that inherit from our standard base class. In there, I have our setup and teardown methods which create Applitools and destroy Applitools and all the relevant browsers just like before.
Then here, I have a test method called Content match level. You can see that the very first thing I do is I say “Eyes.MatchLevel”. This is a property that allows me to set Applitools eyes match level.
In this case, there’s an enum that I use that allows me to specify content. There are multiple different enums that are available.
Watch! If I click here, you can see content, layout. Here’s layout legacy, not really used anymore, none and strict. Those are all the options that you have available from the match level.
Then, the next method goes to the pricing page, just like before.
If I pick this definition, you can see that “GoToPricingPage”. You’ve already seen this method before, all it does is go to the fake pricing page.
Now, this method is new and I made it specifically for the different match levels. So if we scroll down and see what this method does, it says that it’s called “ChangeToEuroAndUpdateColor”.
As you can see here, the JavaScript class which I’ve actually moved into the base class. We can pick this definition as well. You can see that the JavaScript class I’m now storing in the base class, because before I was creating it in every single method and I removed that duplication, so in my base class I’ve got the JavaScript class.
Then I’m actually instantiating the JavaScript class inside of my setup method which you can see right here. So, whenever I need a JavaScript execution class, I instantiate it right there.
Using that class, I execute a script which you’ve seen before. Here, I locate an element by class name, using this class name. It’s the first element in the collection and I update its inner text to be the Euro with the zero.
You’ve already seen that change before.
What’s different here is, I use this JavaScript to set the style of the element to this yellow color. How I found this yellow color? I just googled different colors and so then I just inserted it in here.
Based on that, I was able to update the color of the element to yellow instead of the white that was before. Using all of that, I updated the colors.
Then, we do the standard as all before. We’re opening Applitools with the driver passing in the same AppName as before. Which, if we pick the definition, you can see it is “sample app 1”. The test case name is “Test1”. The resolution is 1080p, our standard baseline.
Then, we do a “Eyes.CheckWindow”. Now, the check window method actually has two overloads. You can pass in nothing or you can pass in something. What you can pass in is known as a tag. So, here I’m just tagging my test. Kind of giving it more information.
I’m saying that this test is “MatchLevel.Content”. You can put anything you want here to describe your test case. I’m putting in “MatchLevel.Content” so that it’s easier for me to see when I’m actually looking at the Applitools.
Let me show you where that comes up. So, if I come here, you can see that “MatchLevel.Content” appears right here.
I know that this is our checkpoint that I had with this specific tag. That’s how I know that this is different from the other test cases that I may have run.
All right, so that’s pretty much it for the content level. Let’s proceed.
Layout match level
Let’s talk about the final match level type which is known as Layout.
Actually, Layout is basically exactly like it says. What it’s going to do is compare the layout of the page to see if the elements are in the correct position in relation to each other. It ignores content, color, and other style changes.
Let me actually fit this, so you can see that with the diffs turned on, it highlights absolutely nothing. Because the layout of the page has not changed at all.
Let me show you some examples of layouts changing so that you can better understand exactly what I mean. Here is an example of Paychex.com. Not sure if you have heard about it, but it’s a pretty popular website.
Take a look at these two pages side by side and let me know if you can spot any differences whatsoever. Take a look, pause this video and then we’ll come back, I’ll show you the answer.
So, if you were not able to spot the differences, I mean, they are pretty hard to catch. However, if we do our standard process of putting these on top of each other. Then tabbing through it, you might be able to see the differences.
So, the differences down here at the bottom, and I can point it out to you like that. You can see that the layout has changed. We’ve actually got a new element in there that wasn’t there previously.
In fact, we got the element and this horizontal box. But you can see that now Applitools has highlighted this element that has appeared. The reason it’s highlighted is because the layout has changed now, right? This button has a new element inside of it and so the layout algorithm catches that.
Let me show you another example, here you can see Twitter.
You see Twitter, this is on Android and Samsung Galaxy S5 in this resolution right here. So, my question to you again is, are you able to spot any layout differences? Keep in mind that it’s in elements in relation to each other.
Take a look, pause the video, think about it and when you come back I’ll give you the answer.
So, again, we can turn on the differences and were able to see that Applitools has highlighted all of that. We can highlight the differences like this, you can see that there are all of the differences.
The reason why the layout algorithm has identified this, as you can see that the layout of the tweets has actually changed. So, before, as you can see the tweets here, were actually all aligned like this. But over here, for whatever reason, there was a bug and the layout has changed so that the content has wrapped underneath the actual logo. You can see that, right?
Finally, there’s another layout change which is over here. You can see that the TechCrunch logo has disappeared. So, you see the TechCrunch logo here but you don’t see it here. The layout again has changed because there is no element here.
One other thing to keep in mind in regards to the layout algorithm. Notice how it didn’t highlight these differences, right? It didn’t highlight any of the content, it’s not highlighting this. it’s not even highlighting the differences in these logos, right? You see that?
Even though everything is different, it’s not highlighting and if we were to use strict mode it would highlight everything. Because everything has changed, see that? But, since we’re using layout mode, it only cares about the layout. Ignores content, ignores colors and different styling effects. That’s what layout is useful for.
Coding layout match level
Let’s go take a look at the code. Over here you can see that I’ve created a layout match level method. As previously, what I’m doing is setting the match level to be “MatchLevel.Layout” in this case.
Then, I’m repeating all the steps as before. I’m going to the pricing page, then, I’m changing to Euro and updating the color. Then, opening eyes and then I’m doing a check window, passing in a different tag to identify my test case, saying “MatchLevel.Content” which is not right. It should be “MatchLevel.Layout”. There we go!
That’s the only difference! If you wanted to use layout match level for one of your tests in an automated way, this is how you would do it, the rest of the code would remain the same.
Your turn
What do you think is the most useful match level? Why?
Let me know by leaving a comment below.